Skip to main content

QoS Orchestration

Technical deep dive into the QoS orchestrator daemon.

Overview​

abng-qos is the QoS orchestrator daemon. It receives session up/down/rate-change events from abngd over its Unix domain socket and applies or removes per-subscriber CAKE qdiscs and IFB (Intermediate Functional Block) devices for both egress and ingress shaping. Overhead profiles (e.g. pppoe_ethernet overhead 34, ipoe_ethernet overhead 22, mpu 64) are applied to CAKE.

Shaping is split by direction:

  • Downstream (to subscriber) — egress CAKE qdisc on the subscriber's access interface: the demux VLAN for DHCP/IPoE (e.g. sub0.1234.2) or the ppp interface for PPPoE (e.g. ppp0).
  • Upstream (from subscriber) — ingress shaping on a per-session IFB device named ifb<session_id> (e.g. ifb10).

A subscriber with no rate (no plan) is left unmetered: abng-qos skips shaping when the rate is 0 rather than applying a 0-bandwidth qdisc. This applies to both DHCP/IPoE and PPPoE.

CAKE Qdisc​

Qdisc Creation​

When a session is established, abng-qos creates a CAKE qdisc on the subscriber's access interface (the demux VLAN for DHCP/IPoE, or the ppp interface for PPPoE):

tc qdisc add dev ppp0 root cake \
  bandwidth 100Mbit \
  overhead 34 \
  mpu 64 \
  rtt 20ms \
  ack-filter \
  split-gst \
  diffserv4 \
  nat \
  nowash

Parameters​

ParameterValuePurpose
bandwidthRateDownload/upload rate
overheadBytesEncapsulation overhead
mpuBytesMinimum packet unit
rttTimeRound-trip time estimate
ack-filter—Filter duplicate ACKs
split-gst—Split GST (get-set-test)
diffserv4—4-class QoS
nat—NAT detection
nowash—Disable wash

Overhead Calculation​

CAKE uses overhead to accurately calculate transmission time:

Transmission Time = (Packet Size + Overhead) / Bandwidth

Example (PPPoE):

  • Packet: 1500 bytes
  • Overhead: 34 bytes
  • Bandwidth: 100Mbit
  • Transmission Time = (1500 + 34) / 100Mbit = 122.72 µs

Ingress Shaping (Upload)​

IFB Device​

For upload rate limiting, abng-qos creates a per-session IFB (Intermediate Functional Block) device named ifb<session_id> (e.g. ifb10 for session 10):

# Create per-session IFB device
ip link add ifb10 type ifb

# Bring up IFB
ip link set ifb10 up

# Redirect ingress to IFB
tc filter add dev ppp0 parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev ifb10

# Apply CAKE to IFB
tc qdisc add dev ifb10 root cake bandwidth 100Mbit ...

Why IFB?​

Linux kernel doesn't support ingress qdisc directly. IFB provides a workaround:

  1. Redirect ingress traffic to IFB egress
  2. Apply qdisc to IFB egress
  3. Packets are then transmitted normally

Rate Changes (CoA)​

In-Place Update​

When RADIUS CoA changes rates, abng-qos uses tc qdisc change (not replace) to update bandwidth without packet loss:

# Old way (causes packet loss)
tc qdisc replace dev ppp0 root cake bandwidth 25Mbit ...

# New way (no packet loss)
tc qdisc change dev ppp0 root cake bandwidth 25Mbit ...

Process​

  1. abngd receives CoA-Request with new rates
  2. abngd updates session in database
  3. abngd notifies abng-qos via IPC
  4. abng-qos updates egress CAKE: tc qdisc change dev ppp0 ...
  5. abng-qos updates ingress CAKE: tc qdisc change dev ifb<session_id> ...
  6. No packet loss during update

Session Cleanup​

When a session terminates:

  1. abngd notifies abng-qos of session removal
  2. abng-qos removes egress CAKE: tc qdisc del dev ppp0 root
  3. abng-qos removes ingress filter: tc filter del dev ppp0 parent ffff:
  4. abng-qos removes the per-session IFB device: ip link del ifb<session_id>

Statistics Sampling​

abng-qos samples all CAKE qdiscs every 5 seconds with a single tc -s -j qdisc show, computes a per-session throughput snapshot (a 5-second average of the byte delta), and caches it in memory. It tracks shaped sessions from the QoS up/down events and serves the cache over a qos.statistics request/response IPC.

This makes show qos statistics a fast in-memory read — abngd queries the cached snapshot rather than spawning tc per call:

  1. CLI issues show qos statistics (alias show qos stats)
  2. abngd reads the active QoS sessions from its session DB
  3. abngd requests the cached throughput snapshot from abng-qos over IPC
  4. abngd merges the configured rates with the per-direction counters (bytes, drops, backlog, 5-second-averaged current rate) and returns the table

Overhead Profiles​

PPPoE over Ethernet​

pppoe_ethernet:
  overhead: 34
  mpu: 64

Breakdown:

  • Ethernet frame: 14 bytes (dst 6 + src 6 + type 2)
  • PPPoE header: 6 bytes (version/type 1 + code 1 + session 2)
  • PPP header: 2 bytes (protocol 2)
  • IP header: 20 bytes (minimum)
  • FCS: 4 bytes (frame check sequence)
  • Total: 46 bytes (rounded to 34)

IPoE over Ethernet​

ipoe_ethernet:
  overhead: 22
  mpu: 64

Breakdown:

  • Ethernet frame: 14 bytes
  • VLAN tag: 4 bytes
  • FCS: 4 bytes
  • Total: 22 bytes

PPPoE over QinQ​

pppoe_qinq:
  overhead: 38
  mpu: 64

Breakdown:

  • Ethernet: 14 bytes
  • Outer VLAN: 4 bytes
  • Inner VLAN: 4 bytes
  • PPPoE: 6 bytes
  • PPP: 2 bytes
  • IP: 20 bytes
  • FCS: 4 bytes
  • Total: 54 bytes (rounded to 38)

Monitoring​

View CAKE Stats​

tc -s qdisc show dev ppp0

Output includes:

  • Packets sent/dropped
  • Bytes sent
  • Marks (ECN marks)
  • Overlimit drops
  • Collisions

View Live Throughput​

abng> show qos statistics

Per-session, per-direction throughput (configured rate, 5-second-averaged current rate, transferred bytes, dropped, backlog) read from the in-memory sampler cache. See the operational-mode CLI reference.

Check IFB Device​

ip link show | grep ifb
tc -s qdisc show dev ifb<session_id>

Monitor abng-qos Logs​

journalctl -u abng-qos -f

Performance Considerations​

CPU Usage​

  • CAKE is CPU-efficient (uses O(log n) operations)
  • Typical CPU usage: <1% per 1000 sessions
  • Overhead profile selection affects accuracy but not CPU

Memory Usage​

  • One qdisc per session
  • ~1KB per qdisc for kernel state
  • 10,000 sessions ≈ 10MB memory

Bandwidth Accuracy​

Accuracy depends on:

  • Correct overhead profile selection
  • Accurate MTU configuration
  • CAKE algorithm parameters (rtt, ack-filter)

Troubleshooting​

QoS Not Applying​

Check 1: QoS Enabled

show configuration | match qos

Check 2: CAKE Qdisc Exists

tc qdisc show dev ppp0

Check 3: Session Active

show subscribers

Check 4: abng-qos Logs

journalctl -u abng-qos -f

Rates Not Matching Expected​

Check 1: Overhead Profile

show configuration | match overhead_profiles

Check 2: Test with iperf

iperf -c <subscriber-ip> -t 10

Check 3: CAKE Stats

tc -s qdisc show dev ppp0

Check 4: Adjust Overhead If rates consistently off, adjust overhead profile.

IFB Ingress Shaping Not Working​

Check 1: IFB Device Exists

ip link show | grep ifb

Check 2: Ingress Filter

tc filter show dev ppp0 parent ffff:

Check 3: IFB Qdisc

tc qdisc show dev ifb<session_id>

Check 4: abng-qos Logs

journalctl -u abng-qos -f

Next Steps​