Skip to main content

QoS Orchestration

Technical deep dive into the QoS orchestrator daemon.

Overview​

abng-qos manages per-subscriber traffic shaping using CAKE qdisc and IFB (Intermediate Functional Block) devices for both egress and ingress shaping.

CAKE Qdisc​

Qdisc Creation​

When a session is established, abng-qos creates CAKE qdisc on the subscriber interface:

tc qdisc add dev ppp0 root cake \
  bandwidth 100Mbit \
  overhead 34 \
  mpu 64 \
  rtt 20ms \
  ack-filter \
  split-gst \
  diffserv4 \
  nat \
  nowash

Parameters​

ParameterValuePurpose
bandwidthRateDownload/upload rate
overheadBytesEncapsulation overhead
mpuBytesMinimum packet unit
rttTimeRound-trip time estimate
ack-filter—Filter duplicate ACKs
split-gst—Split GST (get-set-test)
diffserv4—4-class QoS
nat—NAT detection
nowash—Disable wash

Overhead Calculation​

CAKE uses overhead to accurately calculate transmission time:

Transmission Time = (Packet Size + Overhead) / Bandwidth

Example (PPPoE):

  • Packet: 1500 bytes
  • Overhead: 34 bytes
  • Bandwidth: 100Mbit
  • Transmission Time = (1500 + 34) / 100Mbit = 122.72 µs

Ingress Shaping (Upload)​

IFB Device​

For upload rate limiting, abng-qos creates IFB (Intermediate Functional Block) device:

# Create IFB device
ip link add ifb-ppp0 type ifb

# Bring up IFB
ip link set ifb-ppp0 up

# Redirect ingress to IFB
tc filter add dev ppp0 parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev ifb-ppp0

# Apply CAKE to IFB
tc qdisc add dev ifb-ppp0 root cake bandwidth 40Mbit ...

Why IFB?​

Linux kernel doesn't support ingress qdisc directly. IFB provides a workaround:

  1. Redirect ingress traffic to IFB egress
  2. Apply qdisc to IFB egress
  3. Packets are then transmitted normally

Rate Changes (CoA)​

In-Place Update​

When RADIUS CoA changes rates, abng-qos uses tc qdisc change (not replace) to update bandwidth without packet loss:

# Old way (causes packet loss)
tc qdisc replace dev ppp0 root cake bandwidth 25Mbit ...

# New way (no packet loss)
tc qdisc change dev ppp0 root cake bandwidth 25Mbit ...

Process​

  1. abngd receives CoA-Request with new rates
  2. abngd updates session in database
  3. abngd notifies abng-qos via IPC
  4. abng-qos updates egress CAKE: tc qdisc change dev ppp0 ...
  5. abng-qos updates ingress CAKE: tc qdisc change dev ifb-ppp0 ...
  6. No packet loss during update

Session Cleanup​

When a session terminates:

  1. abngd notifies abng-qos of session removal
  2. abng-qos removes egress CAKE: tc qdisc del dev ppp0 root
  3. abng-qos removes ingress filter: tc filter del dev ppp0 parent ffff:
  4. abng-qos removes IFB device: ip link del ifb-ppp0

Overhead Profiles​

PPPoE over Ethernet​

pppoe_ethernet:
  overhead: 34
  mpu: 64

Breakdown:

  • Ethernet frame: 14 bytes (dst 6 + src 6 + type 2)
  • PPPoE header: 6 bytes (version/type 1 + code 1 + session 2)
  • PPP header: 2 bytes (protocol 2)
  • IP header: 20 bytes (minimum)
  • FCS: 4 bytes (frame check sequence)
  • Total: 46 bytes (rounded to 34)

IPoE over Ethernet​

ipoe_ethernet:
  overhead: 22
  mpu: 64

Breakdown:

  • Ethernet frame: 14 bytes
  • VLAN tag: 4 bytes
  • FCS: 4 bytes
  • Total: 22 bytes

PPPoE over QinQ​

pppoe_qinq:
  overhead: 38
  mpu: 64

Breakdown:

  • Ethernet: 14 bytes
  • Outer VLAN: 4 bytes
  • Inner VLAN: 4 bytes
  • PPPoE: 6 bytes
  • PPP: 2 bytes
  • IP: 20 bytes
  • FCS: 4 bytes
  • Total: 54 bytes (rounded to 38)

Monitoring​

View CAKE Stats​

tc -s qdisc show dev ppp0

Output includes:

  • Packets sent/dropped
  • Bytes sent
  • Marks (ECN marks)
  • Overlimit drops
  • Collisions

Check IFB Device​

ip link show | grep ifb
tc -s qdisc show dev ifb-ppp0

Monitor abng-qos Logs​

journalctl -u abng-qos -f

Performance Considerations​

CPU Usage​

  • CAKE is CPU-efficient (uses O(log n) operations)
  • Typical CPU usage: <1% per 1000 sessions
  • Overhead profile selection affects accuracy but not CPU

Memory Usage​

  • One qdisc per session
  • ~1KB per qdisc for kernel state
  • 10,000 sessions ≈ 10MB memory

Bandwidth Accuracy​

Accuracy depends on:

  • Correct overhead profile selection
  • Accurate MTU configuration
  • CAKE algorithm parameters (rtt, ack-filter)

Troubleshooting​

QoS Not Applying​

Check 1: QoS Enabled

show configuration | match qos

Check 2: CAKE Qdisc Exists

tc qdisc show dev ppp0

Check 3: Session Active

show subscribers

Check 4: abng-qos Logs

journalctl -u abng-qos -f

Rates Not Matching Expected​

Check 1: Overhead Profile

show configuration | match overhead_profiles

Check 2: Test with iperf

iperf -c <subscriber-ip> -t 10

Check 3: CAKE Stats

tc -s qdisc show dev ppp0

Check 4: Adjust Overhead If rates consistently off, adjust overhead profile.

IFB Ingress Shaping Not Working​

Check 1: IFB Device Exists

ip link show | grep ifb

Check 2: Ingress Filter

tc filter show dev ppp0 parent ffff:

Check 3: IFB Qdisc

tc qdisc show dev ifb-ppp0

Check 4: abng-qos Logs

journalctl -u abng-qos -f

Next Steps​