Monitoring

Real-time monitoring and observability features.

Overview

AthenaBNG provides multiple monitoring capabilities:

• CLI Monitoring — Real-time session events via CLI
• Prometheus Metrics — Time-series metrics for Grafana
• On-box monitoring stack — Prometheus + Alertmanager + frr_exporter, all in the management VRF
• SNMP — Integration with NOC monitoring systems
• Syslog — Centralized log forwarding
• REST API — Programmatic access to metrics

CLI Monitoring

Real-Time Session Monitor

abng> monitor subscribers

Displays events:

• session_up — Session established
• session_down — Session terminated
• rate_change — Rate changed via CoA
• error — Session error

Session Queries

# Get session counts
abng> show subscribers summary

# Get QoS coverage
abng> show qos | count

# Get CGNAT sessions
abng> show configuration | match cgnat

Prometheus Metrics

Session Metrics

Metric	Description
athenabng_sessions_total	Total active sessions
athenabng_sessions_pppoe	PPPoE sessions
athenabng_sessions_dhcp	DHCP sessions
athenabng_sessions_created_total	Total sessions created
athenabng_sessions_terminated_total	Total sessions terminated

QoS Metrics

Metric	Description
athenabng_qos_sessions	Sessions with QoS applied
athenabng_qos_qdiscs	Active CAKE qdiscs

CGNAT Metrics

Metric	Description
athenabng_cgnat_sessions	CGNAT sessions
athenabng_cgnat_pool_utilization	CGNAT pool utilization %

Demux Metrics

Metric	Description
athenabng_vlan_interfaces	Dynamic VLAN interfaces
athenabng_vlan_creation_rate	VLAN creation rate

System Metrics

Metric	Description
athenabng_uptime_seconds	System uptime
athenabng_version_info	Version information

Data-Plane /metrics Endpoints

The abng-demux and abng-qos data-plane daemons expose their own Prometheus /metrics endpoints (plain-text exposition over a hand-rolled HTTP responder, no extra dependencies). A bind failure is non-fatal — the daemon keeps running without metrics.

Daemon	Port	Metrics
abng-demux	9101	abng_demux_interfaces_created_total, abng_demux_interfaces_removed_total, abng_demux_creation_rate_limited_total, abng_demux_creation_failures_total, abng_demux_capture_ring_drops_total (counters); abng_demux_interfaces_active (gauge)
abng-qos	9102	abng_qos_qdisc_apply_total, abng_qos_qdisc_apply_failures_total (counters); abng_qos_sessions_shaped, abng_qos_ifb_active (gauges)

These endpoints honour monitoring.prometheus.bind_address (default 0.0.0.0) and monitoring.prometheus.vrf. When vrf is unset they bind inside the management interface's VRF via SO_BINDTODEVICE, so they are reachable only inside that VRF (not on loopback). Scrape from inside the VRF:

ip vrf exec mgmt curl http://<mgmt-ip>:9102/metrics

Set monitoring.prometheus.vrf to an empty string to bind in the default VRF instead. See Monitoring Configuration for details.

On-Box Monitoring Stack

AthenaBNG ships a complete, self-contained monitoring stack that scrapes the local endpoints, evaluates alert rules, and routes alerts — closing the gap that previously let a dead zebra / flapping BGP session go unnoticed for weeks because no metric for BGP or per-daemon liveness existed anywhere.

Every component runs inside the management VRF (ip vrf exec mgmt). This is mandatory, not incidental: the AthenaBNG metrics endpoints bind to the mgmt VRF (SO_BINDTODEVICE), so a scraper in the default VRF cannot reach them. All scrape targets are addressed by the mgmt0 IP (not loopback).

Component	Port (mgmt VRF)	Role
Prometheus	9090	Scrapes every target below; evaluates the alert rules
Alertmanager	9093	Routes/dedupes alerts (receiver ships as a placeholder)
frr_exporter	9342	BGP peer state + per-FRR-daemon liveness
node-exporter	9100	Host metrics + node_systemd_unit_state (per-unit daemon liveness)
abng-demux	9101	Data-plane demux counters
abng-qos	9102	Data-plane QoS counters

Deployment (the idempotent contrib/monitoring/install.sh installer) is covered in Monitoring Configuration; operating the stack (viewing UIs, wiring notifiers, the alert rules) is covered in the admin Monitoring guide.

frr_exporter — BGP and daemon liveness

frr_exporter (a GitHub-release binary, run under ip vrf exec mgmt) is the key addition. It talks to the local FRR daemons and exposes:

Metric	Meaning
frr_bgp_peer_state	BGP peer state (1 = Established, 0 = Down, 2 = AdminDown)
frr_bgp_peer_uptime_seconds	Peer uptime — resets on every flap (used to detect flapping)
frr_collector_up{collector="bgp"}	Whether the exporter can read bgpd/zebra at all

Before this, BGP session state and FRR daemon health were invisible to metrics by design. node-exporter's systemd collector complements it with per-unit state (node_systemd_unit_state{name="...",state="active"}) for daemon-down alerts and node_systemd_unit_restarts_total for restart-storm detection.

Alerting

Prometheus evaluates the rules in contrib/monitoring/athenabng-rules.yml, each targeting a failure mode that has actually bitten production:

Alert	Fires when
ScrapeTargetDown	Any scrape target (up == 0) is unreachable for 2m
DaemonNotActive	A core unit (abngd/demux/qos/dhcpd/api/snmp/frr) is not active
BgpPeerDown	A BGP peer is not Established for 3m
BgpFlapping	A BGP peer resets more than twice in 15m
FrrBgpCollectorDown	frr_exporter can't read bgpd/zebra (e.g. zebra down)
ClockUnsynchronised	Clock not NTP-synced for 10m (corrupts accounting timestamps)
FrrRestartLooping	frr.service is restart-looping (watchfrr storm)
DiskFilling	Root filesystem below 10% free
Watchdog	Always firing — route to an external dead-man's-switch so its absence pages on total box loss

Alertmanager ships with a placeholder receiver (no notifier wired). Wiring a real webhook/email plus the dead-man's-switch route for Watchdog is documented in the admin guide.

Daemon-failure heartbeat

Independent of Prometheus, an OnFailure=heartbeat-failed@<unit> systemd hook on FRR (and, optionally, the AthenaBNG daemons) makes any daemon failure loud: it logs at daemon.alert priority and drops a /run/athenabng/failed.<unit> marker file that a health check or dead-man's-switch can watch. Without this unit those OnFailure hooks would be silent no-ops.

SNMP Monitoring

ATHENA-BNG-MIB

28 metrics under enterprise OID 1.3.6.1.4.1.65245 across 6 groups:

Group	OID Branch	Key Metrics
System	.1	Version, uptime, hostname
Subscribers	.2	Session counts, traffic counters (real per-session bytes/packets), provisioned bandwidth
QoS	.3	CAKE sessions, IFB devices, pending QoS
CGNAT	.4	CGNAT vs public session split
Demux	.5	Dynamic VLANs, unique access interfaces
Services	.6	Daemon health checks (abngd, demux, accel-ppp, qos, dhcpd, API)

SNMP Queries

# Get session count
snmpget -v2c -c my-community localhost 1.3.6.1.4.1.65245.2.1.0

# Check service health
snmpwalk -v2c -c my-community localhost 1.3.6.1.4.1.65245.6

# Walk all metrics
snmpwalk -v2c -c my-community localhost 1.3.6.1.4.1.65245

Per-Session Traffic Counters

Per-session traffic counters are now populated, so the Athena MIB's aggregate subscriber byte totals (e.g. total_bytes_in / total_bytes_out) reflect real usage. The show subscribers data now includes per-session bytes and packets (in and out), which the SNMP agent aggregates into the subscriber group totals.

Syslog Forwarding

Forward logs to centralized syslog server:

monitoring:
  syslog:
    enabled: true
    server: "10.255.0.20"
    port: 514
    protocol: "udp"

REST API Monitoring

Health Check

curl -k https://localhost:8443/api/health

Get Metrics

curl -k https://localhost:8443/api/metrics

Get Subscriber List

curl -k -H "Authorization: Bearer $TOKEN" \
  https://localhost:8443/api/subscribers

Grafana Dashboard

Pre-built dashboard available at:

grafana/athenabng-dashboard.json

Import into Grafana and select Prometheus data source.

Alerting

The shipped stack already includes infrastructure alert rules (see On-Box Monitoring Stack above and the admin guide). The examples below are service-level rules you can layer on top for capacity/quality signals.

Session Count Alert

Alert when session count exceeds threshold:

- alert: HighSessionCount
  expr: athenabng_sessions_total > 5000
  for: 5m
  annotations:
    summary: "High session count: {{ $value }}"

QoS Coverage Alert

Alert when QoS coverage is low:

- alert: LowQoSCoverage
  expr: athenabng_qos_sessions / athenabng_sessions_total < 0.95
  for: 5m
  annotations:
    summary: "Low QoS coverage: {{ $value }}"

Best Practices

1. Prometheus — Scrape every 15-30 seconds
2. Retention — Keep 15 days of Prometheus data
3. Syslog — Use TCP for reliable delivery
4. Alerts — Set thresholds for session count and resource usage
5. Dashboards — Create custom Grafana dashboards
6. Archival — Archive logs for compliance
7. Security — Restrict access to monitoring endpoints

Troubleshooting

Prometheus Metrics Not Available

1. Check Prometheus is enabled: show configuration | match prometheus
2. Check port is listening: sudo netstat -tlnp | grep 9100
3. Check firewall allows port 9100
4. Query metrics: curl http://localhost:9100/metrics

Syslog Not Forwarding

1. Check syslog is enabled: show configuration | match syslog
2. Check server is reachable: ping 10.255.0.20
3. Check port is open: nc -u 10.255.0.20 514
4. Check abngd logs: journalctl -u abngd -f

Next Steps

• Configuration Overview — All configuration sections
• Architecture Overview — System design