Monitoring

Monitoring AthenaBNG services and performance.

Overview

AthenaBNG provides multiple monitoring capabilities:

• CLI Monitoring — Real-time session events
• Prometheus Metrics — Time-series metrics
• SNMP — NOC integration
• Syslog — Centralized logging
• Service Logs — systemd journal

CLI Monitoring

Real-Time Session Monitor

abng> monitor subscribers

Displays live session events:

• session_up — Session established
• session_down — Session terminated
• rate_change — Rate changed via CoA

Session Queries

# Get session count
abng> show subscribers summary

# Get QoS coverage
abng> show qos | count

# Get interface list
abng> show interfaces

Prometheus Metrics

Scrape Configuration

Add to Prometheus prometheus.yml:

scrape_configs:
  - job_name: 'athenabng'
    static_configs:
      - targets: ['localhost:9100']
    scrape_interval: 15s

Key Metrics

Metric	Description
athenabng_sessions_total	Total active sessions
athenabng_sessions_pppoe	PPPoE sessions
athenabng_sessions_dhcp	DHCP sessions
athenabng_qos_sessions	Sessions with QoS
athenabng_vlan_interfaces	Dynamic VLAN interfaces
athenabng_uptime_seconds	System uptime

Data-Plane /metrics Endpoints

The abng-demux (port 9101) and abng-qos (port 9102) daemons expose their own Prometheus /metrics endpoints. By default these bind inside the management interface's VRF (controlled by monitoring.prometheus.bind_address and monitoring.prometheus.vrf), so scrape from inside that VRF:

# abng-qos data-plane metrics
ip vrf exec mgmt curl http://<mgmt-ip>:9102/metrics

# abng-demux data-plane metrics
ip vrf exec mgmt curl http://<mgmt-ip>:9101/metrics

Key metrics: abng_demux_interfaces_active, abng_demux_creation_failures_total, abng_demux_capture_ring_drops_total (demux); abng_qos_sessions_shaped, abng_qos_qdisc_apply_failures_total, abng_qos_ifb_active (qos). A bind failure is non-fatal. See Monitoring Configuration for the full list.

Operating the Monitoring Stack

The on-box stack (Prometheus, Alertmanager, frr_exporter, node-exporter) runs entirely inside the management VRF. See Monitoring Configuration for how to deploy/update it with contrib/monitoring/install.sh.

Viewing the UIs

The Prometheus and Alertmanager UIs are not exposed on the firewall and are unauthenticated — do not open them up. Reach them over an SSH tunnel to the mgmt IP:

ssh -L 9090:<mgmt-ip>:9090 -L 9093:<mgmt-ip>:9093 root@<box>
# then, locally:
#   http://localhost:9090   Prometheus (targets, graphs, alerts)
#   http://localhost:9093   Alertmanager (firing/silenced alerts)

Useful first checks: Status → Targets in Prometheus should show all five jobs (prometheus, node, frr, abng-demux, abng-qos) as UP; the Alerts tab shows rule state. A quick liveness query for BGP:

frr_bgp_peer_state          # 1 = Established, 0 = Down, 2 = AdminDown
frr_collector_up{collector="bgp"}   # 0 = exporter can't read bgpd/zebra

Alert Rules

Prometheus evaluates contrib/monitoring/athenabng-rules.yml (installed to /etc/prometheus/rules/athenabng.yml):

Alert	Severity	Fires when
ScrapeTargetDown	critical	Any scrape target up == 0 for 2m
DaemonNotActive	critical	A core unit (abngd/demux/qos/dhcpd/api/snmp/frr) is not active for 2m
BgpPeerDown	critical	A BGP peer is not Established for 3m
BgpFlapping	warning	A BGP peer resets more than twice in 15m
FrrBgpCollectorDown	warning	frr_exporter can't read bgpd/zebra for 2m
ClockUnsynchronised	warning	node_timex_sync_status == 0 for 10m
FrrRestartLooping	warning	frr.service restart-looping (watchfrr storm)
DiskFilling	warning	Root filesystem below 10% free for 5m
Watchdog	none	Always firing — route to an external dead-man's-switch

The Watchdog alert exists so its absence pages you: if the whole box (or Prometheus itself) dies, the external dead-man's-switch stops receiving the heartbeat and fires. After editing the rules, reload Prometheus:

ip vrf exec mgmt curl -X POST http://<mgmt-ip>:9090/-/reload
# or: systemctl reload prometheus

Wiring a Real Notifier

Alertmanager ships with a placeholder receiver — the plumbing is in place but no notifier is wired, so alerts currently go nowhere. To wire one, edit /etc/prometheus/alertmanager.yml on the box:

1. Add a webhook_configs (or email_configs) block to the default receiver.
2. Add a dedicated route + receiver for the Watchdog alert (matchers: [ severity = "none" ]) pointing at a dead-man's-switch URL (e.g. Healthchecks.io / Dead Man's Snitch), with a short repeat_interval.
3. Reload: systemctl reload prometheus-alertmanager.

route:
  receiver: default
  routes:
    - matchers: [ severity = "none" ]   # Watchdog
      receiver: deadmanssnitch
      repeat_interval: 5m
receivers:
  - name: default
    webhook_configs: [{ url: 'https://hooks.slack.com/services/...' }]
  - name: deadmanssnitch
    webhook_configs: [{ url: 'https://hc-ping.com/<uuid>' }]

Daemon-Failure Heartbeat

Separately from Prometheus, an OnFailure=heartbeat-failed@<unit> systemd hook on FRR (and optionally the AthenaBNG daemons) makes a daemon crash detectable even before the stack notices: it logs at daemon.alert priority and drops a /run/athenabng/failed.<unit> marker file. Watch for markers:

ls /run/athenabng/failed.* 2>/dev/null
journalctl -t athenabng-alert

SNMP Monitoring

Query Session Count

snmpget -v2c -c my-community localhost 1.3.6.1.4.1.65245.2.1.0

Check Service Health

# Walk all daemon health checks (TruthValue: 1=up, 2=down)
snmpwalk -v2c -c my-community localhost 1.3.6.1.4.1.65245.6

Walk All Metrics

snmpwalk -v2c -c my-community localhost 1.3.6.1.4.1.65245

Syslog Forwarding

Configuration

monitoring:
  syslog:
    enabled: true
    server: "10.255.0.20"
    port: 514
    protocol: "udp"

Verify Forwarding

sudo journalctl -u abngd | grep "syslog"

Service Logs

View All Logs

sudo journalctl -u AthenaBNG.target -f

View Specific Service

sudo journalctl -u abngd -f
sudo journalctl -u abng-demux -f
sudo journalctl -u accel-ppp -f

Filter by Level

# Errors only
sudo journalctl -u abngd -p err

# Warnings and errors
sudo journalctl -u abngd -p warning

Health Checks

Check Service Status

sudo systemctl status AthenaBNG.target

Check Socket Connectivity

echo '{"jsonrpc":"2.0","method":"show.system","params":{},"id":1}' | \
  nc -U /opt/athena-bng/run/abngd.sock

Check Session Count

abng -c "show subscribers" | wc -l

Alerting

Session Count Alert

Alert when sessions exceed threshold:

- alert: HighSessionCount
  expr: athenabng_sessions_total > 5000
  for: 5m

Service Down Alert

Alert when service is down:

- alert: ServiceDown
  expr: up{job="athenabng"} == 0
  for: 1m

Next Steps

• Troubleshooting — Common issues and solutions
• Performance Tuning — Optimization tips