Monitoring fd.io VPP with Vigilmon

fd.io VPP (Vector Packet Processing) is an open source, user-space packet forwarding framework originally developed by Cisco and contributed to The Linux Foundation's fd.io project in 2016. VPP processes packets in vectorized batches through a directed acyclic graph (DAG) of processing nodes — achieving multi-million-packets-per-second rates that would be impossible with per-packet kernel network stack processing. If you're running VPP as a software router, NATaaS gateway, vRouter in an NFV environment, or as the data plane for Calico/VPP in Kubernetes, any silent failure means complete packet forwarding loss. Vigilmon gives you real-time monitoring of VPP's process health, DPDK interface rates, packet drop counters, and control-plane responsiveness without requiring code changes to your VPP deployment.

What You'll Set Up

• VPP daemon process health monitor via cron heartbeat
• DPDK interface input/output rate tracking
• Packet drop rate alerting (threshold: >0.1% sustained drops)
• VPP worker thread CPU utilization monitoring
• Graph node error counter change detection
• VPP heap memory usage tracking
• VPP API (vppctl) control-plane latency check
• Alert channels for network operations teams

Prerequisites

• fd.io VPP installed and running (VPP 23.x or later recommended)
• vppctl accessible on the host (or via socket at /run/vpp/cli.sock)
• A monitoring host or the VPP host itself with cron access
• A free Vigilmon account

Step 1: Monitor the VPP Daemon Process

VPP runs as a single daemon process (vpp). A crash stops all packet forwarding immediately — no keepalive, no graceful degradation. A cron heartbeat to Vigilmon is the most reliable way to catch a crashed VPP daemon before your network operations team notices via downstream failures.

First, create a heartbeat monitor in Vigilmon:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to Cron / Heartbeat.
3. Name it VPP Daemon Heartbeat.
4. Set Expected interval to 60 seconds.
5. Set Alert after missing to 2 heartbeats.
6. Copy the generated heartbeat URL.

Then add a cron job on the VPP host:

# /etc/cron.d/vpp-heartbeat
* * * * * root pgrep -x vpp > /dev/null && curl -fsS --retry 3 "https://hb.vigilmon.online/YOUR_HEARTBEAT_ID" > /dev/null 2>&1

The heartbeat fires only when vpp is found in the process list. If VPP crashes and stays down for 2 minutes, Vigilmon raises an alert.

Step 2: Monitor Packet Drop Rate

Packet drops are the primary VPP health signal — drops indicate the data plane is overloaded or misconfigured. VPP exposes drop counts via the vppctl show errors command. Export a script that calculates the drop rate and sends a metric heartbeat:

#!/bin/bash
# /usr/local/bin/vpp-drop-check.sh

THRESHOLD_PCT=0.1  # alert if drop% > 0.1%
HB_URL="https://hb.vigilmon.online/YOUR_DROP_HB_ID"

# Get total and drop packet counts from VPP runtime stats
RUNTIME=$(vppctl show runtime 2>/dev/null)
if [ $? -ne 0 ]; then
  # VPP not responding — don't send heartbeat (triggers alert)
  exit 1
fi

ERRORS=$(vppctl show errors 2>/dev/null | grep -E "^[[:space:]]+[0-9]" | awk '{sum += $1} END {print sum+0}')
PACKETS=$(vppctl show runtime 2>/dev/null | grep "vectors:" | awk '{sum += $2} END {print sum+0}')

if [ "$PACKETS" -gt 0 ]; then
  DROP_PCT=$(awk "BEGIN {printf \"%.4f\", ($ERRORS / $PACKETS) * 100}")
  # Only send heartbeat if drop rate is acceptable
  OVER=$(awk "BEGIN {print ($DROP_PCT > $THRESHOLD_PCT) ? 1 : 0}")
  if [ "$OVER" -eq 0 ]; then
    curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
  fi
else
  # No traffic yet — send heartbeat (not a failure condition)
  curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
fi

Create a second heartbeat monitor in Vigilmon named VPP Drop Rate OK with a 60-second interval, then schedule the script:

* * * * * root /usr/local/bin/vpp-drop-check.sh

When drops exceed 0.1% of total traffic for two consecutive minutes, the heartbeat stops and Vigilmon alerts.

Step 3: Monitor DPDK Interface Rates

VPP uses DPDK for high-performance NIC access, bypassing the Linux kernel. A sudden drop in DPDK interface input rate means upstream traffic has stopped reaching the NIC — which could indicate a physical link failure, upstream router issue, or DPDK driver crash.

Export DPDK interface stats and send them to a Vigilmon heartbeat:

#!/bin/bash
# /usr/local/bin/vpp-dpdk-check.sh
# Verify DPDK interfaces are receiving traffic above threshold

MIN_RX_PPS=100  # alert if RX drops below 100 PPS (adjust for your traffic baseline)
HB_URL="https://hb.vigilmon.online/YOUR_DPDK_HB_ID"

INTERFACES=$(vppctl show interface 2>/dev/null | grep "^[A-Za-z]" | awk '{print $1}')
if [ -z "$INTERFACES" ]; then
  exit 1
fi

TOTAL_RX=0
for IFACE in $INTERFACES; do
  # Only check DPDK interfaces
  echo "$IFACE" | grep -qE "^(GigabitEthernet|TenGigabitEthernet|dpdk)" || continue
  RX=$(vppctl show interface "$IFACE" 2>/dev/null | grep "rx packets" | awk '{print $3}')
  TOTAL_RX=$((TOTAL_RX + ${RX:-0}))
done

# Send heartbeat only if receiving traffic
if [ "$TOTAL_RX" -ge "$MIN_RX_PPS" ]; then
  curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
fi

Create a heartbeat monitor named VPP DPDK Interface Active and schedule this script every minute. Adjust MIN_RX_PPS to a value representing your expected minimum baseline traffic.

Step 4: Monitor VPP Worker Thread CPU

VPP dedicates CPU cores to worker threads for packet processing. If any worker thread exceeds 90% CPU sustained for more than 60 seconds, VPP is approaching its throughput limit and packet drops will soon follow.

#!/bin/bash
# /usr/local/bin/vpp-cpu-check.sh

CPU_THRESHOLD=90
HB_URL="https://hb.vigilmon.online/YOUR_CPU_HB_ID"

RUNTIME=$(vppctl show runtime verbose 2>/dev/null)
if [ $? -ne 0 ]; then
  exit 1
fi

# Check worker thread CPU — VPP worker threads appear as "vpp_wk_N" in ps
OVER_THRESHOLD=0
while IFS= read -r line; do
  CPU=$(echo "$line" | awk '{print $3}' | cut -d. -f1)
  if [ "${CPU:-0}" -ge "$CPU_THRESHOLD" ]; then
    OVER_THRESHOLD=1
    break
  fi
done < <(ps -eo comm,pid,pcpu | grep "vpp_wk_")

if [ "$OVER_THRESHOLD" -eq 0 ]; then
  curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
fi

Schedule this script every minute as a cron job. Create the corresponding heartbeat monitor VPP Worker CPU OK in Vigilmon.

Step 5: Monitor VPP Graph Node Errors

VPP's graph node architecture means each processing node (ip4-input, ethernet-input, ip4-lookup) tracks its own error counters. New error types appearing in vppctl show errors indicate a new failure mode — misrouted packets, checksum failures, TTL-exceeded floods, or ARP resolution failures.

#!/bin/bash
# /usr/local/bin/vpp-node-errors-check.sh
# Baseline: track total error count; alert if it grows significantly

STATE_FILE="/tmp/vpp_errors_last"
HB_URL="https://hb.vigilmon.online/YOUR_NODE_HB_ID"
GROWTH_THRESHOLD=1000  # alert if errors grew by >1000 since last check

CURRENT_ERRORS=$(vppctl show errors 2>/dev/null | grep -E "^[[:space:]]+[0-9]" | awk '{sum += $1} END {print sum+0}')

if [ ! -f "$STATE_FILE" ]; then
  echo "$CURRENT_ERRORS" > "$STATE_FILE"
  curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
  exit 0
fi

LAST_ERRORS=$(cat "$STATE_FILE")
DELTA=$((CURRENT_ERRORS - LAST_ERRORS))
echo "$CURRENT_ERRORS" > "$STATE_FILE"

if [ "$DELTA" -lt "$GROWTH_THRESHOLD" ]; then
  curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
fi

Create a heartbeat VPP Node Errors Stable and run this script every minute.

Step 6: Monitor VPP API Latency (Control Plane)

The VPP API (accessible via vppctl) handles configuration updates. High API latency (>500ms) means the VPP control plane is under load or stuck — configuration changes are not taking effect, and VPP graph topology cannot be updated.

#!/bin/bash
# /usr/local/bin/vpp-api-latency-check.sh

LATENCY_THRESHOLD_MS=500
HB_URL="https://hb.vigilmon.online/YOUR_API_HB_ID"

START=$(date +%s%3N)
vppctl show version > /dev/null 2>&1
STATUS=$?
END=$(date +%s%3N)

LATENCY=$((END - START))

if [ "$STATUS" -eq 0 ] && [ "$LATENCY" -lt "$LATENCY_THRESHOLD_MS" ]; then
  curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
fi

Create a heartbeat VPP API Latency OK in Vigilmon with a 60-second interval and schedule this script.

Step 7: Monitor VPP Heap Memory

VPP allocates large memory heaps for packet buffers using huge pages. If the heap fills above 90%, VPP will start dropping packets due to buffer exhaustion rather than processing overload.

#!/bin/bash
# /usr/local/bin/vpp-memory-check.sh

HEAP_THRESHOLD_PCT=90
HB_URL="https://hb.vigilmon.online/YOUR_MEM_HB_ID"

MEM_INFO=$(vppctl show memory verbose 2>/dev/null | grep -i "heap" | head -1)
USED=$(echo "$MEM_INFO" | grep -oP 'used\s+\K[0-9]+')
TOTAL=$(echo "$MEM_INFO" | grep -oP 'total\s+\K[0-9]+')

if [ -n "$USED" ] && [ -n "$TOTAL" ] && [ "$TOTAL" -gt 0 ]; then
  USED_PCT=$(awk "BEGIN {printf \"%d\", ($USED / $TOTAL) * 100}")
  if [ "$USED_PCT" -lt "$HEAP_THRESHOLD_PCT" ]; then
    curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
  fi
else
  # Can't read memory stats — still send heartbeat if VPP is alive
  curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
fi

Step 8: Configure Alert Channels

In Vigilmon, navigate to Settings → Alert Channels and configure the channels appropriate for a networking data-plane failure:

1. Slack / Teams webhook — for your network operations channel. VPP failures are often urgent: packet forwarding stops instantly on a VPP crash.
2. PagerDuty or on-call webhook — VPP daemon heartbeat and drop rate monitors should page on-call. A packet forwarding failure is a P1 incident.
3. Email — for non-urgent monitors like node error growth or memory headroom.

Assign alert channels per monitor:

• VPP Daemon Heartbeat → PagerDuty + Slack
• VPP Drop Rate OK → PagerDuty + Slack (sustained drops are a P1)
• VPP DPDK Interface Active → PagerDuty + Slack
• VPP Worker CPU OK → Slack only (leading indicator, not yet a failure)
• VPP Node Errors Stable → Slack only
• VPP API Latency OK → Slack only
• VPP Heap Memory OK → Slack only

Step 9: Set Alert Thresholds and Timing

For each heartbeat monitor, set the alert timing to match VPP's failure profile:

| Monitor | Interval | Alert after | |---|---|---| | VPP Daemon Heartbeat | 60s | 2 missed | | VPP Drop Rate OK | 60s | 2 missed | | VPP DPDK Interface Active | 60s | 3 missed | | VPP Worker CPU OK | 60s | 3 missed | | VPP Node Errors Stable | 60s | 3 missed | | VPP API Latency OK | 60s | 2 missed | | VPP Heap Memory OK | 60s | 5 missed |

The daemon and drop rate monitors alert quickly (2 missed = 2 minutes) because a VPP crash or high drop rate means packets are being lost right now. Memory and node error monitors have more tolerance since they represent leading indicators.

Conclusion

fd.io VPP's user-space, vectorized architecture delivers extraordinary packet processing performance — but that performance comes with a monitoring obligation. Unlike kernel network stacks that degrade gracefully, a VPP process crash or DPDK interface failure stops packet forwarding completely and instantly. The Vigilmon setup in this guide covers VPP's critical failure modes: daemon crashes via heartbeat, packet drops via threshold-based heartbeat suppression, DPDK interface rate drops, worker thread CPU saturation, and control-plane API latency. With these monitors in place, your network operations team gets a 1–2 minute heads-up on any VPP failure — before downstream services start timing out and users start filing tickets.

Get started at vigilmon.online.