Monitoring ONOS with Vigilmon

ONOS (Open Network Operating System) is an open source SDN network operating system developed by the Open Networking Foundation (ONF) and launched in 2014, purpose-built for carrier-grade, high-availability deployments. Unlike traditional network management, ONOS expresses network behavior through intents — your application says "connect host A to host B" and ONOS compiles that intent into OpenFlow or P4Runtime rules across the fabric. ONOS runs as a distributed cluster using Apache Atomix (Raft consensus) for coordination, and it's used in production by AT&T, SK Telecom, NTT, and China Unicom. When ONOS degrades — an Atomix leader election stalls, an intent gets stuck in FAILED state, or cluster membership drops below quorum — your SDN fabric stops responding to network events. Vigilmon gives you proactive visibility into every layer of the ONOS stack before your carrier network feels the impact.

What You'll Set Up

• ONOS HTTP management API uptime monitor
• Cluster node membership and quorum health check
• Device connectivity count monitor (OpenFlow / P4Runtime)
• Intent compilation health monitor (pending and failed intents)
• Atomix distributed store leader health check
• Northbound REST API latency monitor
• Segment routing fabric health check
• ONOS JVM heap cron heartbeat
• Alert channels with appropriate thresholds

Prerequisites

• ONOS running as a single node or cluster (Karaf-based, ONOS 2.x or later)
• ONOS REST API accessible (default port: 8181)
• onos CLI or REST credentials available
• A free Vigilmon account

Step 1: Monitor the ONOS Management API

The ONOS REST API is the primary signal for controller health. If it stops responding, ONOS is down and your SDN fabric is running blind.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. URL: http://your-onos-host:8181/onos/v1/info
4. Set Method to GET.
5. Under Basic Auth, enter onos / rocks (the default credentials — change these in production).
6. Set Expected HTTP status to 200.
7. Set Check interval to 1 minute.
8. Under Keyword check, enter version to confirm the ONOS info response body is present.
9. Click Save.

This catches ONOS crashes, Karaf startup failures, and REST API bundle failures.

Step 2: Monitor Cluster Node Membership

ONOS is designed as a distributed system — losing nodes below quorum causes write failures across the cluster. Monitor the cluster membership endpoint to detect node departures.

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://your-onos-host:8181/onos/v1/cluster
3. Method: GET
4. Basic Auth: ONOS credentials
5. Expected HTTP status: 200
6. Under Keyword check, enter READY to verify at least one node reports as ready.
7. Check interval: 2 minutes
8. Click Save.

For a three-node ONOS cluster, you need at least two nodes READY for quorum. Add this monitor on each ONOS node IP so you catch per-node failures:

• http://onos-node-1:8181/onos/v1/cluster
• http://onos-node-2:8181/onos/v1/cluster
• http://onos-node-3:8181/onos/v1/cluster

Step 3: Monitor Device Connectivity

ONOS manages OpenFlow switches, P4Runtime devices, and NETCONF devices. The device count dropping signals switches disconnecting from SDN control.

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://your-onos-host:8181/onos/v1/devices
3. Method: GET
4. Basic Auth: ONOS credentials
5. Expected HTTP status: 200
6. Under Keyword check, enter AVAILABLE to verify at least one device shows as available.
7. Check interval: 2 minutes
8. Click Save.

Also monitor the OpenFlow southbound listening port:

1. Click Add Monitor → TCP Port.
2. Host: your-onos-host
3. Port: 6653 (OpenFlow 1.3)
4. Check interval: 1 minute
5. Click Save.

Step 4: Monitor Intent Compilation Health

Intents are the core of ONOS's northbound model. When intents get stuck in INSTALLING or FAILED state, connectivity requirements from your applications are not being realized as flow rules.

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://your-onos-host:8181/onos/v1/intents
3. Method: GET
4. Basic Auth: ONOS credentials
5. Expected HTTP status: 200
6. Under Keyword check, enter INSTALLED to verify intents are compiling successfully.
7. Check interval: 3 minutes
8. Click Save.

For deeper intent monitoring, use a cron heartbeat script that counts FAILED intents:

1. Click Add Monitor → Cron Heartbeat.
2. Set expected interval to 5 minutes.
3. Copy the heartbeat URL.
4. On your ONOS server, create a script:

#!/bin/bash
# /usr/local/bin/onos-intent-check.sh

ONOS_URL="http://localhost:8181"
ONOS_CREDS="onos:rocks"
HEARTBEAT_URL="https://vigilmon.online/heartbeat/your-intent-heartbeat-id"
MAX_FAILED=5

FAILED_COUNT=$(curl -sf -u "$ONOS_CREDS" \
  "$ONOS_URL/onos/v1/intents" \
  | python3 -c "
import sys, json
data = json.load(sys.stdin)
intents = data.get('intents', [])
failed = sum(1 for i in intents if i.get('state') in ['FAILED', 'INSTALL_REQ'])
print(failed)
" 2>/dev/null)

if [ -z "$FAILED_COUNT" ] || [ "$FAILED_COUNT" -le "$MAX_FAILED" ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
fi
# More than MAX_FAILED intents stuck: don't ping — Vigilmon alerts

chmod +x /usr/local/bin/onos-intent-check.sh

Add to crontab:

*/5 * * * * /usr/local/bin/onos-intent-check.sh

Step 5: Monitor Atomix Distributed Store Health

ONOS uses Apache Atomix for distributed state replication. If the Atomix leader is unreachable or Raft log replication lags, cluster writes fail.

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://your-onos-host:5678/v1/primitives (Atomix REST API, if enabled)
3. Method: GET
4. Expected HTTP status: 200
5. Check interval: 2 minutes
6. Click Save.

For Atomix management port availability:

1. Click Add Monitor → TCP Port.
2. Host: your-onos-host
3. Port: 5679 (Atomix cluster communication port)
4. Check interval: 1 minute
5. Click Save.

Step 6: Monitor the Topology and Links

ONOS maintains a real-time topology model. Monitoring the topology endpoint confirms that ONOS is processing link events and maintaining an accurate network view.

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://your-onos-host:8181/onos/v1/topology
3. Method: GET
4. Basic Auth: ONOS credentials
5. Expected HTTP status: 200
6. Under Keyword check, enter devices to verify topology data is present.
7. Check interval: 2 minutes
8. Click Save.

Also monitor the links endpoint to detect link topology loss:

1. URL: http://your-onos-host:8181/onos/v1/links
2. Expected HTTP status: 200
3. Check interval: 3 minutes
4. Click Save.

Step 7: Monitor Segment Routing Fabric Health

If you use ONOS with the Trellis segment routing application (leaf-spine fabric), monitor the route programming endpoint:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://your-onos-host:8181/onos/v1/routes
3. Method: GET
4. Basic Auth: ONOS credentials
5. Expected HTTP status: 200
6. Under Keyword check, enter prefix to verify route prefixes are present.
7. Check interval: 5 minutes
8. Click Save.

For ECMP group health in a Trellis fabric:

1. URL: http://your-onos-host:8181/onos/v1/groups
2. Expected HTTP status: 200
3. Under Keyword check, enter SELECT to verify ECMP select groups exist.
4. Check interval: 5 minutes
5. Click Save.

Step 8: Monitor P4Runtime Device Programming

If you use ONOS with P4Runtime-enabled devices (Barefoot Tofino, BMv2 software switches), monitor the pipeline push health:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://your-onos-host:8181/onos/v1/devices (filter for P4Runtime devices)
3. Method: GET
4. Basic Auth: ONOS credentials
5. Expected HTTP status: 200
6. Under Keyword check, enter p4runtime to verify P4Runtime devices are listed.
7. Check interval: 5 minutes
8. Click Save.

Step 9: Monitor ONOS JVM Heap via Cron Heartbeat

Like OpenDaylight, ONOS runs on a Karaf JVM and is susceptible to heap exhaustion under heavy topology event loads.

1. Click Add Monitor → Cron Heartbeat.
2. Set expected interval to 5 minutes.
3. Copy the heartbeat URL.
4. Create a monitoring script on your ONOS server:

#!/bin/bash
# /usr/local/bin/onos-heap-check.sh

ONOS_URL="http://localhost:8181"
ONOS_CREDS="onos:rocks"
HEARTBEAT_URL="https://vigilmon.online/heartbeat/your-heap-heartbeat-id"
HEAP_THRESHOLD=85

# Check ONOS is responding first
HTTP_STATUS=$(curl -sf -o /dev/null -w "%{http_code}" \
  -u "$ONOS_CREDS" "$ONOS_URL/onos/v1/info")

if [ "$HTTP_STATUS" != "200" ]; then
  # ONOS is not responding — don't ping heartbeat
  exit 0
fi

# Check heap via JMX/Jolokia if available
HEAP_PCT=$(curl -sf -u "$ONOS_CREDS" \
  "$ONOS_URL/jolokia/read/java.lang:type=Memory/HeapMemoryUsage" \
  | python3 -c "
import sys, json
d = json.load(sys.stdin)
used = d['value']['used']
mx = d['value']['max']
print(int(used * 100 / mx))
" 2>/dev/null)

if [ -z "$HEAP_PCT" ] || [ "$HEAP_PCT" -lt "$HEAP_THRESHOLD" ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
fi

chmod +x /usr/local/bin/onos-heap-check.sh

Add to crontab:

*/5 * * * * /usr/local/bin/onos-heap-check.sh

Step 10: Configure Alert Channels and Thresholds

1. Go to Alert Channels in Vigilmon and configure Slack, email, or PagerDuty.
2. For the ONOS management API monitor, set Consecutive failures before alert to 2 — ONOS cluster leader re-election can cause brief HTTP unavailability.
3. For cluster membership monitors, set Consecutive failures to 1 — node departure from quorum is immediately critical.
4. For device connectivity and intent monitors, set Consecutive failures to 2.
5. For OpenFlow TCP port and Atomix TCP monitors, set Consecutive failures to 1.
6. For JVM heap and intent health heartbeats, Vigilmon alerts automatically after the expected window passes.
7. Enable Recovery notifications on all monitors.

Conclusion

You now have comprehensive monitoring across ONOS's distributed control plane: the management API confirms the controller is running, cluster membership monitors catch quorum loss before it causes write failures, device connectivity and intent monitors detect fabric programming issues, and Atomix store health checks guard against distributed coordination failures. The JVM heap heartbeat gives early warning before memory pressure crashes the carrier-grade controller. For more self-hosted infrastructure monitoring guides, see vigilmon.online.