tutorial

How to Monitor Security Onion Health with Vigilmon

Security Onion Elasticsearch cluster failures, Suricata ingestion drops, and Zeek log gaps cause SOC blind spots that are invisible until analysts try to investigate an active incident and find no data. Learn how to monitor Security Onion cluster health, IDS alert ingestion, and sensor connectivity with Vigilmon HTTP probes and heartbeat monitors.

Security Onion is the open source Linux distribution that powers enterprise security operations centers — bundling Suricata IDS, Zeek network analysis, Elasticsearch log storage, and the Security Onion Console (SOC) into a cohesive NSM platform. When the Elasticsearch cluster drops to RED status because a data node goes offline mid-ingestion, Suricata alerts and Zeek connection logs stop indexing; analysts opening the SOC console see stale data or empty result sets with no indication that detection has halted; when Suricata's alert ingestion rate drops to zero because a ruleset update corrupted the rule syntax and Suricata failed to reload, the IDS is silently not detecting anything while the Elasticsearch index continues receiving zero events with no alarm; when a distributed sensor node loses connectivity to the Security Onion manager, that network segment becomes a monitoring blind spot — traffic across that sensor's interface is not captured, not analyzed, and not visible in the SOC console. These failures compound: a disk full event on a PCAP capture node prunes the most recent PCAP first (closest to the current investigation window), the Elasticsearch data disk fills and rejects new index writes, and the analyst investigating a live incident has neither alerts nor PCAP to work with.

Vigilmon gives you external visibility into Security Onion's multi-component health through HTTP probe monitoring and heartbeat monitors for Elasticsearch cluster health, Suricata and Zeek ingestion rates, and sensor connectivity. This tutorial covers the full stack.

Why Security Onion Needs External Monitoring

Security Onion failure modes are silent by design — the platform's own alerting requires the platform to be working:

  • Elasticsearch cluster RED: Security Onion stores all SOC data in a multi-node Elasticsearch cluster; when a data node goes offline (OOM kill, disk full, hardware failure), the cluster enters YELLOW (degraded) and eventually RED (no primary shards); in RED status, indexing of Suricata alerts and Zeek logs fails with ClusterBlockException; analysts see no new alerts but the SOC console shows the last indexed timestamp, not a live failure indicator; by the time analysts notice the stale data, an active intrusion may have been proceeding for hours without detection
  • Suricata ingestion drop: Suricata rule updates can corrupt the active ruleset if a new rule contains a syntax error; Suricata reloads rules on update (suricata-update) and if the reload fails, Suricata may continue running with no rules loaded (generating no alerts) or may crash entirely; alert events per second in Elasticsearch drop to zero; no native Security Onion alert fires because the alerting pipeline itself depends on Suricata output
  • Zeek crash: Zeek (formerly Bro) generates the high-fidelity protocol logs (conn.log, dns.log, http.log, ssl.log) that SOC analysts use for investigation; when Zeek crashes due to a memory exhaustion or script error, the log ingestion pipeline stops; Logstash continues running but has no Zeek logs to ingest; the Elasticsearch indices stop receiving Zeek events; analysts doing retrospective investigation find a gap in protocol logs that exactly corresponds to the window the threat actor was active
  • PCAP disk full: Security Onion sensor nodes capture full PCAP using Stenographer; when the PCAP storage disk exceeds the configured threshold, Stenographer's rotation policy begins aggressively pruning old PCAP; the pruning window may shrink from 7 days to hours; during a post-incident forensic investigation, analysts attempt to retrieve PCAP for a session from 3 days ago and find it has already been rotated out
  • Sensor disconnect: Distributed Security Onion deployments have a manager node and multiple sensor nodes watching different network segments; when a sensor loses network connectivity to the manager, the sensor continues capturing locally but cannot forward logs or receive rule updates; the manager has no visibility into that segment; a threat actor who knows your network topology can operate in a segment where the sensor is known to be disconnected
  • Alert backlog growth: When the analyst team is understaffed or during a high-volume attack campaign, the open alert count in the SOC console grows unbounded; older unreviewed alerts expire from the investigation queue before they are acted on; active incidents go undetected because the relevant alerts are buried in the backlog

External monitoring with Vigilmon adds:

  • Elasticsearch cluster status probing before analysts depend on it for active investigations
  • Suricata and Zeek ingestion rate heartbeats that fire when event rates drop to zero
  • Sensor liveness checks that detect segment blind spots within minutes
  • PCAP and Elasticsearch disk monitoring before storage saturation causes data loss

Step 1: Build a Security Onion Health Endpoint

Security Onion does not expose a unified HTTP health API. Build a health aggregator that queries the Elasticsearch cluster health API and exposes SOC metrics over HTTP.

Python Health Aggregator

# health/security_onion_health.py
import os
import time
import json
import subprocess
from flask import Flask, jsonify
from elasticsearch import Elasticsearch

app = Flask(__name__)

ES_HOST  = os.environ.get('SO_ES_HOST', 'localhost')
ES_PORT  = int(os.environ.get('SO_ES_PORT', 9200))
ES_USER  = os.environ.get('SO_ES_USER', 'elastic')
ES_PASS  = os.environ.get('SO_ES_PASS', '')
SOC_URL  = os.environ.get('SO_CONSOLE_URL', 'http://localhost:443')

es = Elasticsearch(
    [{'host': ES_HOST, 'port': ES_PORT, 'scheme': 'https'}],
    http_auth=(ES_USER, ES_PASS),
    verify_certs=False,
    timeout=10,
)

def get_es_cluster_health():
    try:
        health = es.cluster.health(timeout='5s')
        return {
            'status': health['status'],
            'number_of_nodes': health['number_of_nodes'],
            'number_of_data_nodes': health['number_of_data_nodes'],
            'unassigned_shards': health['unassigned_shards'],
            'active_shards_percent': health.get('active_shards_percent_as_number', 0),
        }
    except Exception as e:
        return {'error': str(e)}

def get_suricata_rate():
    try:
        # Query ES for Suricata events in the last 60 seconds
        result = es.count(
            index='so-suricata-*',
            body={
                'query': {
                    'range': {
                        '@timestamp': {'gte': 'now-60s', 'lt': 'now'}
                    }
                }
            }
        )
        return {'events_last_60s': result['count'], 'rate_per_second': result['count'] / 60.0}
    except Exception as e:
        return {'error': str(e)}

def get_zeek_rate():
    try:
        result = es.count(
            index='so-zeek-*',
            body={
                'query': {
                    'range': {
                        '@timestamp': {'gte': 'now-60s', 'lt': 'now'}
                    }
                }
            }
        )
        return {'events_last_60s': result['count'], 'rate_per_second': result['count'] / 60.0}
    except Exception as e:
        return {'error': str(e)}

def get_es_disk_usage():
    try:
        stats = es.cluster.stats()
        fs = stats['nodes']['fs']
        total_bytes = fs['total_in_bytes']
        available_bytes = fs['available_in_bytes']
        used_bytes = total_bytes - available_bytes
        usage_pct = (used_bytes / total_bytes * 100) if total_bytes > 0 else 0
        return {
            'total_gb': round(total_bytes / 1e9, 1),
            'used_gb': round(used_bytes / 1e9, 1),
            'usage_percent': round(usage_pct, 1),
        }
    except Exception as e:
        return {'error': str(e)}

@app.route('/health')
def health():
    cluster = get_es_cluster_health()
    suricata = get_suricata_rate()
    zeek = get_zeek_rate()
    disk = get_es_disk_usage()

    cluster_status = cluster.get('status', 'unknown')
    es_disk_pct    = disk.get('usage_percent', 0) if 'error' not in disk else 100
    suricata_ok    = 'error' not in suricata and suricata.get('rate_per_second', 0) > 0
    zeek_ok        = 'error' not in zeek and zeek.get('rate_per_second', 0) > 0

    healthy = (
        cluster_status == 'green' and
        es_disk_pct < 80 and
        suricata_ok and
        zeek_ok
    )

    status_code = 200 if healthy else 503
    return jsonify({
        'healthy': healthy,
        'elasticsearch': {
            'cluster_status': cluster_status,
            'nodes': cluster.get('number_of_nodes'),
            'unassigned_shards': cluster.get('unassigned_shards'),
            'disk': disk,
        },
        'suricata': suricata,
        'zeek': zeek,
    }), status_code

@app.route('/health/elasticsearch')
def health_es():
    cluster = get_es_cluster_health()
    disk = get_es_disk_usage()
    status = cluster.get('status', 'unknown')
    healthy = status == 'green' and disk.get('usage_percent', 100) < 80
    return jsonify({'healthy': healthy, 'cluster': cluster, 'disk': disk}), (200 if healthy else 503)

@app.route('/health/suricata')
def health_suricata():
    data = get_suricata_rate()
    healthy = 'error' not in data and data.get('rate_per_second', 0) > 0
    return jsonify({'healthy': healthy, **data}), (200 if healthy else 503)

@app.route('/health/zeek')
def health_zeek():
    data = get_zeek_rate()
    healthy = 'error' not in data and data.get('rate_per_second', 0) > 0
    return jsonify({'healthy': healthy, **data}), (200 if healthy else 503)

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8765)

Install dependencies and run:

pip install flask elasticsearch
SO_ES_HOST=localhost SO_ES_USER=elastic SO_ES_PASS=yourpassword \
  python health/security_onion_health.py &

Step 2: Monitor Elasticsearch Cluster Health

The Elasticsearch cluster is the central data store for all SOC data. A cluster entering RED status means no new events are indexed.

Add an HTTP Monitor in Vigilmon

  1. Open Vigilmon → Monitors → New Monitor
  2. Set Name: Security Onion — Elasticsearch Cluster
  3. Set URL: http://your-so-manager:8765/health/elasticsearch
  4. Set Method: GET
  5. Set Expected status: 200
  6. Set Check interval: 60 seconds
  7. Set Regions: select at least two probe regions for consensus
  8. Click Save

For direct Elasticsearch API probing (no sidecar required):

  • URL: http://your-so-manager:9200/_cluster/health
  • Add Basic Auth header: Authorization: Basic <base64(user:pass)>
  • Set Expected status: 200
  • Add Response body assertion: body must not contain "status":"red"

Alert Configuration

Create an alert for this monitor:

  1. Open the monitor → Alerts → New Alert
  2. Set Trigger: Status is DOWN
  3. Set Message: Security Onion Elasticsearch cluster is unreachable or RED — Suricata and Zeek alert ingestion has stopped. Check ES node status with: so-elastic-container status
  4. Add email and Slack notification channels
  5. Set Recovery message: Elasticsearch cluster is back to GREEN — SOC data ingestion resumed

Step 3: Monitor Suricata Alert Ingestion Rate

A Suricata rule reload failure or Suricata crash results in zero alert ingestion — the IDS is silently not detecting threats.

Heartbeat Monitor for Suricata Ingestion

Use Vigilmon's heartbeat monitor to detect when Suricata stops sending events. Add a cron job on the Security Onion manager that queries the Suricata ingestion rate and pings Vigilmon only when the rate is above zero:

# /etc/cron.d/so-suricata-heartbeat
* * * * * root /opt/so/scripts/check_suricata_rate.sh

# /opt/so/scripts/check_suricata_rate.sh
#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_SURICATA_KEY"
ES_USER="elastic"
ES_PASS="yourpassword"

COUNT=$(curl -sk -u "$ES_USER:$ES_PASS" \
  'https://localhost:9200/so-suricata-*/_count' \
  -H 'Content-Type: application/json' \
  -d '{"query":{"range":{"@timestamp":{"gte":"now-60s","lt":"now"}}}}' \
  | python3 -c "import sys,json; print(json.load(sys.stdin).get('count',0))")

if [ "$COUNT" -gt "0" ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
fi

Make the script executable:

chmod +x /opt/so/scripts/check_suricata_rate.sh

Configure the Heartbeat in Vigilmon

  1. Open Vigilmon → Heartbeats → New Heartbeat
  2. Set Name: Security Onion — Suricata Alert Ingestion
  3. Set Expected interval: 2 minutes
  4. Set Grace period: 3 minutes
  5. Copy the heartbeat URL into the script above
  6. Click Save

Set alert message: Suricata alert ingestion has dropped to zero — IDS may have crashed or reloaded with invalid rules. Check: so-suricata-container status; suricata-update --dry-run

Step 4: Monitor Zeek Log Ingestion Rate

Zeek crash or pipeline failure stops the protocol log stream used for investigation context.

Heartbeat Monitor for Zeek Ingestion

# /opt/so/scripts/check_zeek_rate.sh
#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_ZEEK_KEY"
ES_USER="elastic"
ES_PASS="yourpassword"

COUNT=$(curl -sk -u "$ES_USER:$ES_PASS" \
  'https://localhost:9200/so-zeek-*/_count' \
  -H 'Content-Type: application/json' \
  -d '{"query":{"range":{"@timestamp":{"gte":"now-60s","lt":"now"}}}}' \
  | python3 -c "import sys,json; print(json.load(sys.stdin).get('count',0))")

if [ "$COUNT" -gt "0" ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
fi

Add to cron alongside the Suricata check. Create a corresponding Vigilmon heartbeat with a 2-minute interval and 3-minute grace period.

Set alert message: Zeek log ingestion has stopped — Zeek may have crashed. Connection logs, DNS logs, and HTTP logs are not being indexed. Check: so-zeek-container status

Step 5: Monitor PCAP and Elasticsearch Disk Usage

Disk saturation on either PCAP storage or Elasticsearch data nodes stops the platform.

Add HTTP Monitors for Disk Usage

Use the health endpoint from Step 1, or query the Elasticsearch disk API directly:

# Direct Elasticsearch disk check endpoint
curl -sk -u elastic:pass \
  'https://localhost:9200/_cat/allocation?v&format=json' \
  | python3 -c "
import sys, json
nodes = json.load(sys.stdin)
for n in nodes:
    pct = float(n.get('disk.percent', 0) or 0)
    if pct > 80:
        sys.exit(1)
sys.exit(0)
"

Alternatively, expose disk usage through the sidecar's /health/elasticsearch endpoint (which reports disk.usage_percent) and create an assertion in Vigilmon:

  1. Open Vigilmon → Monitors → New Monitor
  2. Set Name: Security Onion — Elasticsearch Disk Usage
  3. Set URL: http://your-so-manager:8765/health/elasticsearch
  4. Set Response assertion: $.disk.usage_percent is less than 80
  5. Set Check interval: 300 seconds (5 minutes)

For PCAP disk monitoring, add a heartbeat script on each sensor node:

# /opt/so/scripts/check_pcap_disk.sh
#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_PCAP_DISK_KEY"
PCAP_DIR="/nsm/pcap"  # adjust to your PCAP mount point

USAGE=$(df "$PCAP_DIR" | awk 'NR==2 {gsub(/%/,""); print $5}')
if [ "$USAGE" -lt "85" ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
fi

Run every 5 minutes via cron. Configure the Vigilmon heartbeat with an 8-minute interval.

Step 6: Monitor Security Onion Console Availability

The SOC console (web UI) must be reachable for analysts to triage alerts and investigate incidents.

  1. Open Vigilmon → Monitors → New Monitor
  2. Set Name: Security Onion — SOC Console
  3. Set URL: https://your-so-manager (the Security Onion web interface)
  4. Set Method: GET
  5. Set Expected status: 200 or 302 (login redirect is healthy)
  6. Set SSL expiry alert: 30 days
  7. Set Check interval: 60 seconds
  8. Set Regions: at least two regions

Alert message: Security Onion Console is unreachable — analysts cannot access the SOC investigation interface. Check: so-nginx-container status

Step 7: Monitor Sensor Connectivity

Distributed sensor nodes that lose connectivity to the manager create network visibility gaps.

Heartbeat Monitor per Sensor Node

Run on each sensor node:

# /etc/cron.d/so-sensor-heartbeat
* * * * * root curl -sf https://vigilmon.online/heartbeat/YOUR_SENSOR_KEY > /dev/null

# Add to each sensor's crontab, using a unique heartbeat key per sensor

Configure one Vigilmon heartbeat per sensor node with:

  • Name: Security Onion — Sensor Node [hostname]
  • Expected interval: 2 minutes
  • Grace period: 5 minutes

Alert message: Sensor node [hostname] has lost connectivity to Vigilmon — this network segment may have a monitoring blind spot. Verify sensor health and manager connectivity.

Step 8: Monitor Suricata Rule Freshness

Outdated IDS rules reduce detection coverage. Alert when rules have not been updated in over 48 hours.

# /opt/so/scripts/check_rule_freshness.sh
#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_RULES_KEY"
RULES_DIR="/etc/suricata/rules"
UPDATED_MARKER="/var/log/suricata/rule-update.log"

# Check last modification time of the rules directory
LAST_MOD=$(stat -c %Y "$RULES_DIR" 2>/dev/null || echo 0)
NOW=$(date +%s)
AGE_HOURS=$(( (NOW - LAST_MOD) / 3600 ))

if [ "$AGE_HOURS" -lt "48" ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
fi

Configure with a 6-hour heartbeat interval and 49-hour grace period.

Step 9: Alerting Configuration Summary

| Monitor | Type | Interval | Alert Threshold | |---------|------|----------|-----------------| | Elasticsearch cluster health | HTTP probe | 60s | Non-200 or "status":"red" | | SOC Console availability | HTTP probe | 60s | Non-200 | | Elasticsearch disk usage | HTTP assertion | 300s | disk.usage_percent > 80 | | Suricata alert ingestion rate | Heartbeat | 2 min | Missing for 3+ min | | Zeek log ingestion rate | Heartbeat | 2 min | Missing for 3+ min | | PCAP disk usage (per sensor) | Heartbeat | 8 min | Missing when >85% used | | Sensor node liveness (per sensor) | Heartbeat | 2 min | Missing for 5+ min | | Suricata rule freshness | Heartbeat | 6 hr | Missing for 49+ hrs |

Conclusion

Security Onion's failure modes are particularly dangerous because the platform is both the detection system and the investigation tool — when it fails, you lose the ability to detect threats and to investigate incidents simultaneously. Vigilmon's external HTTP probe and heartbeat monitoring closes this gap: Elasticsearch cluster health is monitored before analysts depend on it, Suricata and Zeek ingestion rates are validated every minute, sensor nodes are tracked for connectivity gaps, and disk saturation is caught before it causes data loss.

The eight monitors above give your SOC team early warning of platform degradation — before an analyst opens the console during an active incident and finds stale data, missing protocol logs, or unretrievable PCAP.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles