tutorial

How to Monitor Maltrail Health with Vigilmon

Maltrail Sensor crashes, trail database staleness, and Sensor-to-Server reporting failures silently eliminate malicious traffic detection without any indication that the security tool has stopped working. Learn how to monitor Maltrail Server health, Sensor liveness, trail database freshness, and detection event rates with Vigilmon HTTP probes and heartbeat monitors.

Maltrail is the open source malicious traffic detection system developed by Miroslav Stampar at CERT.HR that passively monitors network traffic against a built-in threat intelligence database compiled from dozens of public feeds — Google Safe Browsing, abuse.ch URLhaus, Emerging Threats, PhishTank, OpenPhish, and more. When the Maltrail Sensor process crashes on a distributed network tap because the Python libpcap binding encounters a segmentation fault at 2 AM on a Saturday, malicious traffic continues flowing across the monitored interface and the threat intelligence database continues matching against real threats — but no detection events reach the Maltrail Server because the Sensor is not running; the Server web dashboard shows events from Friday afternoon as the most recent activity, and security analysts who check the dashboard on Monday morning assume the network was quiet over the weekend; when the Maltrail trail database fails to update for three days because the feed pull script encounters a connectivity timeout to an external threat intelligence source, the trail count freezes at last Thursday's indicator count and new threat indicators added to the feeds — including a new C2 domain that compromised a vendor last week — are not included in detection; when a single noisy Maltrail feed (a misconfigured or overly aggressive threat intel source) begins generating 5,000 events per hour from legitimate business IP ranges, the Maltrail Server event log floods with false positives and the signal-to-noise ratio drops to the point where analysts stop reviewing events entirely, allowing genuine C2 traffic and malware detections to go unnoticed in the noise.

Vigilmon gives you external visibility into Maltrail's lightweight multi-component architecture through HTTP probe monitoring and heartbeat monitors for Server health, Sensor liveness per node, trail database freshness, and detection event rates. This tutorial covers the full Maltrail monitoring stack.

Why Maltrail Needs External Monitoring

Maltrail failure modes are silent by design — when the Sensor stops, the Server simply stops receiving new events with no error indicator:

  • Maltrail Sensor crash: The Maltrail Sensor is a Python daemon (sensor.py) that uses libpcap to capture traffic and match packets against the loaded trail database; if the process crashes (OOM kill, libpcap error, Python exception from a malformed packet), traffic capture stops; the Maltrail Server receives no new reports from that Sensor; the Server web dashboard shows the last event timestamp from before the crash with no indication that reporting has stopped; from the dashboard, a security analyst cannot distinguish between "no malicious traffic detected" and "Sensor is not running"
  • Trail database staleness: Maltrail refreshes its built-in trail database by pulling from dozens of public threat intelligence feeds; the update script runs daily as a cron job; when an upstream feed source changes its URL, blocks the Maltrail pull request, or returns malformed data, the update fails silently; the trail database freezes at the last successful update; new threat indicators added to the feeds after the update failure — new C2 infrastructure, new phishing domains, newly registered malware distribution servers — are not detected because they are not in the stale trail database
  • Sensor-to-Server connectivity failure: In distributed Maltrail deployments, Sensors on remote network taps communicate with the central Server over HTTP; if the Server becomes unreachable from a Sensor node due to a network change (ACL update, routing change, Server IP change), the Sensor continues capturing and matching locally but cannot send events to the Server; the Server shows no events from that segment while the Sensor has detected threats locally that were never reported
  • False positive flood hiding genuine detections: Maltrail's threat intel feeds have varying quality; a single aggressive feed (e.g., an overly broad blocklist that includes CDN IP ranges) can generate thousands of false positive events per hour; when analysts are overwhelmed by false positives from one feed, they begin ignoring the Maltrail dashboard entirely; genuine C2 communication, active malware beaconing, and inbound scanner traffic from real threat actors are buried in the noise and go unresponded to
  • Server process failure: The Maltrail Server is a Python HTTP server that receives event reports from Sensors and serves the web investigation dashboard; if the Server process crashes, Sensors continue running locally but cannot report events; analysts cannot review the dashboard; in this state, the entire distributed detection system is operating but producing no accessible output
  • Packet capture performance degradation: At high traffic rates on underpowered hardware, the Maltrail Sensor may drop packets at the libpcap layer; the Sensor continues running and reporting events for packets it did process, but a percentage of traffic — potentially including malicious traffic — is not inspected; drop rate statistics require direct inspection of the Sensor host because the Sensor does not expose this metric to the Server

External monitoring with Vigilmon adds:

  • Server web interface probing that detects Server crashes within minutes
  • Sensor liveness heartbeats per node that distinguish "no traffic" from "Sensor crashed"
  • Trail database freshness monitoring that fires when the threat intel database becomes stale
  • Detection rate monitoring that distinguishes genuine zero-detection periods from Sensor failure
  • False positive rate alerting that detects feed noise before it degrades analyst effectiveness

Step 1: Build a Maltrail Health Endpoint

Maltrail does not expose a native JSON health API, but the Server serves a basic HTTP dashboard and accepts Sensor reports over HTTP. Build a lightweight health sidecar that checks the Server status and trail database metadata.

Python Health Sidecar

# health/maltrail_health.py
import os
import re
import time
import sqlite3
import subprocess
from flask import Flask, jsonify
import requests

app = Flask(__name__)

MALTRAIL_SERVER_URL  = os.environ.get('MALTRAIL_SERVER_URL', 'http://localhost:8338')
MALTRAIL_LOG_DIR     = os.environ.get('MALTRAIL_LOG_DIR', '/var/log/maltrail')
MALTRAIL_TRAILS_FILE = os.environ.get('MALTRAIL_TRAILS_FILE', '/tmp/maltrail_trails.sqlite')
MALTRAIL_INSTALL_DIR = os.environ.get('MALTRAIL_INSTALL_DIR', '/opt/maltrail')

def get_server_health():
    try:
        r = requests.get(MALTRAIL_SERVER_URL, timeout=10)
        return {'reachable': r.ok or r.status_code in (200, 301, 302), 'status_code': r.status_code}
    except Exception as e:
        return {'reachable': False, 'error': str(e)}

def get_trail_db_freshness():
    """Check the age of the Maltrail trail database by reading its mtime or the update log."""
    trails_file = os.path.join(MALTRAIL_INSTALL_DIR, 'trails.sqlite')
    if not os.path.exists(trails_file):
        # Try alternate locations
        for candidate in ['/tmp/maltrail_trails.sqlite', '/opt/maltrail/trails.sqlite']:
            if os.path.exists(candidate):
                trails_file = candidate
                break
        else:
            return {'error': 'trails database file not found', 'age_hours': 999}
    
    mtime = os.path.getmtime(trails_file)
    now = time.time()
    age_hours = (now - mtime) / 3600.0
    return {
        'trails_file': trails_file,
        'last_updated_hours_ago': round(age_hours, 1),
        'stale': age_hours > 48,
    }

def get_event_rate():
    """Count detection events logged in the last 60 minutes."""
    import datetime
    log_dir = MALTRAIL_LOG_DIR
    
    if not os.path.isdir(log_dir):
        return {'error': f'log directory not found: {log_dir}'}
    
    # Maltrail logs events in date-based files: YYYY-MM-DD.log
    today = datetime.date.today().strftime('%Y-%m-%d')
    today_log = os.path.join(log_dir, f'{today}.log')
    
    if not os.path.exists(today_log):
        return {'events_last_60min': 0, 'events_today': 0}
    
    now = time.time()
    cutoff = now - 3600  # 60 minutes ago
    events_recent = 0
    events_today = 0
    
    try:
        with open(today_log, 'r', errors='replace') as f:
            for line in f:
                events_today += 1
                # Maltrail log format: <epoch_timestamp> <src_ip> <dst_ip> <trail> <info> ...
                parts = line.split(' ', 1)
                if parts and parts[0].replace('.', '', 1).isdigit():
                    if float(parts[0]) >= cutoff:
                        events_recent += 1
    except Exception as e:
        return {'error': str(e)}
    
    return {
        'events_last_60min': events_recent,
        'events_today': events_today,
        'events_per_hour': events_recent,
    }

def get_sensor_status():
    """Check if the Maltrail Sensor process is running locally."""
    try:
        result = subprocess.run(
            ['pgrep', '-f', 'sensor.py'],
            capture_output=True, text=True, timeout=5,
        )
        running = result.returncode == 0
        pids = result.stdout.strip().split('\n') if running else []
        return {'running': running, 'pid_count': len([p for p in pids if p])}
    except Exception as e:
        return {'running': False, 'error': str(e)}

@app.route('/health')
def health():
    server    = get_server_health()
    trails    = get_trail_db_freshness()
    events    = get_event_rate()
    sensor    = get_sensor_status()

    healthy = (
        server.get('reachable', False) and
        not trails.get('stale', True) and
        sensor.get('running', False)
    )

    return jsonify({
        'healthy': healthy,
        'server': server,
        'trail_database': trails,
        'events': events,
        'sensor': sensor,
    }), (200 if healthy else 503)

@app.route('/health/server')
def health_server():
    data = get_server_health()
    return jsonify(data), (200 if data.get('reachable') else 503)

@app.route('/health/trails')
def health_trails():
    data = get_trail_db_freshness()
    healthy = not data.get('stale', True) and 'error' not in data
    return jsonify({'healthy': healthy, **data}), (200 if healthy else 503)

@app.route('/health/sensor')
def health_sensor():
    data = get_sensor_status()
    return jsonify(data), (200 if data.get('running') else 503)

@app.route('/health/events')
def health_events():
    data = get_event_rate()
    return jsonify(data), 200

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8768)

pip install flask requests
MALTRAIL_SERVER_URL=http://localhost:8338 \
MALTRAIL_LOG_DIR=/var/log/maltrail \
MALTRAIL_INSTALL_DIR=/opt/maltrail \
  python health/maltrail_health.py &

Step 2: Monitor Maltrail Server Health

The Maltrail Server must be reachable for Sensors to report events and for analysts to review detections.

Add an HTTP Monitor in Vigilmon

  1. Open Vigilmon → Monitors → New Monitor
  2. Set Name: Maltrail — Server Health
  3. Set URL: http://your-maltrail-server:8768/health/server
  4. Set Method: GET
  5. Set Expected status: 200
  6. Set Check interval: 60 seconds
  7. Set Regions: at least two probe regions

Alternatively, probe the Maltrail Server dashboard directly:

  • URL: http://your-maltrail-server:8338/ (default Maltrail Server port)
  • Expected status: 200

Alert Configuration

  1. Open the monitor → Alerts → New Alert
  2. Set Trigger: Status is DOWN
  3. Set Message: Maltrail Server is unreachable — Sensors cannot report detection events and the investigation dashboard is unavailable. Check: pgrep -f server.py; systemctl status maltrail-server
  4. Set Recovery message: Maltrail Server is back online — detection event reporting resumed
  5. Add email and Slack channels

Step 3: Monitor Maltrail Sensor Liveness (Per Node)

Each Sensor node must be actively running and reporting to the Server. A crashed Sensor is a monitoring blind spot.

Heartbeat Monitor per Sensor Node

Run this script on each Maltrail Sensor node as a cron job:

# /opt/maltrail/scripts/sensor_heartbeat.sh
#!/bin/bash
# Use a unique HEARTBEAT_URL per sensor node
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_SENSOR_NODE_KEY"

# Check that the Maltrail Sensor process is running
if pgrep -f "sensor.py" > /dev/null 2>&1; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
fi

chmod +x /opt/maltrail/scripts/sensor_heartbeat.sh

# /etc/cron.d/maltrail-sensor-heartbeat
* * * * * root /opt/maltrail/scripts/sensor_heartbeat.sh

Configure the Heartbeat in Vigilmon

Create one Vigilmon heartbeat per Sensor node:

  1. Open Vigilmon → Heartbeats → New Heartbeat
  2. Set Name: Maltrail — Sensor [hostname]
  3. Set Expected interval: 2 minutes
  4. Set Grace period: 4 minutes
  5. Copy the heartbeat URL into the script above
  6. Click Save

Alert message: Maltrail Sensor on [hostname] has stopped — traffic on this network segment is no longer being inspected for malicious indicators. Check: pgrep -f sensor.py on [hostname]; restart with: python /opt/maltrail/sensor.py

Step 4: Monitor Trail Database Freshness

A stale trail database means new threat indicators from the intelligence feeds are not being matched against network traffic.

HTTP Monitor for Trail Freshness

  1. Open Vigilmon → Monitors → New Monitor
  2. Set Name: Maltrail — Trail Database Freshness
  3. Set URL: http://your-maltrail-server:8768/health/trails
  4. Set Expected status: 200
  5. Set Check interval: 3600 seconds (1 hour)
  6. Set Response assertion: body must contain "stale":false

Alert message: Maltrail trail database has not been updated in 48+ hours — new threat indicators from intelligence feeds are not being detected. Check the maltrail update cron job: crontab -l; run manually: cd /opt/maltrail && python update_trails.py

Alternatively, Use a Freshness Heartbeat

# /opt/maltrail/scripts/check_trail_freshness.sh
#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_TRAILS_KEY"
TRAILS_FILE="/opt/maltrail/trails.sqlite"

if [ ! -f "$TRAILS_FILE" ]; then
  exit 1
fi

MTIME=$(stat -c %Y "$TRAILS_FILE")
NOW=$(date +%s)
AGE_HOURS=$(( (NOW - MTIME) / 3600 ))

if [ "$AGE_HOURS" -lt "48" ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
fi

Run hourly. Configure with a 25-hour interval (alerts if no successful update within 25 hours, giving a buffer before the 48-hour stale threshold).

Step 5: Monitor Detection Event Rate

A drop to zero detection events for an extended period may indicate Sensor failure rather than a genuinely quiet network.

Heartbeat for Detection Activity

This heartbeat pings only when there have been recent detections — it does not substitute for Sensor liveness monitoring (Step 3), but adds a layer of confirmation that events are flowing:

# /opt/maltrail/scripts/check_event_rate.sh
#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_EVENTS_KEY"
LOG_DIR="/var/log/maltrail"
TODAY=$(date +%Y-%m-%d)
LOG_FILE="$LOG_DIR/$TODAY.log"

# Count lines in the last 30 minutes of today's log
CUTOFF=$(date -d '-30 minutes' +%s)
COUNT=0

if [ -f "$LOG_FILE" ]; then
  while IFS= read -r line; do
    TS=$(echo "$line" | awk '{print int($1)}')
    if [ "$TS" -ge "$CUTOFF" ] 2>/dev/null; then
      COUNT=$((COUNT + 1))
    fi
  done < "$LOG_FILE"
fi

if [ "$COUNT" -gt "0" ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
fi

Note: Configure the grace period according to your environment's expected detection frequency. In environments with low malicious traffic, set a longer grace period (4–6 hours) to avoid false alerts during quiet periods.

Step 6: Monitor Packet Capture Performance

Packet drop rate indicates the Sensor hardware is insufficient for the traffic rate on the monitored interface.

# /opt/maltrail/scripts/check_pcap_drops.sh
#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_DROPS_KEY"
CAPTURE_IFACE="${MALTRAIL_IFACE:-eth0}"

# Get RX dropped from /proc/net/dev
RX_DROPPED=$(cat /proc/net/dev \
  | awk -v iface="$CAPTURE_IFACE:" '$1==iface{print $5}' 2>/dev/null || echo 0)

PREV_FILE="/tmp/maltrail_drops_prev"
PREV=$(cat "$PREV_FILE" 2>/dev/null || echo 0)
echo "$RX_DROPPED" > "$PREV_FILE"

NEW_DROPS=$(( RX_DROPPED - PREV ))

# Ping heartbeat if fewer than 500 new drops since last check (1 min)
if [ "$NEW_DROPS" -lt "500" ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
fi

Run every minute. Configure with a 3-minute interval.

Alert message: Maltrail Sensor on [hostname] is dropping packets — the capture interface is overloaded. Malicious traffic may not be inspected. Reduce traffic rate, upgrade NIC, or tune the ring buffer with: ethtool -G $IFACE rx 4096

Step 7: Monitor Maltrail Server Storage

Event log disk saturation causes the Maltrail Server to fail writing new events.

# /opt/maltrail/scripts/check_log_disk.sh
#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_DISK_KEY"
LOG_DIR="${MALTRAIL_LOG_DIR:-/var/log/maltrail}"

USAGE=$(df "$LOG_DIR" | awk 'NR==2 {gsub(/%/,""); print $5}')
if [ "$USAGE" -lt "80" ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
fi

Run every 10 minutes. Configure with a 15-minute interval and 17-minute grace period.

Alert message: Maltrail Server log storage is above 80% — event logs may stop being written. Archive or rotate old log files in $MALTRAIL_LOG_DIR

Step 8: Monitor Network Interface Availability

If the capture interface goes down, the Sensor continues running but is not monitoring any traffic.

# /opt/maltrail/scripts/check_iface.sh
#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_IFACE_KEY"
CAPTURE_IFACE="${MALTRAIL_IFACE:-eth0}"

if ip link show "$CAPTURE_IFACE" | grep -q "state UP"; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
fi

Run every 2 minutes. Configure with a 4-minute interval.

Alert message: Maltrail Sensor capture interface $CAPTURE_IFACE is DOWN — the Sensor is not monitoring any network traffic. Check the physical or virtual NIC connected to the network tap.

Step 9: Alerting Configuration Summary

| Monitor | Type | Interval | Alert Threshold | |---------|------|----------|-----------------| | Maltrail Server health | HTTP probe | 60s | Non-200 | | Trail database freshness | HTTP assertion | 1 hr | stale:true | | Sensor liveness (per node) | Heartbeat | 2 min | Missing for 4+ min | | Packet capture performance | Heartbeat | 2 min | Missing for 3+ min | | Network interface status | Heartbeat | 2 min | Missing for 4+ min | | Detection event rate | Heartbeat | 30 min | Missing per environment | | Server log disk usage | Heartbeat | 10 min | Missing when >80% |

Conclusion

Maltrail's greatest operational risk is its silence when it fails: a crashed Sensor, a stale trail database, or a broken Sensor-to-Server connection all produce the same observable output on the Server dashboard — no new events. Security teams routinely interpret absence of events as absence of threats, when in fact it may indicate absence of monitoring. Vigilmon's external monitoring breaks this ambiguity: Server reachability is probed every minute, Sensor liveness is validated per node every two minutes, trail database freshness is checked every hour, and packet capture performance is monitored to catch hardware saturation before it causes systematic gaps in detection coverage.

The monitors above give your security team the operational visibility to distinguish a quiet network from a broken detection system — before a threat actor who knows your monitoring has gaps takes advantage of the silence.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles