tutorial

How to Monitor Rudder IT Automation and Compliance Health with Vigilmon

Rudder agent check-in failures, compliance rate drops, and LDAP outages are invisible until nodes drift out of policy for hours. Learn how to monitor Rudder server health, agent check-in rates, and compliance metrics with Vigilmon HTTP probes and heartbeat monitors.

Rudder is an infrastructure automation and compliance platform built on CFEngine that enforces configuration policy across Linux and Windows nodes. When Rudder's web application becomes unreachable after a Play Framework OOM crash, your team loses visibility into compliance state across the entire managed fleet; when Rudder agents stop checking in — perhaps because the Rudder Relay lost connectivity to the Rudder Server — nodes continue running their last cached policies but configuration drift accumulates silently with no alerts; when PostgreSQL, where Rudder stores all compliance history, becomes unavailable, agents continue applying policies but compliance reports are lost and audit trails go dark. These failures combine to produce a scenario where your fleet appears compliant because the dashboards cannot update, while real drift is accumulating across hundreds of nodes.

Vigilmon gives you external visibility into Rudder's compliance infrastructure through HTTP probe monitoring and heartbeat monitors for scheduled compliance checks and policy generation jobs. This tutorial covers both.

Why Rudder Needs External Monitoring

Rudder's failure modes have compliance-wide blast radius:

  • Web application crash: Rudder's Play Framework application holds all policy authoring, compliance reporting, and node management UI; when the JVM heap exhausts under large fleet sizes (common when managing 500+ nodes), the web application exits and the HTTPS port stops responding — operators lose all visibility into current compliance state and cannot push policy changes
  • Agent silent failure: Rudder agents run every 5 minutes and report compliance back to the server; when an agent's cron entry is accidentally removed, the process is killed, or the agent cannot reach the server, it stops reporting — Rudder shows the node as "unknown" or "never reported" but does not page operations unless an alert is configured on check-in rate
  • Relay disconnection: Rudder Relays act as policy distribution proxies for network-segmented environments; when a relay loses connectivity to the Rudder Server, all nodes behind that relay stop receiving policy updates — they continue applying their last cached policies, but new rules and technique updates do not propagate, and compliance reporting from those nodes stops flowing to the server
  • PostgreSQL failure: Rudder stores its entire compliance history, node inventory, and event log in PostgreSQL; when the database becomes unavailable due to disk full, OOM kill, or connection exhaustion, agents continue applying policies but their reports cannot be written — the compliance dashboard freezes at last-known state while the actual state of the fleet diverges
  • Policy generation timeout: Rudder must compile per-node policy bundles every time a rule or technique changes; on large fleets with hundreds of techniques, generation can take 10–20 minutes; if generation hangs due to a deadlock in technique compilation, operators see "policy generation in progress" indefinitely while nodes wait for updated policies
  • LDAP failure: Rudder uses an embedded OpenLDAP instance to store node and group data; when LDAP becomes unavailable, policy lookups fail and the web application cannot resolve group membership for rule targeting

External monitoring with Vigilmon adds:

  • Proactive alerting when the Rudder web application stops accepting HTTPS connections
  • Agent check-in rate tracking via heartbeat monitors attached to the agent reporting pipeline
  • Policy generation liveness through heartbeat monitors that verify generation jobs complete on schedule
  • Multi-region probe consensus that filters transient GC-pause timeouts from genuine Rudder application crashes

Step 1: Build a Rudder Health Endpoint

Rudder's web application exposes its API over HTTPS. You can probe the API status endpoint directly and build a thin sidecar for deeper compliance metric checks.

Rudder API Liveness Probe

Rudder exposes a REST API at https://rudder.your-org.example.com/rudder/api/. The API info endpoint requires authentication:

# Test Rudder API liveness
curl -k -H "X-API-Token: your-rudder-api-token" \
  https://rudder.your-org.example.com/rudder/api/latest/system/status

A successful response returns HTTP 200 with {"result":"success","action":"getStatus","data":{"global":"OK"}}.

Build a sidecar that proxies this check over a plain HTTP health endpoint for Vigilmon to probe:

Node.js Health Sidecar

// health/rudder.js
const express = require('express');
const https = require('https');

const app = express();
const RUDDER_HOST = process.env.RUDDER_HOST || 'rudder.your-org.example.com';
const RUDDER_TOKEN = process.env.RUDDER_API_TOKEN;

function checkRudderStatus() {
  return new Promise((resolve, reject) => {
    const options = {
      hostname: RUDDER_HOST,
      port: 443,
      path: '/rudder/api/latest/system/status',
      method: 'GET',
      headers: { 'X-API-Token': RUDDER_TOKEN },
      rejectUnauthorized: false, // allow self-signed certs in private deployments
    };
    const req = https.request(options, res => {
      let body = '';
      res.on('data', chunk => body += chunk);
      res.on('end', () => {
        if (res.statusCode === 200) {
          try {
            const parsed = JSON.parse(body);
            if (parsed.result === 'success') return resolve({ status: 'ok' });
            return reject(new Error(`Rudder status: ${parsed.result}`));
          } catch {
            return reject(new Error('Invalid JSON from Rudder API'));
          }
        }
        reject(new Error(`HTTP ${res.statusCode}`));
      });
    });
    req.on('error', reject);
    req.setTimeout(5000, () => { req.destroy(); reject(new Error('Timeout')); });
    req.end();
  });
}

async function checkGlobalCompliance() {
  return new Promise((resolve, reject) => {
    const options = {
      hostname: RUDDER_HOST,
      port: 443,
      path: '/rudder/api/latest/compliance/global',
      method: 'GET',
      headers: { 'X-API-Token': RUDDER_TOKEN },
      rejectUnauthorized: false,
    };
    const req = https.request(options, res => {
      let body = '';
      res.on('data', chunk => body += chunk);
      res.on('end', () => {
        try {
          const parsed = JSON.parse(body);
          const compliance = parsed.data?.globalCompliance?.compliance;
          if (compliance === undefined) return reject(new Error('No compliance field'));
          resolve({ compliance: parseFloat(compliance.toFixed(2)) });
        } catch (e) {
          reject(e);
        }
      });
    });
    req.on('error', reject);
    req.setTimeout(5000, () => { req.destroy(); reject(new Error('Timeout')); });
    req.end();
  });
}

app.get('/health/rudder', async (req, res) => {
  const checks = {};
  let healthy = true;

  try {
    const status = await checkRudderStatus();
    checks.rudder_api = status.status;
  } catch (err) {
    checks.rudder_api = `down: ${err.message}`;
    healthy = false;
  }

  try {
    const { compliance } = await checkGlobalCompliance();
    checks.global_compliance_pct = compliance;
    // Alert if compliance drops below 90%
    if (compliance < 90) {
      checks.compliance_warning = `compliance ${compliance}% < 90% threshold`;
      healthy = false;
    }
  } catch (err) {
    checks.global_compliance = `error: ${err.message}`;
  }

  return res.status(healthy ? 200 : 503).json({
    status: healthy ? 'ok' : 'degraded',
    checks,
  });
});

app.listen(3010, () => console.log('Rudder health sidecar on :3010'));

Run this sidecar on the Rudder server and expose port 3010 to Vigilmon's monitoring probes.

Python (FastAPI) Alternative

# health/rudder_health.py
import os
import httpx
from fastapi import FastAPI, Response

app = FastAPI()

RUDDER_HOST = os.environ.get("RUDDER_HOST", "rudder.your-org.example.com")
RUDDER_TOKEN = os.environ["RUDDER_API_TOKEN"]
BASE_URL = f"https://{RUDDER_HOST}/rudder/api/latest"
HEADERS = {"X-API-Token": RUDDER_TOKEN}

@app.get("/health/rudder")
async def rudder_health():
    checks = {}
    healthy = True

    async with httpx.AsyncClient(verify=False, timeout=5.0) as client:
        try:
            r = await client.get(f"{BASE_URL}/system/status", headers=HEADERS)
            r.raise_for_status()
            data = r.json()
            checks["rudder_api"] = data.get("result", "unknown")
            if data.get("result") != "success":
                healthy = False
        except Exception as e:
            checks["rudder_api"] = f"down: {str(e)}"
            healthy = False

        try:
            r = await client.get(f"{BASE_URL}/compliance/global", headers=HEADERS)
            r.raise_for_status()
            data = r.json()
            compliance = data["data"]["globalCompliance"]["compliance"]
            checks["global_compliance_pct"] = round(compliance, 2)
            if compliance < 90:
                checks["compliance_warning"] = f"{compliance}% < 90% threshold"
                healthy = False
        except Exception as e:
            checks["global_compliance"] = f"error: {str(e)}"

    status_code = 200 if healthy else 503
    return Response(
        content=str({"status": "ok" if healthy else "degraded", "checks": checks}),
        status_code=status_code,
        media_type="application/json",
    )

Step 2: Configure Vigilmon Monitoring

HTTP Monitor — Rudder Web Application

In your Vigilmon dashboard, create an HTTP monitor for the Rudder health endpoint:

| Field | Value | |-------|-------| | Monitor name | Rudder Compliance Server | | URL | http://rudder-sidecar.internal:3010/health/rudder | | Method | GET | | Check interval | Every 1 minute | | Expected status | 200 | | Alert threshold | 2 consecutive failures | | Regions | Select 2+ regions for consensus |

HTTP Monitor — Rudder API Direct

Also probe the Rudder API endpoint directly to catch cases where the sidecar itself is unavailable:

| Field | Value | |-------|-------| | Monitor name | Rudder API Endpoint | | URL | https://rudder.your-org.example.com/rudder/api/latest/system/status | | Method | GET | | Request headers | X-API-Token: your-rudder-api-token | | Check interval | Every 2 minutes | | Expected status | 200 |

Heartbeat Monitor — Rudder Agent Check-in

Rudder agents report every 5 minutes. Create a heartbeat monitor to detect when agent check-in pipelines stop:

In Vigilmon, create a heartbeat monitor with a 15-minute timeout. Then ping it from the Rudder agent reporting hook:

# /opt/rudder/etc/agent-run/post-run.d/vigilmon-heartbeat.sh
#!/bin/bash
# Runs after each Rudder agent run completes and reports to server
curl -fsS --max-time 10 \
  "https://vigilmon.online/hb/your-heartbeat-token" > /dev/null 2>&1

Install this script on a representative sample node (or all nodes) to monitor agent check-in liveness:

chmod +x /opt/rudder/etc/agent-run/post-run.d/vigilmon-heartbeat.sh

If agents stop reporting for more than 15 minutes, Vigilmon pages your team.

Heartbeat Monitor — Policy Generation

Monitor Rudder policy generation jobs to catch hangs:

# /etc/cron.d/rudder-policy-generation-heartbeat
# Run after scheduled policy generation, which runs every 15 minutes in Rudder
*/15 * * * * root \
  rudder remote run "policy-server" && \
  curl -fsS --max-time 10 \
    "https://vigilmon.online/hb/your-policy-gen-heartbeat" > /dev/null 2>&1

Configure the heartbeat monitor with a 20-minute timeout to catch policy generation that runs over.

Step 3: Configure Alerting

Alert Policies

In Vigilmon, configure the following alert policies for your Rudder monitors:

Rudder Web Application Alert

  • Trigger: 2 consecutive failed HTTP probes
  • Severity: Critical
  • Channels: PagerDuty + Slack #infrastructure-alerts
  • Message: "Rudder compliance server is unreachable. Operators cannot view or modify configuration policies."

Compliance Rate Alert (via the health endpoint returning 503)

  • Trigger: HTTP 503 response from health sidecar
  • Severity: High
  • Channels: Slack #compliance-alerts + email to compliance team
  • Message: "Rudder global compliance rate below 90%. Review compliance dashboard for drifting nodes."

Agent Check-in Heartbeat Alert

  • Trigger: Heartbeat not received for 15 minutes
  • Severity: High
  • Channels: Slack #infrastructure-alerts
  • Message: "Rudder agents have stopped reporting. Check agent service on managed nodes and relay connectivity."

Policy Generation Heartbeat Alert

  • Trigger: Heartbeat not received for 20 minutes
  • Severity: Medium
  • Channels: Slack #infrastructure-alerts
  • Message: "Rudder policy generation may be stuck. Check /var/log/rudder/webapp/policy-generation.log."

Rule Compliance Drop Alert

Use the Rudder API to monitor per-rule compliance and feed results to a Vigilmon heartbeat:

#!/bin/bash
# /etc/cron.d/check-rudder-rule-compliance — runs every 10 minutes
RUDDER_HOST="rudder.your-org.example.com"
TOKEN="your-rudder-api-token"

# Get compliance per rule; fail if any rule drops below 90%
COMPLIANT=$(curl -sk -H "X-API-Token: $TOKEN" \
  "https://${RUDDER_HOST}/rudder/api/latest/compliance/rules" | \
  python3 -c "
import json,sys
data = json.load(sys.stdin)
rules = data['data']['rules']
failed = [r['id'] for r in rules if r.get('compliance',100) < 90]
if failed:
    print('FAIL: rules below 90%: ' + ','.join(failed))
    sys.exit(1)
print('OK')
")

if [ $? -eq 0 ]; then
  curl -fsS --max-time 10 \
    "https://vigilmon.online/hb/your-rule-compliance-heartbeat" > /dev/null 2>&1
fi

Key Metrics Summary

| Metric | Alert Threshold | Impact | |--------|----------------|--------| | Rudder web application HTTP | Any failure | Operators lose policy management and compliance visibility | | Agent check-in rate | No heartbeat for >15 min | Drift accumulating silently on nodes | | Global compliance rate | <90% | Fleet-wide configuration drift | | Per-rule compliance | <90% per rule | Specific policy failing across node group | | Policy generation time | >20 minutes | New rules not reaching managed nodes | | PostgreSQL connectivity | Any failure | Compliance reports not being saved | | Rudder Relay connectivity | Heartbeat missing | Segmented network losing policy updates |

Conclusion

Rudder's compliance guarantees are only as reliable as its monitoring coverage. A silent agent failure, relay disconnection, or compliance rate drop can go undetected for hours in environments without external visibility. Vigilmon HTTP probes on the Rudder API catch application failures the moment they occur, while heartbeat monitors on agent reporting hooks and policy generation jobs give you liveness coverage that internal process monitors cannot provide.

Configure the monitors in this tutorial and your Rudder infrastructure will page you before users notice compliance drift.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles