tutorial

Monitoring Spacewalk with Vigilmon

Spacewalk manages patch deployment for hundreds of Linux servers — but if Taskomatic stops or the holding disk fills, critical security patches stop flowing. Here's how to monitor Spacewalk's web UI, background tasks, channel sync, and client check-ins with Vigilmon.

Spacewalk started as Red Hat's internal patch management platform, open sourced in 2008, and today still runs in many production data centers managing errata deployment across large RHEL, CentOS, and Fedora server fleets. When it works, Spacewalk is invisible — patches flow, clients check in, channels sync. When it breaks, security patches stop reaching servers and you don't find out until the next audit. Vigilmon makes Spacewalk's health visible, alerting you when Taskomatic stalls, a channel sync falls behind, or clients stop checking in.

What You'll Set Up

  • Spacewalk web UI uptime monitoring
  • Taskomatic background task health checks
  • Channel synchronization status alerts
  • OSA dispatcher (osad) health monitoring
  • PostgreSQL database connectivity checks
  • Managed client check-in frequency alerts
  • Disk usage monitoring for /var/satellite

Prerequisites

  • Spacewalk 2.10+ installed and managing at least one RHEL/CentOS/Fedora system
  • Spacewalk API accessible (default: https://spacewalk-server/rpc/api)
  • A free Vigilmon account

Step 1: Monitor the Spacewalk Web UI

Spacewalk's web interface is the primary management console — if it goes down, administrators cannot schedule errata, manage channels, or view client status. Add an HTTP monitor to detect web UI failures.

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter your Spacewalk URL: https://spacewalk.yourdomain.com.
  4. Set Check interval to 2 minutes.
  5. Set Expected HTTP status to 200 or 302 (Spacewalk often redirects / to /rhn/YourRhn.do).
  6. Enable Monitor SSL certificate and set alert threshold to 21 days.
  7. Click Save.

For a more accurate health signal than the login page, check the API endpoint:

https://spacewalk.yourdomain.com/rpc/api

This returns an XML-RPC capabilities response when Tomcat and the Spacewalk application are healthy.

Step 2: Monitor Taskomatic Health

Taskomatic is Spacewalk's background task engine — it handles channel synchronization, scheduled errata deployment, and report generation. A failed Taskomatic process means patches stop flowing even though the web UI appears healthy.

Create the Taskomatic health check:

#!/bin/bash
# /usr/local/bin/spacewalk-taskomatic-check.sh
VIGILMON_WEBHOOK="https://vigilmon.online/api/webhook/YOUR_TASKOMATIC_WEBHOOK_ID"

# Check if Taskomatic process is running
if systemctl is-active --quiet taskomatic 2>/dev/null || \
   pgrep -f "taskomatic" > /dev/null 2>&1; then
    # Also check if it's accepting internal connections (default port 25151)
    if nc -z localhost 25151 2>/dev/null; then
        curl -s -X POST "${VIGILMON_WEBHOOK}" \
            -H "Content-Type: application/json" \
            -d '{"status": "up", "message": "Taskomatic running and accepting connections"}'
    else
        curl -s -X POST "${VIGILMON_WEBHOOK}" \
            -H "Content-Type: application/json" \
            -d '{"status": "down", "message": "Taskomatic process up but not accepting connections"}'
    fi
else
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d '{"status": "down", "message": "Taskomatic process is not running"}'
fi

Schedule every 3 minutes:

*/3 * * * * root /usr/local/bin/spacewalk-taskomatic-check.sh

Create a Webhook (Push) monitor in Vigilmon for this check.

Step 3: Monitor Channel Synchronization

Spacewalk synchronizes software channels from Red Hat CDN (or custom repositories) on a schedule. When a channel sync fails, managed clients cannot pull the latest security patches. Monitor sync status via the Spacewalk API.

Create the channel sync check script:

#!/bin/bash
# /usr/local/bin/spacewalk-channel-sync-check.sh
SW_HOST="localhost"
SW_USER="admin"
SW_PASS="your_admin_password"
MAX_SYNC_AGE_HOURS=26  # Alert if last successful sync older than 26 hours
VIGILMON_WEBHOOK="https://vigilmon.online/api/webhook/YOUR_SYNC_WEBHOOK_ID"

# Use spacewalk-report to check channel sync status
python3 << EOF
import xmlrpc.client
import sys
from datetime import datetime, timezone

client = xmlrpc.client.ServerProxy("https://${SW_HOST}/rpc/api", use_datetime=True)
try:
    key = client.auth.login("${SW_USER}", "${SW_PASS}")
    channels = client.channel.listAllChannels(key)
    
    stale = []
    for ch in channels:
        label = ch['label']
        details = client.channel.software.getDetails(key, label)
        last_modified = details.get('last_modified')
        if last_modified:
            age_hours = (datetime.now() - last_modified.replace(tzinfo=None)).total_seconds() / 3600
            if age_hours > ${MAX_SYNC_AGE_HOURS}:
                stale.append(f"{label} ({age_hours:.0f}h ago)")
    
    client.auth.logout(key)
    
    if stale:
        print(f"STALE: {'; '.join(stale[:5])}")
        sys.exit(1)
    else:
        print(f"OK: {len(channels)} channels synchronized")
        sys.exit(0)
except Exception as e:
    print(f"ERROR: {e}")
    sys.exit(2)
EOF

STATUS=$?
MESSAGE=$(tail -1 /dev/stdin 2>/dev/null || echo "Check complete")

if [ "${STATUS}" -ne 0 ]; then
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"down\", \"message\": \"Channel sync issue: ${MESSAGE}\"}"
else
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"up\", \"message\": \"${MESSAGE}\"}"
fi

Step 4: Monitor the OSA Dispatcher (osad)

Spacewalk's osa-dispatcher (osad) service handles real-time push notifications to managed clients, enabling immediate patch scheduling without waiting for the next check-in cycle. If osad fails, patch actions are delayed until the client's next scheduled check-in.

Create the osad check:

#!/bin/bash
# /usr/local/bin/spacewalk-osad-check.sh
VIGILMON_WEBHOOK="https://vigilmon.online/api/webhook/YOUR_OSAD_WEBHOOK_ID"

if systemctl is-active --quiet osa-dispatcher 2>/dev/null; then
    # Verify the Jabber/XMPP connection is established
    CONNECTIONS=$(ss -tnp 2>/dev/null | grep osa-dispatcher | grep ESTABLISHED | wc -l)
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"up\", \"message\": \"osad running with ${CONNECTIONS} active client connections\"}"
else
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d '{"status": "down", "message": "osa-dispatcher is not running — real-time push disabled"}'
fi

Step 5: Monitor PostgreSQL Database Health

Spacewalk stores all channel metadata, errata, client registrations, and action history in PostgreSQL. Database connectivity failures cause complete Spacewalk failure — the web UI returns errors, the API fails, and no clients can check in.

Add a TCP monitor for PostgreSQL:

  1. In Vigilmon, click Add Monitor → TCP Port.
  2. Enter your Spacewalk server hostname and port 5432 (default PostgreSQL port).
  3. Set Check interval to 1 minute.
  4. Click Save.

Also verify query-level health with a script:

#!/bin/bash
# /usr/local/bin/spacewalk-db-check.sh
DB_USER="spacewalk"
DB_NAME="spaceschema"
VIGILMON_WEBHOOK="https://vigilmon.online/api/webhook/YOUR_DB_WEBHOOK_ID"

START=$(date +%s%3N)
RESULT=$(sudo -u postgres psql -U "${DB_USER}" -d "${DB_NAME}" \
    -c "SELECT 1;" -t -q 2>&1)
EXIT_CODE=$?
END=$(date +%s%3N)
LATENCY=$((END - START))

if [ "${EXIT_CODE}" -eq 0 ]; then
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"up\", \"message\": \"DB responding in ${LATENCY}ms\"}"
else
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"down\", \"message\": \"DB query failed: ${RESULT}\"}"
fi

Step 6: Monitor /var/satellite Disk Usage

Spacewalk stores RPM packages for all synchronized channels in /var/satellite. This directory can consume hundreds of gigabytes. When it fills above 80%, channel synchronization fails and Spacewalk cannot cache new packages from the CDN.

Create the disk usage check:

#!/bin/bash
# /usr/local/bin/spacewalk-disk-check.sh
SATELLITE_DIR="/var/satellite"
WARN_THRESHOLD=80
CRIT_THRESHOLD=90
VIGILMON_WEBHOOK="https://vigilmon.online/api/webhook/YOUR_DISK_WEBHOOK_ID"

USAGE=$(df -h "${SATELLITE_DIR}" | awk 'NR==2{print $5}' | tr -d '%')
USED=$(df -h "${SATELLITE_DIR}" | awk 'NR==2{print $3}')
TOTAL=$(df -h "${SATELLITE_DIR}" | awk 'NR==2{print $2}')

if [ "${USAGE}" -ge "${CRIT_THRESHOLD}" ]; then
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"down\", \"message\": \"CRITICAL: /var/satellite at ${USAGE}% (${USED}/${TOTAL}) — channel sync will fail\"}"
elif [ "${USAGE}" -ge "${WARN_THRESHOLD}" ]; then
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"down\", \"message\": \"WARNING: /var/satellite at ${USAGE}% (${USED}/${TOTAL})\"}"
else
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"up\", \"message\": \"/var/satellite at ${USAGE}% (${USED}/${TOTAL})\"}"
fi

Schedule every 30 minutes. Package repositories grow slowly but predictably — early warnings let you clean old channel versions before reaching capacity.

Step 7: Monitor Client Check-in Frequency

Spacewalk clients should check in at regular intervals (default: every 4 hours). Clients that stop checking in may be disconnected, have failed osad, or have a broken network path to the Spacewalk server. Alert when clients miss multiple check-ins.

Create the check-in frequency monitor:

#!/bin/bash
# /usr/local/bin/spacewalk-checkin-check.sh
SW_HOST="localhost"
SW_USER="admin"
SW_PASS="your_admin_password"
MAX_HOURS=24  # Alert if client hasn't checked in for >24 hours
VIGILMON_WEBHOOK="https://vigilmon.online/api/webhook/YOUR_CHECKIN_WEBHOOK_ID"

python3 << 'PYEOF'
import xmlrpc.client
import sys
import json
from datetime import datetime

client = xmlrpc.client.ServerProxy(f"https://localhost/rpc/api", use_datetime=True)
try:
    key = client.auth.login("admin", "your_admin_password")
    systems = client.system.listSystems(key)
    
    stale = []
    for sys_info in systems:
        sid = sys_info['id']
        details = client.system.getDetails(key, sid)
        last_checkin = details.get('last_checkin')
        if last_checkin:
            hours_ago = (datetime.now() - last_checkin.replace(tzinfo=None)).total_seconds() / 3600
            if hours_ago > 24:
                stale.append(f"{sys_info['name']} ({hours_ago:.0f}h)")
    
    client.auth.logout(key)
    print(json.dumps({"stale": stale, "total": len(systems)}))
except Exception as e:
    print(json.dumps({"error": str(e)}))
PYEOF

RESULT=$(python3 /usr/local/bin/spacewalk-checkin-check.sh 2>/dev/null)
STALE_COUNT=$(echo "${RESULT}" | python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d.get('stale', [])))" 2>/dev/null || echo "0")
STALE_NAMES=$(echo "${RESULT}" | python3 -c "import sys,json; d=json.load(sys.stdin); print(', '.join(d.get('stale', [])[:3]))" 2>/dev/null || echo "")

if [ "${STALE_COUNT}" -gt 0 ]; then
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"down\", \"message\": \"${STALE_COUNT} clients not checked in >24h: ${STALE_NAMES}\"}"
else
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d '{"status": "up", "message": "All clients checked in within 24 hours"}'
fi

Step 8: Configure Alert Channels

  1. In Vigilmon, go to Alert Channels and connect your preferred channels (email, Slack, webhook).
  2. For the web UI and Taskomatic monitors, set Consecutive failures before alert to 2 — transient restarts are normal during maintenance.
  3. For the database TCP monitor, alert immediately on first failure.
  4. For disk usage, configure a warning alert at 80% and a critical alert at 90%.
  5. For client check-ins, daily alerts at 8 AM cover overnight issues without being noisy.

Summary

| Monitor | Type | What It Catches | |---|---|---| | Spacewalk web UI | HTTP/HTTPS | Tomcat or application crash | | Taskomatic | Webhook push | Background tasks stopped | | PostgreSQL port 5432 | TCP port | Database unavailable | | Channel sync age | Webhook push | CDN sync failure — stale patches | | osad process | Webhook push | Real-time push disabled | | /var/satellite disk | Webhook push | Disk full — sync failure | | Client check-ins | Webhook push | Clients disconnected from management |

Spacewalk's value is that it brings patches to every managed server automatically — but only if all its moving parts are healthy. With Vigilmon watching the web UI, Taskomatic, database, osad, and client check-ins, you catch issues in the patch pipeline before they become security gaps.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles