Monitoring Katello with Vigilmon

Katello turns Foreman into a full lifecycle content management system — synchronizing RPM and DEB repositories, managing Red Hat subscriptions via Candlepin, publishing Content Views through lifecycle environments (Dev → QA → Production), and executing remote jobs on registered hosts via Rex. When something breaks in that pipeline, hosts stop receiving patches, subscriptions go dark, and Satellite-style operations grind to a halt. Vigilmon gives you a single pane of glass over every critical Katello subsystem: the Foreman web UI, Pulp sync tasks, Content View promotions, Candlepin, Rex job queues, and the PostgreSQL backend.

What You'll Set Up

• Foreman web UI uptime and HTTP health monitoring
• Pulp content sync task success rate alerting
• Content View publish and promote health tracking
• Subscription manifest validity and over-consumption alerts
• Host registration success rate monitoring
• Remote Execution (Rex) job success rate alerts
• Errata applicability calculation health
• Candlepin subscription service health
• PostgreSQL database connectivity and latency monitoring
• Katello agent (goferd) connectivity tracking

Prerequisites

• Katello 4.x+ deployed (typically on RHEL 8/9 or CentOS Stream)
• Foreman web UI accessible over HTTPS
• A free Vigilmon account
• Hammer CLI installed and configured for querying Katello API

Step 1: Monitor the Foreman Web UI

The Foreman web UI is the primary entry point for all Katello operations. If it's down, administrators cannot manage hosts, Content Views, or subscriptions.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter your Foreman URL: https://foreman.example.com.
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200 (or 302 if your Foreman redirects / to /users/login).
6. Enable Monitor SSL certificate and set Alert when certificate expires in less than 21 days.
7. Click Save.

Test the endpoint manually:

curl -sk -o /dev/null -w "%{http_code}" https://foreman.example.com
# Expected: 200 or 302

If you have a /katello API health endpoint, add a second monitor for it:

curl -sk https://foreman.example.com/katello/api/v2/ping | python3 -m json.tool
# Expected: {"services": {...}, "status": "ok"}

Add an HTTP monitor for https://foreman.example.com/katello/api/v2/ping with an expected 200 status and keyword check for "status".

Step 2: Track Pulp Content Sync Health

Katello uses Pulp 3 as its content management backend. Repository syncs are long-running async tasks — RPM repo syncs for large mirrors can take hours. A silent sync failure means hosts never receive updated packages.

Create a cron heartbeat in Vigilmon to track sync health:

1. Click Add Monitor → Cron Heartbeat.
2. Name it Katello Pulp Sync Health.
3. Set the expected ping interval to 2 hours (adjust to match your sync schedule).
4. Copy the heartbeat URL.

Add a sync monitoring script that pings Vigilmon only when the last sync succeeded:

#!/bin/bash
# /usr/local/bin/katello-sync-check.sh

FOREMAN_URL="https://foreman.example.com"
HAMMER="hammer --username admin --password $FOREMAN_PASS"
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_ID"

# Check for failed sync tasks in the last 2 hours
FAILED=$($HAMMER task list --search "type = Actions::Katello::Repository::Sync AND result = error" \
         --per-page 1 2>/dev/null | grep -c "error" || true)

if [ "$FAILED" -eq 0 ]; then
    curl -s "$HEARTBEAT_URL"
fi

Schedule via cron:

echo "0 * * * * root /usr/local/bin/katello-sync-check.sh" > /etc/cron.d/katello-sync-check

For direct Pulp API monitoring, add an HTTP monitor for:

https://foreman.example.com/pulpcore/api/v3/status/

Step 3: Monitor Content View Publish and Promote Health

Content Views are how Katello version-controls and distributes content to lifecycle environments. Promotion failures block hosts in Dev/QA/Production from receiving the latest content snapshots.

Create a cron heartbeat for publish/promote health:

1. Add a Cron Heartbeat monitor named Katello CV Promote Health.
2. Set the expected interval to 24 hours.
3. Copy the heartbeat URL.

Script to check CV task status:

#!/bin/bash
# /usr/local/bin/katello-cv-check.sh

HAMMER="hammer --username admin --password $FOREMAN_PASS"
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_CV_HEARTBEAT_ID"

# Count failed publish/promote tasks in last 24h
FAILED=$($HAMMER task list \
  --search "type = Actions::Katello::ContentView::Publish AND result = error" \
  --per-page 1 2>/dev/null | grep -c "error" || true)

FAILED_PROMOTE=$($HAMMER task list \
  --search "type = Actions::Katello::ContentView::Promote AND result = error" \
  --per-page 1 2>/dev/null | grep -c "error" || true)

if [ "$FAILED" -eq 0 ] && [ "$FAILED_PROMOTE" -eq 0 ]; then
    curl -s "$HEARTBEAT_URL"
fi

Step 4: Subscription Manifest Health

Katello manages Red Hat subscriptions through manifests imported from the Red Hat Customer Portal. An expired or over-consumed manifest means new hosts cannot register, and existing hosts may lose entitlements.

Query subscription status using the Katello API:

curl -sk -u admin:$FOREMAN_PASS \
  "https://foreman.example.com/katello/api/v2/subscriptions?per_page=100" \
  | python3 -c "
import json, sys
data = json.load(sys.stdin)
for sub in data.get('results', []):
    consumed = sub.get('consumed', 0)
    quantity = sub.get('quantity', 0)
    name = sub.get('name', 'unknown')
    if quantity > 0 and consumed >= quantity:
        print(f'OVER-CONSUMED: {name} ({consumed}/{quantity})')
"

Create a cron heartbeat monitor named Katello Subscription Health with a 24 hour interval and a script that only pings when no subscriptions are over-consumed:

#!/bin/bash
RESULT=$(curl -sk -u admin:$FOREMAN_PASS \
  "https://foreman.example.com/katello/api/v2/subscriptions?per_page=100" \
  | python3 -c "
import json, sys
data = json.load(sys.stdin)
over = [s for s in data.get('results', [])
        if s.get('quantity', 0) > 0 and s.get('consumed', 0) >= s.get('quantity', 0)]
print(len(over))
")

if [ "$RESULT" = "0" ]; then
    curl -s "https://vigilmon.online/heartbeat/YOUR_SUB_HEARTBEAT_ID"
fi

Step 5: Host Registration and Rex Job Monitoring

New hosts register with Katello using activation keys. If registration is failing, you have a provisioning outage. Remote Execution (Rex) failures mean errata and configuration jobs are not reaching managed hosts.

Check host registration health:

# Count hosts registered in the last hour
hammer host list \
  --search "last_report > \"1 hour ago\"" \
  --per-page 1 2>/dev/null | grep -c "."

For Rex job monitoring, add a cron heartbeat that checks recent job invocations:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_REX_HEARTBEAT_ID"

# Check for failed Rex jobs in last hour
FAILED=$(hammer job-invocation list \
  --search "status = failed" \
  --per-page 1 2>/dev/null | grep -c "failed" || true)

if [ "$FAILED" -eq 0 ]; then
    curl -s "$HEARTBEAT_URL"
fi

Schedule both scripts hourly via cron.

Step 6: Monitor Candlepin and PostgreSQL

Candlepin is the subscription management service that runs alongside Katello. If Candlepin is down, subscription operations fail even if Foreman is healthy.

Add an HTTP monitor for the Candlepin status endpoint:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: https://foreman.example.com:8443/candlepin/status
3. Expected status: 200.
4. Keyword check: "running":true.

For PostgreSQL, add a cron heartbeat that tests connectivity and query latency:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_PG_HEARTBEAT_ID"

START=$(date +%s%N)
RESULT=$(PGPASSWORD="$FOREMAN_DB_PASS" psql -h localhost -U foreman \
         -d foreman -c "SELECT 1;" -t 2>/dev/null | tr -d ' ')
END=$(date +%s%N)
LATENCY=$(( (END - START) / 1000000 ))

if [ "$RESULT" = "1" ] && [ "$LATENCY" -lt 2000 ]; then
    curl -s "$HEARTBEAT_URL"
fi

Step 7: Monitor Katello Agent (goferd) Connectivity

Registered hosts use goferd (a Go-based daemon) to receive remote action requests from Katello. If goferd connections drop, remote errata deployment and remote actions silently fail.

Check active goferd connections on the Katello server:

# Count active goferd connections (AMQP on port 5647)
ss -tn state established '( dport = :5647 or sport = :5647 )' | grep -c ESTAB

Create a cron heartbeat monitor named Katello goferd Connections with a 30 minute interval:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_GOFERD_HEARTBEAT_ID"
MIN_CONNECTIONS=1  # Alert if fewer than this many hosts are connected

COUNT=$(ss -tn state established '( dport = :5647 or sport = :5647 )' 2>/dev/null | grep -c ESTAB || true)

if [ "$COUNT" -ge "$MIN_CONNECTIONS" ]; then
    curl -s "$HEARTBEAT_URL"
fi

Step 8: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and configure email, Slack, or PagerDuty.
2. For the Foreman UI monitor, set Consecutive failures before alert to 2 — the Foreman Rails app may take 15–30 seconds to restart after a service reload.
3. For sync, CV promote, and subscription heartbeats, a single missed ping warrants an alert — these processes run on predictable schedules and a missed window means something failed silently.
4. For Candlepin, set consecutive failures to 3 — the service may be briefly unavailable during Katello upgrades.
5. Add a Maintenance Window in Vigilmon before running foreman-installer upgrades, which restart all services simultaneously.

Automate maintenance windows via the Vigilmon API:

# Suppress all monitors during Katello upgrade
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"monitor_ids": ["ui-id", "candlepin-id"], "duration_minutes": 30}'

Conclusion

Katello's value is its automated software lifecycle pipeline — but that pipeline has many moving parts: Pulp sync workers, Content View promotions, Candlepin entitlements, Rex job queues, and goferd connections. Any one of them can fail silently while the Foreman UI stays green. With Vigilmon monitoring the full stack, you'll catch sync failures before hosts miss patch cycles, detect subscription over-consumption before compliance audits, and know immediately when goferd connectivity drops — keeping your entire software lifecycle management pipeline observable and auditable.