tutorial

Monitoring Pulp with Vigilmon

Pulp is the open source content management backend for Katello and Foreman — managing RPM, DEB, Python, Ansible, and container image repositories. Here's how to monitor Pulp's API, task workers, content storage, PostgreSQL, Redis, and distribution endpoints with Vigilmon.

Pulp is Red Hat's open source content repository management system — the backend engine that synchronizes RPM, DEB, Python, Ansible, and container image content for on-premises software distribution. In Katello/Foreman deployments, Pulp is what actually pulls packages from upstream mirrors and makes them available to managed hosts. But Pulp's architecture is distributed: a Django REST API, async task workers (for sync/publish/copy operations), PostgreSQL for metadata, Redis for task coordination, and local (or NFS) storage for content. Any one of those components failing causes silent content delivery disruption — hosts continue running but stop receiving package updates. Vigilmon gives you active monitoring across all Pulp subsystems: API health, worker process counts, task queue depth, storage capacity, database latency, and distribution endpoint reachability.

What You'll Set Up

  • Pulp API health endpoint monitoring
  • Content sync task success rate via cron heartbeats
  • Pulp task queue depth and worker health tracking
  • Content storage disk usage alerts
  • PostgreSQL database connectivity and latency monitoring
  • Redis connectivity and health monitoring
  • Publication and distribution endpoint reachability
  • Artifact deduplication efficiency tracking
  • Pulp API response time monitoring

Prerequisites

  • Pulp 3.x installed (standalone or as Katello's backend)
  • Access to the Pulp API (typically at /pulp/api/v3/)
  • PostgreSQL and Redis running as Pulp dependencies
  • A free Vigilmon account

Step 1: Monitor the Pulp API Health Endpoint

Pulp 3 exposes a /pulp/api/v3/status/ endpoint that reports the health of all connected services — database, Redis, and workers. This is your first line of defense.

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter: https://pulp.example.com/pulp/api/v3/status/
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Add a keyword check for "online":true to verify the API is actually healthy, not just responding.
  7. Click Save.

Test the endpoint manually:

curl -s https://pulp.example.com/pulp/api/v3/status/ | python3 -m json.tool

Expected response:

{
  "versions": [...],
  "online_workers": [{"name": "resource-manager@pulp-worker", "last_heartbeat": "..."}],
  "online_content_apps": [...],
  "database_connection": {"connected": true},
  "redis_connection": {"connected": true},
  "storage": {"total": 500000000000, "used": 120000000000}
}

If the API is also used by Katello/Foreman, monitor API response time with a second monitor targeting the same URL and setting an alert if response time exceeds 5 seconds.

Step 2: Track Content Sync Task Success Rate

Repository synchronization is the most critical Pulp operation. A failed sync means a repository stops receiving updates — hosts continue to install old package versions without any visible error.

Create a cron heartbeat for sync health:

  1. Click Add Monitor → Cron Heartbeat.
  2. Name it Pulp Sync Task Health.
  3. Set the expected interval to 2 hours (adjust to match your sync schedules).
  4. Copy the heartbeat URL.

Script to check sync task failures:

#!/bin/bash
# /usr/local/bin/pulp-sync-check.sh
PULP_URL="https://pulp.example.com"
PULP_USER="admin"
PULP_PASS="$PULP_ADMIN_PASS"
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_SYNC_HEARTBEAT_ID"

# Check for failed sync tasks in the last 2 hours
FAILED=$(curl -s -u "$PULP_USER:$PULP_PASS" \
  "${PULP_URL}/pulp/api/v3/tasks/?state=failed&limit=10" 2>/dev/null | \
  python3 -c "
import json, sys
from datetime import datetime, timedelta, timezone
data = json.load(sys.stdin)
cutoff = datetime.now(timezone.utc) - timedelta(hours=2)
recent_failures = [
    t for t in data.get('results', [])
    if t.get('finished_at') and
       datetime.fromisoformat(t['finished_at'].replace('Z', '+00:00')) > cutoff
]
print(len(recent_failures))
")

if [ "$FAILED" = "0" ]; then
    curl -s "$HEARTBEAT_URL"
fi

Schedule via cron to run every 30 minutes:

echo "*/30 * * * * root /usr/local/bin/pulp-sync-check.sh" > /etc/cron.d/pulp-sync-check

Step 3: Monitor Task Queue Depth and Worker Health

Pulp uses async task workers for all long-running operations (sync, publish, copy, migrate). If workers crash or the task queue grows unboundedly, operations queue indefinitely without error messages to clients.

Add an HTTP monitor that checks the status endpoint for worker count:

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: https://pulp.example.com/pulp/api/v3/status/
  3. Keyword check: "online_workers" with a count greater than zero.

For task queue depth monitoring, create a cron heartbeat:

#!/bin/bash
# /usr/local/bin/pulp-worker-check.sh
PULP_URL="https://pulp.example.com"
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_WORKER_HEARTBEAT_ID"
MIN_WORKERS=2          # Alert if fewer than this many workers are online
MAX_QUEUE_DEPTH=50     # Alert if this many tasks are waiting

# Check worker count
WORKERS=$(curl -s -u admin:"$PULP_ADMIN_PASS" \
  "${PULP_URL}/pulp/api/v3/status/" 2>/dev/null | \
  python3 -c "import json,sys; d=json.load(sys.stdin); print(len(d.get('online_workers',[])))")

# Check pending task count
PENDING=$(curl -s -u admin:"$PULP_ADMIN_PASS" \
  "${PULP_URL}/pulp/api/v3/tasks/?state=waiting&limit=1" 2>/dev/null | \
  python3 -c "import json,sys; print(json.load(sys.stdin).get('count', 0))")

if [ "${WORKERS:-0}" -ge "$MIN_WORKERS" ] && [ "${PENDING:-0}" -lt "$MAX_QUEUE_DEPTH" ]; then
    curl -s "$HEARTBEAT_URL"
fi

Step 4: Monitor Content Storage Disk Usage

Pulp stores all content (RPMs, DEBs, container layers, Ansible collections) in a content-addressable store on disk. When the storage directory fills up, sync tasks fail with disk-full errors and content becomes unavailable.

Create a cron heartbeat for storage monitoring:

  1. Add a Cron Heartbeat named Pulp Content Storage.
  2. Set the expected interval to 30 minutes.

Script:

#!/bin/bash
# /usr/local/bin/pulp-storage-check.sh
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_STORAGE_HEARTBEAT_ID"
STORAGE_THRESHOLD=80  # Alert if storage is >80% full

# Check from Pulp API
STORAGE=$(curl -s -u admin:"$PULP_ADMIN_PASS" \
  "https://pulp.example.com/pulp/api/v3/status/" 2>/dev/null | \
  python3 -c "
import json, sys
d = json.load(sys.stdin)
storage = d.get('storage', {})
total = storage.get('total', 0)
used = storage.get('used', 0)
if total > 0:
    print(int(used * 100 / total))
else:
    print(0)
")

# Also check the filesystem directly as a backup
FS_USAGE=$(df /var/lib/pulp 2>/dev/null | awk 'NR==2 {gsub("%",""); print $5}')

OVER_THRESHOLD=0
[ -n "$STORAGE" ] && [ "$STORAGE" -ge "$STORAGE_THRESHOLD" ] && OVER_THRESHOLD=1
[ -n "$FS_USAGE" ] && [ "$FS_USAGE" -ge "$STORAGE_THRESHOLD" ] && OVER_THRESHOLD=1

if [ "$OVER_THRESHOLD" -eq 0 ]; then
    curl -s "$HEARTBEAT_URL"
fi

Pulp's content-addressable storage deduplicates by SHA-256 — if you add multiple repositories with overlapping packages, each package is stored only once. Track storage growth rate monthly to plan capacity.

Step 5: Monitor PostgreSQL and Redis Health

Pulp stores all metadata (repository definitions, content units, task history, distribution endpoints) in PostgreSQL. Redis is used for task locking and caching. Both are hard dependencies — if either fails, Pulp cannot accept API requests or coordinate task execution.

PostgreSQL — add a cron heartbeat:

#!/bin/bash
# /usr/local/bin/pulp-postgres-check.sh
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_PG_HEARTBEAT_ID"
MAX_LATENCY_MS=2000

START=$(date +%s%N)
RESULT=$(PGPASSWORD="$PULP_DB_PASS" psql -h localhost -U pulp -d pulpcore \
         -c "SELECT 1;" -t 2>/dev/null | tr -d ' ')
END=$(date +%s%N)
LATENCY=$(( (END - START) / 1000000 ))

if [ "$RESULT" = "1" ] && [ "$LATENCY" -lt "$MAX_LATENCY_MS" ]; then
    curl -s "$HEARTBEAT_URL"
fi

Redis — add a cron heartbeat:

#!/bin/bash
# /usr/local/bin/pulp-redis-check.sh
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_REDIS_HEARTBEAT_ID"

PONG=$(redis-cli -h localhost PING 2>/dev/null)

if [ "$PONG" = "PONG" ]; then
    curl -s "$HEARTBEAT_URL"
fi

Schedule both every 5 minutes. Pulp's status endpoint also reports database_connection and redis_connection — you can use the existing HTTP monitor for the status endpoint as a coarse Redis+DB health check and use dedicated heartbeats for latency-aware alerting.

Step 6: Monitor Distribution and Publication Endpoints

Pulp makes content available through Distributions — URL endpoints that clients (yum, apt, pip, Ansible Galaxy) use to download packages. A Pulp distribution serving errors means managed clients cannot install software even if the content sync is healthy.

Find your distribution base paths:

curl -s -u admin:"$PULP_ADMIN_PASS" \
  "https://pulp.example.com/pulp/api/v3/distributions/rpm/rpm/?limit=100" | \
  python3 -c "
import json, sys
data = json.load(sys.stdin)
for d in data.get('results', []):
    print(d.get('base_url', ''))
"

Add an HTTP monitor for each critical distribution:

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: https://pulp.example.com/pulp/content/my-repo/ (the base_url from the API).
  3. Set Expected HTTP status to 200 or 301 (redirects are normal for content directories).
  4. Set Check interval to 5 minutes.

For RPM distributions, you can check for the repomd.xml file which confirms the repo is fully published:

https://pulp.example.com/pulp/content/my-repo/repodata/repomd.xml

Add a keyword check for <repomd to confirm the file is valid XML, not an error page.

Step 7: Track Artifact Deduplication Efficiency

Pulp deduplicates content using SHA-256 content-addressable storage — the same RPM file in 10 different repositories is stored once on disk. Monitoring the deduplication ratio helps you understand storage efficiency and detect when deduplication is degrading (for example, after a Pulp upgrade that changes how artifacts are stored).

#!/bin/bash
# /usr/local/bin/pulp-dedup-check.sh

# Total artifact count vs unique on-disk files
TOTAL_ARTIFACTS=$(curl -s -u admin:"$PULP_ADMIN_PASS" \
  "https://pulp.example.com/pulp/api/v3/artifacts/?limit=1" 2>/dev/null | \
  python3 -c "import json,sys; print(json.load(sys.stdin).get('count', 0))")

# Unique file count on disk
UNIQUE_FILES=$(find /var/lib/pulp/media -type f 2>/dev/null | wc -l)

echo "Total artifact references: $TOTAL_ARTIFACTS"
echo "Unique files on disk: $UNIQUE_FILES"

if [ "$TOTAL_ARTIFACTS" -gt 0 ] && [ "$UNIQUE_FILES" -gt 0 ]; then
    RATIO=$(python3 -c "print(f'{($TOTAL_ARTIFACTS - $UNIQUE_FILES) / $TOTAL_ARTIFACTS * 100:.1f}%')")
    echo "Deduplication savings: $RATIO"
fi

Run this as a daily cron job and log the output. A sudden drop in deduplication ratio warrants investigation — it may indicate orphaned content or storage issues.

Step 8: Configure Alert Channels

  1. Go to Alert Channels in Vigilmon and add email, Slack, or PagerDuty.
  2. For the Pulp API status monitor, set Consecutive failures before alert to 2 — the Pulp API may take up to 30 seconds to restart after a service reload.
  3. For sync task health heartbeats, alert on the first missed ping — a missed sync heartbeat means packages are not being updated and every passing minute increases the patch gap.
  4. For storage disk usage, alert immediately at 80% — Pulp sync tasks fail hard when storage is full, and there is no graceful degradation.
  5. For PostgreSQL and Redis heartbeats, alert on the first missed ping — both are hard dependencies with no fallback mode.
  6. For distribution endpoint monitors, set consecutive failures to 3 — CDN or reverse proxy transient errors should not page on-call.
  7. Use Maintenance Windows during Pulp upgrades (which involve database migrations and service restarts):
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"monitor_ids": ["pulp-api-id", "sync-id", "redis-id", "pg-id"], "duration_minutes": 45}'

Conclusion

Pulp's distributed architecture — REST API, async task workers, PostgreSQL metadata store, Redis coordinator, and content-addressable storage — means failures can appear in multiple places at once, or in only one layer while others appear healthy. A sync failure shows up as a stale repository, not an API error. A Redis failure shows up as hung tasks, not a 500 response. With Vigilmon monitoring Pulp's API health endpoint, worker counts, task queue depth, storage capacity, PostgreSQL latency, Redis connectivity, and distribution endpoint reachability, you get the complete observability picture that Pulp's architecture requires — catching content delivery failures before your managed hosts notice they're running outdated packages.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles