Monitoring Ansible AWX with Vigilmon

Ansible AWX is the open source upstream project for Red Hat Ansible Automation Platform, giving you a web UI, REST API, and distributed job execution for Ansible playbook automation across your infrastructure. When AWX goes wrong — a crashed job runner, Redis cache failure, or offline Receptor worker node — your CI/CD pipelines and scheduled automation jobs fail silently or queue up indefinitely. Vigilmon gives you continuous visibility into AWX health, from the web UI and REST API down to individual Receptor worker nodes and the PostgreSQL database backing it all.

What You'll Set Up

• AWX web UI and REST API health monitoring
• Job execution success rate and queue depth tracking
• Receptor worker node availability monitoring
• Execution environment pull health checks
• PostgreSQL database connectivity monitoring
• Redis cache health monitoring
• Workflow job success rate tracking

Prerequisites

• Ansible AWX deployed via Kubernetes (AWX Operator) or Docker Compose
• AWX web UI accessible over HTTP/HTTPS
• AWX REST API token for script-based health checks
• A free Vigilmon account

Step 1: Monitor AWX Web UI Health

The AWX web interface is your operators' primary tool for managing job templates, inventories, and credentials. If the UI is down, your team loses visibility into automation status:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter your AWX URL:

   https://awx.yourdomain.com
   

4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

For Kubernetes deployments, also monitor the ingress directly and your load balancer endpoint. If your AWX runs without TLS internally (behind a reverse proxy), monitor the internal service URL at port 80 or 8052.

Step 2: Monitor the AWX REST API Health Endpoint

AWX exposes a /api/v2/ping/ endpoint that CI/CD pipelines and external integrations use to check API availability. This is separate from the web UI and can fail independently:

1. Click Add Monitor → HTTP / HTTPS.
2. Enter the AWX ping endpoint:

   https://awx.yourdomain.com/api/v2/ping/
   

3. Set Check interval to 1 minute.
4. Set Expected HTTP status to 200.
5. Optionally set Expected body contains to ha_capacity_remaining to verify the full response.
6. Click Save.

The /api/v2/ping/ endpoint returns JSON with AWX version, instance status, and HA capacity. If AWX is degraded but partially responding, this check surfaces it before users report broken CI/CD pipelines.

Step 3: Track Job Execution Success Rate

Failed Ansible jobs are the core AWX alert to watch. Use the AWX REST API to check recent job outcomes and ping a Vigilmon heartbeat only when success rate is within threshold:

#!/bin/bash
# /opt/awx/scripts/check-job-health.sh
AWX_HOST="https://awx.yourdomain.com"
AWX_TOKEN="your-api-token-here"
FAILURE_THRESHOLD=3   # max failed jobs in last 15 minutes
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_JOB_HEARTBEAT_ID"

# Get jobs from the last 15 minutes
SINCE=$(date -u -d "15 minutes ago" +"%Y-%m-%dT%H:%M:%SZ" 2>/dev/null || \
        date -u -v-15M +"%Y-%m-%dT%H:%M:%SZ")

FAILED=$(curl -sf \
  -H "Authorization: Bearer ${AWX_TOKEN}" \
  "${AWX_HOST}/api/v2/jobs/?status=failed&created__gte=${SINCE}&page_size=100" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['count'])")

if [ "$FAILED" -le "$FAILURE_THRESHOLD" ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
else
  echo "AWX job failures in last 15 minutes: ${FAILED} (threshold: ${FAILURE_THRESHOLD})" >&2
fi

Schedule this script every 5 minutes:

*/5 * * * * /opt/awx/scripts/check-job-health.sh

Create a Cron Heartbeat monitor in Vigilmon with a 10-minute expected interval. To generate an API token, go to AWX UI → User → Tokens → Add.

Step 4: Monitor Job Queue Depth

AWX queues jobs when execution capacity is exhausted. A growing pending queue means jobs aren't running — either Receptor workers are offline or the AWX task engine has stalled:

#!/bin/bash
# /opt/awx/scripts/check-job-queue.sh
AWX_HOST="https://awx.yourdomain.com"
AWX_TOKEN="your-api-token-here"
PENDING_THRESHOLD=10
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_QUEUE_HEARTBEAT_ID"

PENDING=$(curl -sf \
  -H "Authorization: Bearer ${AWX_TOKEN}" \
  "${AWX_HOST}/api/v2/jobs/?status=pending&page_size=1" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['count'])")

WAITING=$(curl -sf \
  -H "Authorization: Bearer ${AWX_TOKEN}" \
  "${AWX_HOST}/api/v2/jobs/?status=waiting&page_size=1" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['count'])")

TOTAL=$((PENDING + WAITING))

if [ "$TOTAL" -le "$PENDING_THRESHOLD" ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
else
  echo "AWX job queue depth: ${TOTAL} pending/waiting (threshold: ${PENDING_THRESHOLD})" >&2
fi

Step 5: Monitor Receptor Worker Nodes

AWX uses the Receptor mesh network to distribute job execution to remote worker nodes. If a Receptor node goes offline, jobs targeting that network segment fail or hang:

#!/bin/bash
# /opt/awx/scripts/check-receptor-nodes.sh
AWX_HOST="https://awx.yourdomain.com"
AWX_TOKEN="your-api-token-here"
MIN_HEALTHY_NODES=2
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_RECEPTOR_HEARTBEAT_ID"

HEALTHY=$(curl -sf \
  -H "Authorization: Bearer ${AWX_TOKEN}" \
  "${AWX_HOST}/api/v2/instances/?node_state=ready&page_size=100" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['count'])")

if [ "$HEALTHY" -ge "$MIN_HEALTHY_NODES" ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
else
  echo "Only ${HEALTHY} healthy Receptor nodes (expected >= ${MIN_HEALTHY_NODES})" >&2
fi

For individual Receptor worker nodes that you want to monitor specifically, add a TCP port monitor to each worker's Receptor port (default 27199):

1. Click Add Monitor → TCP Port.
2. Enter the worker hostname and port 27199.
3. Set Check interval to 2 minutes.
4. Click Save.

Step 6: Monitor Execution Environment Pull Health

AWX runs Ansible playbooks inside OCI container images called Execution Environments (EEs). If the container registry is unreachable or the EE image pull fails, no jobs can execute:

#!/bin/bash
# /opt/awx/scripts/check-ee-registry.sh
# Check that the EE registry used by AWX is reachable
EE_REGISTRY_HOST="quay.io"  # or your private registry
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_EE_HEARTBEAT_ID"

if curl -sf --max-time 10 "https://${EE_REGISTRY_HOST}/v2/" > /dev/null 2>&1; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
else
  echo "EE registry ${EE_REGISTRY_HOST} unreachable" >&2
fi

For a private registry, authenticate the curl request or use docker manifest inspect to verify a specific EE image is pullable.

Step 7: Monitor PostgreSQL Database Connectivity

AWX stores all its data — job history, inventories, credentials, workflow definitions — in PostgreSQL. Database unavailability causes AWX to stop accepting requests entirely:

#!/bin/bash
# /opt/awx/scripts/check-postgres.sh
PG_HOST="localhost"
PG_PORT="5432"
PG_USER="awx"
PG_DB="awx"
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_PG_HEARTBEAT_ID"

if PGPASSWORD="$AWX_PG_PASSWORD" psql -h "$PG_HOST" -p "$PG_PORT" \
   -U "$PG_USER" -d "$PG_DB" -c "SELECT 1" > /dev/null 2>&1; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
else
  echo "AWX PostgreSQL connectivity check failed" >&2
fi

Also add a TCP port monitor to port 5432 directly from Vigilmon for an external connectivity signal:

1. Click Add Monitor → TCP Port.
2. Enter your database host and port 5432.
3. Set Check interval to 1 minute.
4. Click Save.

Step 8: Monitor Redis Cache Health

AWX uses Redis for its task queue and caching layer. A Redis failure stalls job scheduling — jobs may appear to accept but never actually start:

#!/bin/bash
# /opt/awx/scripts/check-redis.sh
REDIS_HOST="localhost"
REDIS_PORT="6379"
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_REDIS_HEARTBEAT_ID"

if redis-cli -h "$REDIS_HOST" -p "$REDIS_PORT" ping | grep -q "PONG"; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
else
  echo "AWX Redis connectivity check failed" >&2
fi

For Kubernetes deployments, the Redis service name from your AWX Operator install will be something like awx-redis-svc. Adjust REDIS_HOST to the service hostname.

Step 9: Configure Alerting

With all monitors in place, set up alert routing:

1. In Vigilmon, click Alert Contacts → Add Contact.
2. Add your ops team email, Slack #automation-alerts channel webhook, or PagerDuty integration.
3. For critical monitors (web UI, REST API, database), set Alert after 1 failure — these should never go down without immediate notification.
4. For heartbeat monitors (job success rate, queue depth), set Alert after 2 missed pings to tolerate a single missed cron execution.

Recommended thresholds:

| Monitor | Alert Sensitivity | |---|---| | AWX web UI | 1 failure | | AWX REST API /ping | 1 failure | | PostgreSQL TCP port | 1 failure | | Redis TCP port | 1 failure | | Receptor TCP port (per node) | 2 failures | | Job success rate heartbeat | 2 missed pings | | Job queue depth heartbeat | 2 missed pings | | Receptor node count heartbeat | 2 missed pings | | EE registry heartbeat | 3 missed pings |

Conclusion

Ansible AWX is your automation control plane — when it fails, your entire infrastructure automation pipeline fails with it. Vigilmon gives you layered visibility across every critical AWX component: the web UI and REST API your operators depend on, the job execution engine and Receptor workers running playbooks across your infrastructure, and the PostgreSQL and Redis services that underpin it all.

Start with the web UI and REST API monitors (Steps 1–2), add job health and queue depth tracking (Steps 3–4), then layer in Receptor worker monitoring (Step 5) and database/cache health (Steps 7–8). Your Ansible automation infrastructure will have the same uptime visibility as the services it manages.