tutorial

Monitoring ManageIQ with Vigilmon

ManageIQ manages your entire hybrid cloud — VMware, OpenStack, AWS, Azure, OpenShift — and when its workers crash or providers stop refreshing, you lose visibility across your infrastructure. Here's how to monitor ManageIQ application health, worker processes, provider refresh, and database connectivity with Vigilmon.

ManageIQ is the open source cloud management platform that gives you a single pane of glass over VMware vSphere, OpenStack, Amazon AWS, Microsoft Azure, Google Cloud, Red Hat Virtualization, and Kubernetes — discovering VMs, enforcing policies, automating provisioning, and tracking costs across all of them simultaneously. Originally acquired by Red Hat and open sourced in 2012, ManageIQ is the upstream for Red Hat CloudForms and runs in enterprises managing thousands of VMs across multiple clouds. But when a ManageIQ worker crashes, provider inventories go stale. When the database hiccups, the UI freezes. Vigilmon gives you early warning across the full ManageIQ stack.

What You'll Set Up

  • ManageIQ web UI health monitoring
  • Background worker process health checks
  • Provider inventory refresh status alerts
  • PostgreSQL database connectivity monitoring
  • Event processing queue depth monitoring
  • SmartState analysis job success alerts
  • Memcached connectivity checks

Prerequisites

  • ManageIQ Quintero/Radjabov or later installed and configured with at least one provider
  • ManageIQ API accessible (default: https://manageiq-server/api)
  • PostgreSQL accessible from the monitoring host
  • A free Vigilmon account

Step 1: Monitor the ManageIQ Web UI

ManageIQ is a Ruby on Rails application served via the Puma web server behind an Apache or nginx reverse proxy. A crashed Puma worker means administrators lose access to the entire management platform.

Add an HTTP uptime monitor:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter your ManageIQ URL: https://manageiq.yourdomain.com.
  4. Set Check interval to 2 minutes.
  5. Set Expected HTTP status to 200 or 302.
  6. Enable Monitor SSL certificate with a 21 day alert threshold.
  7. Click Save.

For a deeper check, use the ManageIQ API health endpoint:

https://manageiq.yourdomain.com/api

This returns a JSON response confirming the Rails application and API are processing requests:

{
  "name": "ManageIQ",
  "description": "ManageIQ REST API",
  "version": "..."
}

Set the Vigilmon monitor to check /api and look for HTTP 200 with JSON content.

Step 2: Monitor Worker Process Health

ManageIQ runs dozens of background worker processes that handle different responsibilities: EMS Refresh workers pull inventory from cloud providers, Event Handler workers consume VMware/OpenStack events, Policy workers enforce compliance rules, Automate workers execute provisioning workflows, and SmartState workers analyze VM disk images. Each worker type can crash independently.

Create the worker health check script:

#!/bin/bash
# /usr/local/bin/manageiq-workers-check.sh
MIQ_HOST="localhost"
MIQ_USER="admin"
MIQ_PASS="smartvm"  # Default ManageIQ password
VIGILMON_WEBHOOK="https://vigilmon.online/api/webhook/YOUR_WORKERS_WEBHOOK_ID"

# Query the ManageIQ API for worker status
RESPONSE=$(curl -s -u "${MIQ_USER}:${MIQ_PASS}" \
    "https://${MIQ_HOST}/api/servers?expand=resources&attributes=started_workers" \
    2>/dev/null)

if [ $? -ne 0 ] || [ -z "${RESPONSE}" ]; then
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d '{"status": "down", "message": "Cannot reach ManageIQ API to check worker status"}'
    exit 1
fi

# Count workers by status
STARTED=$(echo "${RESPONSE}" | python3 -c "
import sys, json
try:
    d = json.load(sys.stdin)
    workers = []
    for s in d.get('resources', []):
        workers.extend(s.get('started_workers', []))
    running = [w for w in workers if w.get('status') == 'started']
    stopped = [w for w in workers if w.get('status') != 'started']
    print(f'{len(running)} running, {len(stopped)} stopped')
    sys.exit(1 if stopped else 0)
except Exception as e:
    print(f'parse error: {e}')
    sys.exit(2)
" 2>/dev/null)

STATUS_CODE=$?

if [ "${STATUS_CODE}" -ne 0 ]; then
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"down\", \"message\": \"Worker issues: ${STARTED}\"}"
else
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"up\", \"message\": \"Workers healthy: ${STARTED}\"}"
fi

Schedule every 5 minutes:

*/5 * * * * root /usr/local/bin/manageiq-workers-check.sh

Step 3: Monitor Provider Inventory Refresh

ManageIQ's EMS (External Management System) Refresh workers continuously pull inventory from connected providers — VMware vCenter, OpenStack, AWS, Azure, etc. When a provider refresh fails, your ManageIQ inventory becomes stale and policies/automation operate on outdated data.

Create the provider refresh check:

#!/bin/bash
# /usr/local/bin/manageiq-refresh-check.sh
MIQ_HOST="localhost"
MIQ_USER="admin"
MIQ_PASS="smartvm"
MAX_REFRESH_AGE_MINUTES=60  # Alert if provider not refreshed in >60 minutes
VIGILMON_WEBHOOK="https://vigilmon.online/api/webhook/YOUR_REFRESH_WEBHOOK_ID"

python3 << 'PYEOF'
import urllib.request
import urllib.error
import json
import base64
import sys
from datetime import datetime

host = "localhost"
user = "admin"
password = "smartvm"
max_age_min = 60

auth = base64.b64encode(f"{user}:{password}".encode()).decode()
headers = {"Authorization": f"Basic {auth}", "Content-Type": "application/json"}

try:
    req = urllib.request.Request(
        f"https://{host}/api/providers?expand=resources&attributes=last_refresh_date,name,type",
        headers=headers
    )
    # Note: in production add SSL context with proper cert verification
    import ssl
    ctx = ssl.create_default_context()
    ctx.check_hostname = False
    ctx.verify_mode = ssl.CERT_NONE
    
    with urllib.request.urlopen(req, context=ctx, timeout=10) as r:
        data = json.loads(r.read())
    
    stale = []
    for provider in data.get("resources", []):
        name = provider.get("name", "unknown")
        last_refresh = provider.get("last_refresh_date")
        if last_refresh:
            refresh_dt = datetime.fromisoformat(last_refresh.replace("Z", "+00:00"))
            age_min = (datetime.now().astimezone() - refresh_dt).total_seconds() / 60
            if age_min > max_age_min:
                stale.append(f"{name} ({age_min:.0f}min ago)")
    
    if stale:
        print(f"STALE: {'; '.join(stale)}")
        sys.exit(1)
    else:
        print(f"OK: {len(data.get('resources', []))} providers refreshed within {max_age_min}min")
        sys.exit(0)
except Exception as e:
    print(f"ERROR: {e}")
    sys.exit(2)
PYEOF

EXIT_CODE=$?
MESSAGE=$(python3 /usr/local/bin/manageiq-refresh-check.sh 2>/dev/null | tail -1)

if [ "${EXIT_CODE}" -ne 0 ]; then
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"down\", \"message\": \"Provider refresh issue: ${MESSAGE}\"}"
else
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"up\", \"message\": \"${MESSAGE}\"}"
fi

Step 4: Monitor PostgreSQL Database Health

ManageIQ stores all inventory data, event history, automate workflows, and configuration in PostgreSQL. A database outage causes complete ManageIQ failure — the UI becomes unresponsive and all background workers stop processing.

Add TCP and query-level monitors:

  1. In Vigilmon, click Add Monitor → TCP Port.
  2. Enter your ManageIQ server hostname and port 5432.
  3. Set Check interval to 1 minute.
  4. Click Save.

For query-level health and replication lag (if using streaming replication for HA):

#!/bin/bash
# /usr/local/bin/manageiq-db-check.sh
DB_HOST="localhost"
DB_PORT="5432"
DB_NAME="vmdb_production"
DB_USER="root"
VIGILMON_WEBHOOK="https://vigilmon.online/api/webhook/YOUR_DB_WEBHOOK_ID"

START=$(date +%s%3N)
RESULT=$(psql -h "${DB_HOST}" -p "${DB_PORT}" -U "${DB_USER}" -d "${DB_NAME}" \
    -c "SELECT 1;" -t -q 2>&1)
EXIT_CODE=$?
END=$(date +%s%3N)
LATENCY=$((END - START))

if [ "${EXIT_CODE}" -ne 0 ]; then
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"down\", \"message\": \"DB query failed: ${RESULT}\"}"
    exit 1
fi

# Check replication lag if this is a replica
REPL_LAG=$(psql -h "${DB_HOST}" -p "${DB_PORT}" -U "${DB_USER}" -d "${DB_NAME}" \
    -c "SELECT EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp()))::INT;" \
    -t -q 2>/dev/null | tr -d ' ')

if [ -n "${REPL_LAG}" ] && [ "${REPL_LAG}" -gt 60 ]; then
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"down\", \"message\": \"DB replication lag: ${REPL_LAG}s (query: ${LATENCY}ms)\"}"
else
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"up\", \"message\": \"DB healthy, query: ${LATENCY}ms\"}"
fi

Step 5: Monitor Event Processing and Automate Queue

ManageIQ consumes events from VMware vCenter (via VMRC events), OpenStack (via RabbitMQ), and AWS (via CloudTrail/CloudWatch events). If the event queue backs up, policy enforcement and automate workflows fall behind real-world changes in your infrastructure.

Create the queue depth monitor:

#!/bin/bash
# /usr/local/bin/manageiq-queue-check.sh
DB_HOST="localhost"
DB_NAME="vmdb_production"
DB_USER="root"
EVENT_QUEUE_THRESHOLD=500    # Alert if event queue depth exceeds 500
AUTOMATE_THRESHOLD=100       # Alert if automate queue depth exceeds 100
VIGILMON_WEBHOOK="https://vigilmon.online/api/webhook/YOUR_QUEUE_WEBHOOK_ID"

# Query the ManageIQ MiqQueue table for queue depth
QUEUE_STATS=$(psql -h "${DB_HOST}" -U "${DB_USER}" -d "${DB_NAME}" -t -q << 'SQL'
SELECT 
    queue_name,
    COUNT(*) as depth
FROM miq_queue 
WHERE state = 'ready'
GROUP BY queue_name
ORDER BY depth DESC;
SQL
)

EVENT_DEPTH=$(echo "${QUEUE_STATS}" | grep "ems_event\|event" | awk '{sum += $NF} END {print sum+0}')
AUTOMATE_DEPTH=$(echo "${QUEUE_STATS}" | grep "automate\|automation" | awk '{sum += $NF} END {print sum+0}')
TOTAL_DEPTH=$(echo "${QUEUE_STATS}" | awk '{sum += $NF} END {print sum+0}')

if [ "${EVENT_DEPTH}" -gt "${EVENT_QUEUE_THRESHOLD}" ] || [ "${AUTOMATE_DEPTH}" -gt "${AUTOMATE_THRESHOLD}" ]; then
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"down\", \"message\": \"Queue backlog: events=${EVENT_DEPTH}, automate=${AUTOMATE_DEPTH}, total=${TOTAL_DEPTH}\"}"
else
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"up\", \"message\": \"Queue healthy: total=${TOTAL_DEPTH} pending tasks\"}"
fi

Step 6: Monitor Memcached Connectivity

ManageIQ uses Memcached for session caching. If Memcached becomes unavailable, users are logged out of active sessions and the UI becomes sluggish as it handles uncached requests.

Add a TCP monitor for Memcached:

  1. In Vigilmon, click Add Monitor → TCP Port.
  2. Enter your ManageIQ server hostname and port 11211 (default Memcached port).
  3. Set Check interval to 2 minutes.
  4. Click Save.

For a connectivity test at the protocol level:

#!/bin/bash
# /usr/local/bin/manageiq-memcached-check.sh
MEMCACHED_HOST="localhost"
MEMCACHED_PORT="11211"
VIGILMON_WEBHOOK="https://vigilmon.online/api/webhook/YOUR_MEMCACHED_WEBHOOK_ID"

# Send a stats command to Memcached
STATS=$(echo "stats" | nc -w 2 "${MEMCACHED_HOST}" "${MEMCACHED_PORT}" 2>/dev/null)

if echo "${STATS}" | grep -q "^STAT"; then
    UPTIME=$(echo "${STATS}" | grep "^STAT uptime" | awk '{print $3}')
    CURR_ITEMS=$(echo "${STATS}" | grep "^STAT curr_items" | awk '{print $3}')
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"up\", \"message\": \"Memcached up ${UPTIME}s, ${CURR_ITEMS} cached items\"}"
else
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d '{"status": "down", "message": "Memcached not responding — session cache unavailable"}'
fi

Step 7: Monitor SmartState Analysis Success Rate

ManageIQ's SmartState Analysis feature scans VM disk images offline to inventory installed packages, running services, and user accounts without requiring an agent in the VM. SmartState jobs run on a schedule and failures mean your VM inventory data goes stale.

Create the SmartState health check:

#!/bin/bash
# /usr/local/bin/manageiq-smartstate-check.sh
MIQ_HOST="localhost"
MIQ_USER="admin"
MIQ_PASS="smartvm"
VIGILMON_WEBHOOK="https://vigilmon.online/api/webhook/YOUR_SMARTSTATE_WEBHOOK_ID"

# Query recent SmartState jobs via the API
python3 << 'PYEOF'
import urllib.request
import json
import base64
import sys
import ssl
from datetime import datetime, timedelta

host, user, password = "localhost", "admin", "smartvm"
auth = base64.b64encode(f"{user}:{password}".encode()).decode()
headers = {"Authorization": f"Basic {auth}"}

ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

try:
    # Get recent scan requests from the last 24 hours
    req = urllib.request.Request(
        f"https://{host}/api/scan_requests?expand=resources&attributes=status,created_on,target_name",
        headers=headers
    )
    with urllib.request.urlopen(req, context=ctx, timeout=10) as r:
        data = json.loads(r.read())
    
    now = datetime.utcnow()
    recent = [
        r for r in data.get("resources", [])
        if r.get("created_on") and 
        (now - datetime.fromisoformat(r["created_on"].replace("Z", ""))).total_seconds() < 86400
    ]
    
    if not recent:
        print("OK: No SmartState scans in last 24h")
        sys.exit(0)
    
    failed = [r for r in recent if r.get("status") in ("Error", "Failed")]
    success = [r for r in recent if r.get("status") in ("Finished", "Ok")]
    
    if failed:
        rate = len(failed) / len(recent) * 100
        names = ", ".join(r.get("target_name", "?") for r in failed[:3])
        print(f"FAIL: {len(failed)}/{len(recent)} ({rate:.0f}%) SmartState scans failed: {names}")
        sys.exit(1)
    else:
        print(f"OK: {len(success)}/{len(recent)} SmartState scans succeeded in last 24h")
        sys.exit(0)
except Exception as e:
    print(f"ERROR: {e}")
    sys.exit(2)
PYEOF

EXIT_CODE=$?
MESSAGE=$(python3 /usr/local/bin/miq-ss.py 2>/dev/null | tail -1)

if [ "${EXIT_CODE}" -ne 0 ]; then
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"down\", \"message\": \"${MESSAGE}\"}"
else
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"up\", \"message\": \"${MESSAGE}\"}"
fi

Step 8: Configure Alert Channels

  1. In Vigilmon, go to Alert Channels and connect email, Slack, or PagerDuty.
  2. For the web UI monitor, set Consecutive failures before alert to 2 — ManageIQ occasionally restarts Puma workers and recovers in under a minute.
  3. For the database TCP monitor and worker health, alert immediately on first failure.
  4. For queue depth monitors, alert immediately when thresholds are exceeded — queue backlogs indicate worker overload that won't self-resolve.
  5. For Memcached, alert immediately — session loss impacts all active users.
  6. Set a Maintenance window during ManageIQ version upgrades, which require a full application restart.

Summary

| Monitor | Type | What It Catches | |---|---|---| | ManageIQ web UI | HTTP/HTTPS | Application crash or Puma failure | | Background workers | Webhook push | EMS, Policy, Automate worker crash | | PostgreSQL port 5432 | TCP port | Database unavailable | | DB query health | Webhook push | Query latency, replication lag | | Provider refresh | Webhook push | Stale inventory from cloud providers | | Event/Automate queue | Webhook push | Worker overload — backlog growing | | Memcached port 11211 | TCP port | Session cache down | | SmartState analysis | Webhook push | VM disk scan failures |

ManageIQ is the control plane for your hybrid cloud — when it's healthy, your VMs are inventoried, policies enforced, and provisioning automated. With Vigilmon watching the web UI, all worker types, provider refresh cycles, database health, and queue depths, you know the moment the control plane starts degrading rather than finding out when a VM provisioning workflow silently fails.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles