Monitoring Apache Qpid with Vigilmon

Apache Qpid is an open source implementation of the Advanced Message Queuing Protocol (AMQP), providing both a Java broker (Qpid Broker-J) for enterprise messaging and a high-performance C-based message router (Qpid Dispatch Router) for large-scale messaging fabrics. When Qpid goes silent — broker crash, queue overflow, or a broken Dispatch Router mesh — your downstream services queue up failures with no visibility into why. Vigilmon gives you continuous health checks on both broker and router, so you catch AMQP infrastructure problems before they cascade.

What You'll Set Up

Qpid Broker-J process health and REST management API monitoring
Queue depth monitoring with alerts for consumer lag
AMQP connection count tracking
BDB persistence store health
JVM heap and GC health monitoring
Qpid Dispatch Router process and inter-router mesh health
Dead letter queue accumulation alerts
High availability failover health checks

Prerequisites

Apache Qpid Broker-J 9.x+ or Qpid Dispatch Router 1.x+ running on a server or VM
Qpid Broker-J REST management API accessible (default port 8080)
A free Vigilmon account

Step 1: Monitor Qpid Broker-J Process Health

Qpid Broker-J runs as a JVM process and exposes a REST management API. A broker crash silently stops all AMQP message delivery. Monitor the management API endpoint to detect broker failures:

Log in to vigilmon.online and click Add Monitor.
Set Type to HTTP / HTTPS.

Enter the Qpid REST management API health URL:

http://your-qpid-host:8080/api/latest/broker

Set Check interval to 1 minute.
Set Expected HTTP status to 200.
Add a Basic Auth header if your broker has authentication enabled (username/password configured in config.json).
Click Save.

The /api/latest/broker endpoint returns broker state including ACTIVE status. If the broker process dies, this endpoint becomes unreachable — Vigilmon will alert you within one check interval.

To also verify the AMQP port itself is accepting connections, add a second monitor:

Click Add Monitor → TCP Port.
Enter your server hostname and AMQP port 5672 (or 5671 for AMQP over TLS).
Set Check interval to 1 minute.
Click Save.

Step 2: Monitor Queue Depth for Consumer Lag

Messages accumulating in queues without being consumed indicate a consumer failure or processing bottleneck. Qpid Broker-J exposes per-queue statistics via its REST API. Create a script that queries queue depth and pings Vigilmon:

#!/bin/bash
# /opt/qpid/scripts/check-queue-depth.sh
BROKER_HOST="localhost"
BROKER_PORT="8080"
QUEUE_NAME="orders"
DEPTH_THRESHOLD=1000
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_ID"

DEPTH=$(curl -sf -u admin:password \
  "http://${BROKER_HOST}:${BROKER_PORT}/api/latest/queue/${QUEUE_NAME}" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['queueDepthMessages'])")

if [ "$DEPTH" -lt "$DEPTH_THRESHOLD" ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
else
  echo "Queue ${QUEUE_NAME} depth ${DEPTH} exceeds threshold ${DEPTH_THRESHOLD}" >&2
fi

Set this up as a cron job every minute:

* * * * * /opt/qpid/scripts/check-queue-depth.sh

In Vigilmon, create the matching cron heartbeat:

Click Add Monitor → Cron Heartbeat.
Set expected ping interval to 2 minutes.
Copy the heartbeat URL into the script above.
Click Save.

The monitor goes down if queue depth exceeds your threshold — the script simply stops pinging the heartbeat.

Step 3: Track AMQP Connection Count

A sudden drop in AMQP client connections (clients disconnecting en masse) can indicate a network partition, client crash, or broker configuration change. Monitor connection count via the Qpid REST API:

#!/bin/bash
# /opt/qpid/scripts/check-connections.sh
BROKER_HOST="localhost"
BROKER_PORT="8080"
MIN_CONNECTIONS=5
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_CONNECTION_HEARTBEAT_ID"

CONNECTIONS=$(curl -sf -u admin:password \
  "http://${BROKER_HOST}:${BROKER_PORT}/api/latest/connection" \
  | python3 -c "import sys,json; print(len(json.load(sys.stdin)))")

if [ "$CONNECTIONS" -ge "$MIN_CONNECTIONS" ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
else
  echo "Only ${CONNECTIONS} AMQP connections (expected >= ${MIN_CONNECTIONS})" >&2
fi

Schedule as a cron job and create a corresponding Cron Heartbeat monitor in Vigilmon with a 2-minute expected interval. Adjust MIN_CONNECTIONS to match your baseline number of producer/consumer connections.

Step 4: Monitor BDB Persistence Store Health

Qpid Broker-J persists messages to Berkeley DB Java Edition (BDB). When the BDB store fills to its configured limit, the broker stops accepting new messages — a silent, catastrophic failure mode. Check BDB store utilization:

#!/bin/bash
# /opt/qpid/scripts/check-bdb-store.sh
BROKER_HOST="localhost"
BROKER_PORT="8080"
STORE_THRESHOLD_PERCENT=80
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_BDB_HEARTBEAT_ID"

STORE_INFO=$(curl -sf -u admin:password \
  "http://${BROKER_HOST}:${BROKER_PORT}/api/latest/virtualhostnode/default/virtualhost/default")

USED=$(echo "$STORE_INFO" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('bytesEvicted',0))")
# If store statistics are available, compare used vs configured max
# Simplified: ping heartbeat if broker is ACTIVE (not in STOPPED/ERRORED state)
STATE=$(echo "$STORE_INFO" | python3 -c "import sys,json; print(json.load(sys.stdin).get('state','UNKNOWN'))")

if [ "$STATE" = "ACTIVE" ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
else
  echo "VirtualHost state: ${STATE}" >&2
fi

For more precise BDB disk usage, check the BDB store directory size against your storeOverfullSize broker configuration:

STORE_DIR="/var/lib/qpid/default/default"
MAX_SIZE_BYTES=10737418240  # 10 GB — match your storeOverfullSize config
USED_BYTES=$(du -sb "$STORE_DIR" | cut -f1)
PERCENT=$((USED_BYTES * 100 / MAX_SIZE_BYTES))

if [ "$PERCENT" -lt "$STORE_THRESHOLD_PERCENT" ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
fi

Step 5: Monitor JVM Heap and GC Health

Qpid Broker-J is a Java application. JVM heap exhaustion causes broker crashes; long GC pauses cause AMQP timeouts. Enable JMX or use jstat to check heap usage:

#!/bin/bash
# /opt/qpid/scripts/check-jvm.sh
QPID_PID=$(pgrep -f "org.apache.qpid.server.Main")
HEAP_THRESHOLD_PERCENT=85
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_JVM_HEARTBEAT_ID"

if [ -z "$QPID_PID" ]; then
  echo "Qpid Broker-J process not found" >&2
  exit 1
fi

# Get heap usage via jstat (requires same JDK as broker)
HEAP_INFO=$(jstat -gc "$QPID_PID" | tail -1)
EDEN_USED=$(echo "$HEAP_INFO" | awk '{print $3}')
EDEN_CAP=$(echo "$HEAP_INFO" | awk '{print $4}')
OLD_USED=$(echo "$HEAP_INFO" | awk '{print $8}')
OLD_CAP=$(echo "$HEAP_INFO" | awk '{print $9}')

TOTAL_USED=$(echo "$EDEN_USED + $OLD_USED" | bc)
TOTAL_CAP=$(echo "$EDEN_CAP + $OLD_CAP" | bc)
HEAP_PERCENT=$(echo "scale=0; $TOTAL_USED * 100 / $TOTAL_CAP" | bc)

if [ "$HEAP_PERCENT" -lt "$HEAP_THRESHOLD_PERCENT" ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
else
  echo "JVM heap at ${HEAP_PERCENT}% (threshold: ${HEAP_THRESHOLD_PERCENT}%)" >&2
fi

Alternatively, expose JMX metrics via Prometheus JMX Exporter and add an HTTP monitor pointing at the /metrics endpoint to verify the metrics exporter itself is alive.

Step 6: Monitor Qpid Dispatch Router Health

If you use Qpid Dispatch Router as your messaging fabric (common in Eclipse Hono and large IoT deployments), monitor the router process and its management interface:

Click Add Monitor → TCP Port.
Enter your Dispatch Router host and management port 5673 (AMQP management).
Set Check interval to 1 minute.
Click Save.

For inter-router connection mesh health, use the qdstat tool:

#!/bin/bash
# /opt/qpid/scripts/check-dispatch-router.sh
ROUTER_HOST="localhost"
ROUTER_PORT="5673"
MIN_ROUTER_LINKS=2
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_ROUTER_HEARTBEAT_ID"

# Count active inter-router connections
ROUTER_CONNECTIONS=$(qdstat -b "amqp://${ROUTER_HOST}:${ROUTER_PORT}" -c 2>/dev/null \
  | grep -c "inter-router" || echo "0")

if [ "$ROUTER_CONNECTIONS" -ge "$MIN_ROUTER_LINKS" ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
else
  echo "Inter-router connections: ${ROUTER_CONNECTIONS} (expected >= ${MIN_ROUTER_LINKS})" >&2
fi

Schedule this as a cron job with a 2-minute expected heartbeat interval.

Step 7: Monitor Dead Letter Queue Accumulation

Undeliverable messages end up in Qpid dead letter queues (DLQs). A growing DLQ indicates persistent delivery failures that need operator attention:

#!/bin/bash
# /opt/qpid/scripts/check-dlq.sh
BROKER_HOST="localhost"
BROKER_PORT="8080"
DLQ_NAME="orders.DLQ"
DLQ_THRESHOLD=10
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_DLQ_HEARTBEAT_ID"

DLQ_DEPTH=$(curl -sf -u admin:password \
  "http://${BROKER_HOST}:${BROKER_PORT}/api/latest/queue/${DLQ_NAME}" \
  | python3 -c "import sys,json; print(json.load(sys.stdin).get('queueDepthMessages', 0))")

if [ "$DLQ_DEPTH" -lt "$DLQ_THRESHOLD" ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
else
  echo "DLQ ${DLQ_NAME} has ${DLQ_DEPTH} undeliverable messages" >&2
fi

Step 8: Configure Alerting

With monitors in place, configure alerts so the right people are notified:

In Vigilmon, click Alert Contacts → Add Contact.
Add your on-call email, Slack webhook, or PagerDuty integration key.
For each monitor, open its settings and assign the alert contact.
Set Alert after to 2 consecutive failures to avoid false alarms from transient checks.

Recommended thresholds:

| Monitor | Alert When | |---|---| | Broker REST API | Unreachable for 2+ minutes | | AMQP TCP port | Unreachable for 1+ minute | | Queue depth heartbeat | No ping for 2+ minutes | | Connection count heartbeat | No ping for 2+ minutes | | BDB store heartbeat | No ping for 2+ minutes | | JVM heap heartbeat | No ping for 2+ minutes | | Dispatch Router TCP | Unreachable for 1+ minute | | Inter-router mesh heartbeat | No ping for 3+ minutes | | DLQ accumulation heartbeat | No ping for 5+ minutes |

Conclusion

Apache Qpid's silent failure modes — a crashed broker, a clogged queue, a broken router mesh — can halt all AMQP messaging in your system before any application-layer error surfaces. With Vigilmon monitoring the Qpid REST API, AMQP ports, queue depths, BDB persistence health, JVM vitals, and Dispatch Router mesh integrity, you get early warning before messaging failures cascade into application outages.

Start with the broker REST API and AMQP port monitors (Steps 1–2), add queue depth and JVM health (Steps 3–5), then layer in Dispatch Router and DLQ monitoring if your deployment uses them. Your AMQP messaging infrastructure will have the same observability as your HTTP services.