tutorial

Monitoring Apache RocketMQ with Vigilmon

Apache RocketMQ is a distributed messaging platform used at Alibaba-scale for billions of messages per day. Here's how to monitor RocketMQ NameServer health, broker availability, consumer group lag, CommitLog write performance, and master-slave replication with Vigilmon.

Apache RocketMQ is a distributed messaging and streaming platform developed at Alibaba and donated to the Apache Software Foundation in 2016. It handles billions of messages daily at Alibaba scale — including the traffic spikes during Singles' Day — and is used globally for event-driven architectures, transaction messaging, delayed delivery, and pub-sub patterns. When you self-host RocketMQ, you're operating a multi-component cluster: NameServer nodes for routing, Broker nodes for message storage (writing to an append-only CommitLog), Producers sending messages, and Consumers pulling from ConsumeQueues. Any broker failure causes topic unavailability; NameServer failure means producers and consumers can't discover broker locations; CommitLog write slowdowns cause message delivery latency spikes. Vigilmon monitors every component of the RocketMQ cluster with heartbeat-based checks that catch failures before your message processing pipelines stall.

What You'll Set Up

  • NameServer health monitors per node
  • Broker health and master/slave role monitoring
  • Consumer group lag alerting (threshold: >10,000 messages)
  • CommitLog write performance monitoring
  • Master-slave replication lag check (threshold: >100MB)
  • Disk space utilization monitoring per broker
  • Message delivery latency checks
  • Alert channels for the platform team

Prerequisites

  • Apache RocketMQ 4.x or 5.x cluster deployed
  • mqadmin CLI tool available on the monitoring host
  • RocketMQ management API accessible (default port 9876 for NameServer, 10911 for broker)
  • A free Vigilmon account

Step 1: Monitor NameServer Health

The NameServer is RocketMQ's cluster registry — producers and consumers query it to discover which brokers host which topics. Unlike ZooKeeper, NameServer nodes are stateless and don't communicate with each other; each broker heartbeats to all NameServer nodes independently. Losing a NameServer node reduces registry redundancy; losing all NameServer nodes prevents any new connections.

Create a heartbeat monitor in Vigilmon for each NameServer node:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to Cron / Heartbeat.
  3. Name it RocketMQ NameServer ns1 (one per node).
  4. Set Expected interval to 60 seconds.
  5. Set Alert after missing to 2 heartbeats.
  6. Copy the generated heartbeat URL.

Add a cron job on the monitoring host (or the NameServer host itself):

#!/bin/bash
# /usr/local/bin/rocketmq-nameserver-check.sh

NS_HOST="nameserver1.yourdomain.com"
NS_PORT="9876"
HB_URL="https://hb.vigilmon.online/YOUR_NS1_HB_ID"
ROCKETMQ_HOME="/opt/rocketmq"

# Use mqadmin to query cluster info via this NameServer
RESULT=$("${ROCKETMQ_HOME}/bin/mqadmin" clusterList \
  -n "${NS_HOST}:${NS_PORT}" 2>/dev/null | grep -c "BrokerName")

if [ "${RESULT:-0}" -gt 0 ]; then
  curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
fi

Repeat for each NameServer node with separate heartbeat monitor IDs. Schedule:

* * * * * rocketmq /usr/local/bin/rocketmq-nameserver-check.sh

Step 2: Monitor Broker Health and Role

Brokers are the core message storage components. RocketMQ supports master-slave replication for HA — a master broker handles writes while slave brokers replicate and serve reads. A master broker failure makes all topic partitions (message queues) on that broker unavailable for writes.

#!/bin/bash
# /usr/local/bin/rocketmq-broker-check.sh

NS_ADDR="nameserver1.yourdomain.com:9876"
BROKER_NAME="broker-a"  # repeat per broker
HB_URL="https://hb.vigilmon.online/YOUR_BROKER_HB_ID"
ROCKETMQ_HOME="/opt/rocketmq"

# Get broker status — checks both connectivity and role
BROKER_INFO=$("${ROCKETMQ_HOME}/bin/mqadmin" brokerStatus \
  -n "$NS_ADDR" -b "$BROKER_NAME" 2>/dev/null)

if [ $? -ne 0 ] || [ -z "$BROKER_INFO" ]; then
  # Broker unreachable — don't send heartbeat
  exit 1
fi

# Verify master is in the expected role
ROLE=$(echo "$BROKER_INFO" | grep -i "brokerRole" | awk -F= '{print $2}' | tr -d ' ')
if [ "$ROLE" = "ASYNC_MASTER" ] || [ "$ROLE" = "SYNC_MASTER" ]; then
  curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
fi

Create a separate heartbeat monitor per broker (RocketMQ Broker broker-a, RocketMQ Broker broker-b). Adjust the role check to match your configuration — if the broker is expected to be a slave, check for SLAVE instead.

Step 3: Monitor Consumer Group Lag

Consumer group lag (message backlog) is the primary signal that consumers are falling behind producers. A lag exceeding 10,000 messages on any topic/queue indicates consumers are unable to keep up — whether due to consumer bugs, downstream slowness, or a dead consumer process.

#!/bin/bash
# /usr/local/bin/rocketmq-lag-check.sh

NS_ADDR="nameserver1.yourdomain.com:9876"
LAG_THRESHOLD=10000
HB_URL="https://hb.vigilmon.online/YOUR_LAG_HB_ID"
ROCKETMQ_HOME="/opt/rocketmq"
CONSUMER_GROUP="${ROCKETMQ_CONSUMER_GROUP:-}"  # set to check a specific group, or leave empty to check all

check_group() {
  local GROUP="$1"
  local STATS
  STATS=$("${ROCKETMQ_HOME}/bin/mqadmin" consumerProgress \
    -n "$NS_ADDR" -g "$GROUP" 2>/dev/null)
  
  if [ $? -ne 0 ]; then
    return 1
  fi

  # Sum lag across all topic/queue combinations for this group
  local TOTAL_LAG
  TOTAL_LAG=$(echo "$STATS" | awk '/NOTCONSUMED/{sum += $NF} END {print sum+0}')
  
  if [ "${TOTAL_LAG:-0}" -gt "$LAG_THRESHOLD" ]; then
    return 1
  fi
  return 0
}

if [ -n "$CONSUMER_GROUP" ]; then
  check_group "$CONSUMER_GROUP" && curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
else
  # Check all consumer groups
  GROUPS=$("${ROCKETMQ_HOME}/bin/mqadmin" consumerProgress -n "$NS_ADDR" 2>/dev/null | \
    grep -oE "^[A-Za-z0-9_-]+" | sort -u)
  
  ALL_OK=1
  while IFS= read -r GROUP; do
    [ -z "$GROUP" ] && continue
    check_group "$GROUP" || ALL_OK=0
  done <<< "$GROUPS"
  
  [ "$ALL_OK" -eq 1 ] && curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
fi

Create a heartbeat RocketMQ Consumer Lag OK with a 2-minute interval and schedule this script every 2 minutes.

Step 4: Monitor CommitLog Write Performance

RocketMQ persists all messages to an append-only CommitLog on each broker. Write latency above 100ms means message persistence is slow — producers will see higher sendMessage latencies, and under sustained load, the producer SDK may time out.

#!/bin/bash
# /usr/local/bin/rocketmq-commitlog-check.sh

NS_ADDR="nameserver1.yourdomain.com:9876"
BROKER_ADDR="broker1.yourdomain.com:10911"
WRITE_LATENCY_THRESHOLD_MS=100
HB_URL="https://hb.vigilmon.online/YOUR_COMMITLOG_HB_ID"
ROCKETMQ_HOME="/opt/rocketmq"

# Get broker runtime stats including CommitLog write latency
STATS=$("${ROCKETMQ_HOME}/bin/mqadmin" brokerStatus \
  -n "$NS_ADDR" -b "broker-a" 2>/dev/null)

if [ $? -ne 0 ]; then
  exit 1
fi

# Extract putMessageAverageSize and putTps as proxies for write health
# RocketMQ brokerStatus includes putMessageDistributeTime which shows write latency distribution
PUT_LATENCY=$(echo "$STATS" | grep -i "putMessageDistributeTime" | \
  grep -oP '\d+ms=\d+' | awk -F'[=ms]' 'BEGIN{total=0; count=0} 
    {ms=$1; n=$3; total+=ms*n; count+=n} 
    END{if(count>0) printf "%d", total/count; else print 0}')

if [ "${PUT_LATENCY:-0}" -lt "$WRITE_LATENCY_THRESHOLD_MS" ]; then
  curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
fi

Create a heartbeat RocketMQ CommitLog Write OK with a 60-second interval.

Step 5: Monitor Master-Slave Replication Lag

RocketMQ's HA relies on slaves replicating the master's CommitLog. If the replication lag exceeds 100MB, the slave is significantly behind and a master failure would result in that much message loss (in async replication) or production stall (in sync replication mode).

#!/bin/bash
# /usr/local/bin/rocketmq-replication-check.sh

NS_ADDR="nameserver1.yourdomain.com:9876"
LAG_THRESHOLD_BYTES=$((100 * 1024 * 1024))  # 100MB
HB_URL="https://hb.vigilmon.online/YOUR_REPL_HB_ID"
ROCKETMQ_HOME="/opt/rocketmq"

# Compare master and slave CommitLog offsets
MASTER_OFFSET=$("${ROCKETMQ_HOME}/bin/mqadmin" brokerStatus \
  -n "$NS_ADDR" -b "broker-a" 2>/dev/null | \
  grep "commitLogMaxOffset" | awk -F= '{print $2}' | tr -d ' ')

SLAVE_OFFSET=$("${ROCKETMQ_HOME}/bin/mqadmin" brokerStatus \
  -n "$NS_ADDR" -b "broker-a-slave" 2>/dev/null | \
  grep "commitLogMaxOffset" | awk -F= '{print $2}' | tr -d ' ')

if [ -z "$MASTER_OFFSET" ] || [ -z "$SLAVE_OFFSET" ]; then
  # Can't read offsets — don't send heartbeat
  exit 1
fi

LAG=$((MASTER_OFFSET - SLAVE_OFFSET))
LAG=${LAG#-}  # absolute value

if [ "$LAG" -lt "$LAG_THRESHOLD_BYTES" ]; then
  curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
fi

Create a heartbeat RocketMQ Replication Lag OK with a 60-second interval.

Step 6: Monitor Disk Space on Broker Store Path

RocketMQ's CommitLog is an append-only log — it fills disk at the rate of your message throughput. At 80% disk usage, RocketMQ enters a "no write" mode to prevent disk exhaustion. At 90%, it may start deleting expired CommitLog segments aggressively. Running out of disk entirely stops the broker.

#!/bin/bash
# /usr/local/bin/rocketmq-disk-check.sh

STORE_PATH="${ROCKETMQ_STORE_PATH:-/data/rocketmq/store}"
DISK_THRESHOLD_PCT=80
HB_URL="https://hb.vigilmon.online/YOUR_DISK_HB_ID"

# Get disk usage percentage for the RocketMQ store path
DISK_PCT=$(df -P "$STORE_PATH" 2>/dev/null | awk 'NR==2 {gsub(/%/, "", $5); print $5}')

if [ -z "$DISK_PCT" ]; then
  exit 1
fi

if [ "$DISK_PCT" -lt "$DISK_THRESHOLD_PCT" ]; then
  curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
fi

Create a heartbeat RocketMQ Disk Space OK with a 5-minute interval. Set the alert to fire after 2 consecutive missed checks — disk usage changes slowly and one missed check may be a timing issue.

Step 7: Monitor Message Delivery Latency

RocketMQ targets low delivery latency for most workloads. End-to-end p95 delivery latency above 500ms on a topic indicates broker write slowdowns, consumer processing bottlenecks, or network issues between producers and brokers.

Create an HTTP monitor using RocketMQ's management console health endpoint if you have the RocketMQ Dashboard deployed:

  1. In Vigilmon, click Add Monitor → HTTP / HTTPS.
  2. URL: http://rocketmq-dashboard.yourdomain.com/ (or your management console URL).
  3. Set Check interval to 1 minute.
  4. Set Expected HTTP status to 200.
  5. Under Response time threshold, set 2000ms as the alert threshold.
  6. Click Save.

For latency tracking without the dashboard, instrument a probe producer/consumer that sends a test message and measures round-trip time:

#!/bin/bash
# /usr/local/bin/rocketmq-latency-probe.sh
# Requires a test topic "latency-probe" to be created

NS_ADDR="nameserver1.yourdomain.com:9876"
LATENCY_THRESHOLD_MS=500
HB_URL="https://hb.vigilmon.online/YOUR_LATENCY_HB_ID"
ROCKETMQ_HOME="/opt/rocketmq"

START=$(date +%s%3N)
"${ROCKETMQ_HOME}/bin/tools.sh" org.apache.rocketmq.example.quickstart.Producer \
  -n "$NS_ADDR" 2>/dev/null | grep -q "SendResult"
STATUS=$?
END=$(date +%s%3N)

LATENCY=$((END - START))

if [ "$STATUS" -eq 0 ] && [ "$LATENCY" -lt "$LATENCY_THRESHOLD_MS" ]; then
  curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
fi

Step 8: Monitor RocketMQ 5.x Controller (if applicable)

RocketMQ 5.x introduced a new controller mode (similar to Kafka's KRaft) that replaces the traditional master-slave model with dynamic leader election. If no controller leader is elected, no broker can take the master role, and topic writes halt.

#!/bin/bash
# /usr/local/bin/rocketmq-controller-check.sh
# RocketMQ 5.x only

CONTROLLER_ADDR="controller1.yourdomain.com:9878"
HB_URL="https://hb.vigilmon.online/YOUR_CTRL_HB_ID"
ROCKETMQ_HOME="/opt/rocketmq"

RESULT=$("${ROCKETMQ_HOME}/bin/mqadmin" getControllerMetaData \
  -a "${CONTROLLER_ADDR}" 2>/dev/null | grep -i "masterAddress")

if [ -n "$RESULT" ]; then
  curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
fi

Create a heartbeat RocketMQ Controller Leader with a 60-second interval. Skip this check if you're running RocketMQ 4.x with traditional master-slave replication.

Step 9: Configure Alert Channels

In Vigilmon, navigate to Settings → Alert Channels and set up:

  1. PagerDuty / on-call — for NameServer down, broker master down, and consumer lag exceeding threshold. These are P1 incidents that stop message delivery.
  2. Slack — for CommitLog write latency, replication lag, and disk space warnings. These are leading indicators.
  3. Email — for disk space and delivery latency (non-urgent, needs capacity planning review).

Alert channel assignment:

| Monitor | Channel | |---|---| | RocketMQ NameServer ns1/ns2 | PagerDuty + Slack | | RocketMQ Broker broker-a | PagerDuty + Slack | | RocketMQ Consumer Lag OK | PagerDuty + Slack | | RocketMQ CommitLog Write OK | Slack | | RocketMQ Replication Lag OK | Slack + Email | | RocketMQ Disk Space OK | Slack + Email | | RocketMQ Latency Probe | Slack | | RocketMQ Controller Leader | PagerDuty + Slack |

Step 10: Alert Timing Reference

| Monitor | Interval | Alert after | |---|---|---| | NameServer (per node) | 60s | 2 missed | | Broker (per node) | 60s | 2 missed | | Consumer Lag OK | 2m | 2 missed | | CommitLog Write OK | 60s | 3 missed | | Replication Lag OK | 60s | 3 missed | | Disk Space OK | 5m | 2 missed | | Latency Probe | 60s | 3 missed | | Controller Leader | 60s | 2 missed |

Conclusion

Apache RocketMQ delivers exceptional throughput and low latency for distributed messaging — Alibaba uses it to handle billions of messages per day during peak load events. But that capability comes with operational depth: NameServer cluster health, per-broker master/slave roles, consumer group lag across dozens of topic queues, CommitLog write performance, and replication offset lag all require monitoring to keep the platform reliable. The Vigilmon setup in this guide covers all of RocketMQ's critical failure modes with lightweight cron-based heartbeat checks that don't require deploying additional monitoring infrastructure. With these monitors in place, your platform team gets alerted within 2–4 minutes of any RocketMQ component failure — before message processing pipelines stall and application error rates climb.

Get started at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles