Monitoring AKHQ (Kafka HQ) with Vigilmon

AKHQ gives your team a rich UI for Kafka topic management, consumer group lag tracking, Schema Registry browsing, and multi-cluster oversight — but when AKHQ itself goes down or silently loses its broker connection, operators are flying blind on consumer lag and partition health. Vigilmon fills that gap with health endpoint monitoring, Kafka connectivity checks, and JVM memory alerting so you know about AKHQ issues before your on-call engineer starts manually querying kafka-consumer-groups.sh.

What You'll Set Up

• AKHQ application uptime monitor via /api/health
• Kafka broker connectivity check
• Consumer group lag alerting via cron heartbeat
• Schema Registry connectivity monitor
• Kafka Connect status check
• JVM heap memory alerting

Prerequisites

• AKHQ running on port 8080 (Docker, JAR, or Kubernetes)
• Kafka cluster with at least one accessible broker
• A free Vigilmon account

Step 1: Monitor the AKHQ Application Health Endpoint

AKHQ exposes a Micronaut health endpoint at /api/health that returns the application status including Kafka connectivity checks. This is your primary health signal — if this endpoint fails, the entire AKHQ UI is down.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter: http://your-akhq-host:8080/api/health
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

A healthy AKHQ instance returns:

{"name":"akhq","status":"UP"}

When Kafka brokers are unreachable, AKHQ's health endpoint may return 503 with a DOWN status — use this as an early warning of broker connectivity problems even before users notice the UI is degraded.

If AKHQ is deployed behind a reverse proxy, monitor the proxied URL:

https://akhq.yourdomain.com/api/health

Step 2: Monitor Kafka Broker Connectivity

AKHQ connects to Kafka brokers via the AdminClient API. If AKHQ loses connectivity to all brokers, the web UI loads but all operations (topic list, consumer group fetch, message browser) return errors. Monitor broker reachability directly:

1. Add a new monitor → TCP Port.
2. Set Host to your first Kafka broker (e.g., kafka-broker-1.internal).
3. Set Port to 9092 (plaintext) or 9093 (TLS).
4. Set Check interval to 1 minute.
5. Click Save.

Repeat for each broker in your cluster. A single broker being down is usually not critical (Kafka replication handles it), but if all broker TCP monitors fire simultaneously, AKHQ has lost its entire cluster connection.

For Kafka clusters using SASL authentication, also monitor the port exposed for your auth protocol:

# SASL/PLAIN or SASL/SCRAM typically on 9092
# SASL/OAUTHBEARER may use a dedicated port — check your broker config

Step 3: Consumer Group Lag Alerting via Cron Heartbeat

Consumer group lag is one of the most critical Kafka metrics — it tells you whether your consumers are keeping up with message production. AKHQ displays lag in the UI, but if lag exceeds a threshold and no one is watching the dashboard, you need an active alert.

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval to 5 minutes.
3. Copy the heartbeat URL (e.g. https://vigilmon.online/heartbeat/abc123).

Create a script that checks consumer lag via kafka-consumer-groups.sh and only pings Vigilmon when lag is within acceptable limits:

#!/bin/bash
# kafka-lag-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
KAFKA_BOOTSTRAP="kafka-broker-1.internal:9092"
MAX_LAG=10000  # alert if any consumer group exceeds this lag

# Get all consumer groups
GROUPS=$(kafka-consumer-groups.sh --bootstrap-server "$KAFKA_BOOTSTRAP" \
  --list 2>/dev/null)

HIGH_LAG=0
for GROUP in $GROUPS; do
  LAG=$(kafka-consumer-groups.sh --bootstrap-server "$KAFKA_BOOTSTRAP" \
    --describe --group "$GROUP" 2>/dev/null | \
    awk 'NR>1 && $5 ~ /^[0-9]+$/ {sum += $5} END {print sum+0}')

  if [ "${LAG:-0}" -gt "$MAX_LAG" ]; then
    HIGH_LAG=1
    echo "High lag in group $GROUP: $LAG"
  fi
done

if [ "$HIGH_LAG" -eq 0 ]; then
  curl -s "$HEARTBEAT_URL"
fi

Alternatively, use the AKHQ REST API to query lag:

# AKHQ exposes consumer group API
curl -s "http://your-akhq-host:8080/api/{cluster}/group?page=1&pageSize=100" \
  -H "Authorization: Basic $(echo -n 'admin:password' | base64)" | \
  jq '.[] | select(.offsets[].lag > 10000) | .id'

Step 4: Monitor Schema Registry Connectivity

If you're using Confluent Schema Registry (or a compatible implementation like Apicurio), AKHQ connects to it to display schema lists and validate message serialization. Loss of Schema Registry connectivity degrades the AKHQ experience for teams managing Avro or Protobuf schemas.

1. Add a new monitor → HTTP / HTTPS.
2. Enter: http://your-schema-registry:8081/subjects
3. Set Expected HTTP status to 200.
4. Set Check interval to 2 minutes.
5. Click Save.

The /subjects endpoint lists all registered schema subjects. A 200 response confirms the Schema Registry is alive and accessible. For secured Schema Registries, use the root health endpoint instead:

http://your-schema-registry:8081/

If AKHQ's Schema Registry section starts returning errors, this monitor will confirm whether the issue is Schema Registry downtime or an AKHQ configuration problem.

Step 5: Monitor Kafka Connect Status

AKHQ displays the status of Kafka Connect connectors, making it easy to spot failed connectors without the Kafka Connect REST API. Monitor Kafka Connect's health endpoint directly:

1. Add a monitor → HTTP / HTTPS.
2. Enter: http://your-kafka-connect:8083/
3. Set Expected HTTP status to 200.
4. Set Check interval to 2 minutes.
5. Click Save.

The Kafka Connect root endpoint returns a JSON object with version information when healthy. For connector-level health alerting, use a cron heartbeat:

#!/bin/bash
# kafka-connect-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
CONNECT_URL="http://your-kafka-connect:8083"

# Get all connectors and check for FAILED state
FAILED=$(curl -s "$CONNECT_URL/connectors?expand=status" | \
  jq '[to_entries[] | select(.value.status.connector.state == "FAILED")] | length')

if [ "${FAILED:-0}" -eq 0 ]; then
  curl -s "$HEARTBEAT_URL"
fi

Run every 5 minutes. If any connector enters a FAILED state, the heartbeat stops.

Step 6: Monitor AKHQ Authentication Provider

If AKHQ is configured with RBAC and an external auth provider (LDAP or OIDC/SSO), an auth provider outage prevents all users from logging in — even read-only monitoring dashboards become inaccessible.

For LDAP:

1. Add a monitor → TCP Port.
2. Set Host to your LDAP server.
3. Set Port to 389 (LDAP) or 636 (LDAPS).
4. Set Check interval to 2 minutes.
5. Click Save.

For OIDC:

1. Add a monitor → HTTP / HTTPS.
2. Enter the OIDC discovery endpoint: https://your-oidc-provider/.well-known/openid-configuration
3. Set Expected HTTP status to 200.
4. Set Check interval to 2 minutes.
5. Click Save.

If the OIDC discovery endpoint goes down, AKHQ's login flow fails and no users can authenticate.

Step 7: JVM Heap Memory Alerting

AKHQ is a Micronaut Java application. JVM heap exhaustion causes progressively slower responses before the process is killed by the OOM killer. Monitor heap usage via AKHQ's Micronaut metrics endpoint (if enabled):

Enable metrics in application.yml:

micronaut:
  metrics:
    enabled: true
    export:
      prometheus:
        enabled: true
        step: PT1M

This exposes metrics at /prometheus. Then use a cron heartbeat to check heap usage:

#!/bin/bash
# akhq-jvm-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
AKHQ_URL="http://your-akhq-host:8080"
MAX_HEAP_PCT=80

# Fetch heap metrics
HEAP_USED=$(curl -s "$AKHQ_URL/prometheus" | \
  grep 'jvm_memory_used_bytes{.*heap' | \
  awk '{sum += $2} END {print sum+0}')

HEAP_MAX=$(curl -s "$AKHQ_URL/prometheus" | \
  grep 'jvm_memory_max_bytes{.*heap' | \
  awk '{sum += $2} END {print sum+0}')

if [ "${HEAP_MAX:-0}" -gt 0 ]; then
  HEAP_PCT=$(( HEAP_USED * 100 / HEAP_MAX ))
  if [ "$HEAP_PCT" -lt "$MAX_HEAP_PCT" ]; then
    curl -s "$HEARTBEAT_URL"
  fi
fi

Set the Vigilmon heartbeat to 5 minutes. If heap exceeds 80%, the heartbeat stops and you get an alert before AKHQ crashes.

Step 8: Configure Alert Channels and Multi-Cluster Visibility

1. Go to Alert Channels in Vigilmon and configure Slack, email, or PagerDuty.
2. Set Consecutive failures before alert:
   • AKHQ health endpoint: 2 — brief restarts on config reload are normal
   • Kafka broker TCP: 1 per broker, alert when all brokers fail
   • Consumer lag: 1 — lag exceeding thresholds needs immediate attention
   • Schema Registry: 3 — brief timeouts are common under load
3. For multi-cluster AKHQ setups, add separate Vigilmon monitors per Kafka cluster's broker list and label them clearly (e.g., "Kafka Prod EU", "Kafka Prod US").

Summary

| Monitor | Target | What It Catches | |---|---|---| | AKHQ health | http://akhq:8080/api/health | Application crash, startup failure | | Kafka brokers | TCP :9092 per broker | Cluster connectivity loss | | Consumer lag | Cron heartbeat every 5 min | Processing backlog, consumer failures | | Schema Registry | http://registry:8081/subjects | Schema validation loss | | Kafka Connect | http://connect:8083/ + cron | Connector failure, pipeline breaks | | LDAP / OIDC | TCP or HTTP health | Login unavailable for all users | | JVM heap | Cron heartbeat every 5 min | OOM crash before it happens |

AKHQ gives your team a self-hosted alternative to managed Kafka consoles — but that self-hosted control means you own the availability too. With Vigilmon watching AKHQ's health endpoint, Kafka connectivity, and consumer lag, you'll catch Kafka cluster problems and AKHQ outages before your engineering team's morning standup becomes a fire drill.