tutorial

Monitoring Highlight.io with Vigilmon

Highlight.io is an open source session replay and error tracking platform — but who monitors the monitor? Here's how to watch every layer of a self-hosted Highlight deployment with Vigilmon so your observability platform stays observable.

Highlight.io is an open source full-stack monitoring platform: session replay, error tracking, logging, and performance monitoring — all self-hosted. It's the tool your team uses to debug production issues in real time. But a self-hosted Highlight deployment is itself a complex distributed system: a Go backend, ClickHouse for event storage, Kafka for event streaming, PostgreSQL for structured data, Redis for queuing, and object storage for session recordings. When Highlight degrades or goes down, your team loses visibility into production issues at exactly the moment you need it most. Vigilmon gives you a lightweight external monitoring layer over your Highlight deployment so your observability platform stays observable.

What You'll Set Up

  • HTTP uptime monitor for the Highlight backend health endpoint
  • Session ingestion throughput heartbeat
  • Kafka consumer lag monitoring via a scheduled check
  • ClickHouse health check via the Highlight API
  • SSL certificate expiry alert for the Highlight domain
  • Object storage health check

Prerequisites

  • Highlight.io self-hosted and accessible over HTTPS (using the official Docker Compose or Helm chart)
  • ClickHouse, Kafka, PostgreSQL, and Redis running as part of the Highlight stack
  • A free Vigilmon account

Step 1: Monitor the Highlight Backend Health Endpoint

The Highlight Go backend exposes a health endpoint used by its own orchestration layer. Start with an HTTP uptime monitor:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter: https://highlight.yourdomain.com/health (or https://highlight.yourdomain.com for the root if no /health is exposed)
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Enable Monitor SSL certificate and set Alert when certificate expires in less than 21 days.
  7. Click Save.

For the self-hosted Docker Compose deployment, the Highlight frontend is served at https://highlight.yourdomain.com and the backend API at https://highlight.yourdomain.com/public. Use the public API root as a liveness proxy:

https://highlight.yourdomain.com/public

A 200 on the public endpoint confirms the Go backend is up and its nginx/traefik proxy is routing correctly.

Step 2: Monitor the Session Ingestion Endpoint

The session ingestion endpoint at /public receives rrweb recordings from browser SDKs. This is the highest-value path in the Highlight backend — if ingestion is down, all session recordings are silently lost from all monitored applications. Monitor it explicitly:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter: https://highlight.yourdomain.com/public
  3. Set Method to POST.
  4. Set Request body: 
    {"query": "query { ping }"}
    (Highlight's public API is a GraphQL endpoint; a ping query confirms the ingestion layer is alive)
  5. Set Request header: Content-Type: application/json
  6. Set Expected HTTP status to 200.
  7. Set Check interval to 2 minutes.
  8. Click Save.

The GraphQL ping query returns a lightweight response without touching ClickHouse or Kafka, making it suitable for a high-frequency liveness check. A failure here means the ingestion endpoint itself is down — no new session data is arriving.

Step 3: Heartbeat Monitor for Session Ingestion Throughput

The ingestion endpoint being up does not guarantee data is actually flowing through Kafka into ClickHouse. A Kafka consumer group falling behind or a ClickHouse write bottleneck can cause silent data loss even when the HTTP endpoint returns 200. Use a cron heartbeat to detect processing pipeline failures:

  1. Click Add Monitor → Cron Heartbeat.
  2. Set the expected ping interval to 10 minutes.
  3. Copy the heartbeat URL: https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_ID.

Create a script that checks ClickHouse session counts and pings Vigilmon if data is flowing:

#!/bin/bash
# /etc/cron.d/highlight-ingestion-check
# Runs every 10 minutes

HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_ID"
CLICKHOUSE_HOST="localhost"
CLICKHOUSE_PORT="9000"
LOOKBACK_MINUTES=15

# Check that sessions have been written to ClickHouse in the last 15 minutes
SESSION_COUNT=$(clickhouse-client \
  --host="$CLICKHOUSE_HOST" \
  --port="$CLICKHOUSE_PORT" \
  --database="highlight" \
  --query="SELECT count() FROM sessions WHERE created_at > now() - INTERVAL $LOOKBACK_MINUTES MINUTE" \
  2>/dev/null)

if [ -z "$SESSION_COUNT" ] || [ "$SESSION_COUNT" -eq 0 ]; then
  echo "WARNING: No sessions written to ClickHouse in the last $LOOKBACK_MINUTES minutes" >&2
  # Do not ping — missing heartbeat triggers the alert
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Adjust the LOOKBACK_MINUTES threshold based on your traffic volume. For low-traffic deployments monitoring only internal tools, use 60 minutes. For production applications with active user sessions, 15 minutes is appropriate.

Step 4: Monitor Kafka Consumer Lag

Kafka consumer lag is the most important leading indicator of Highlight pipeline health. If the Highlight consumers fall behind, session events pile up in Kafka and the dashboard shows stale data — users see their sessions with a growing delay. Set up a daily Kafka lag check:

  1. Click Add Monitor → Cron Heartbeat.
  2. Set the expected ping interval to 5 minutes.
  3. Copy a separate heartbeat URL for the Kafka check.

Create a Kafka lag monitoring script:

#!/bin/bash
# /etc/cron.d/highlight-kafka-lag-check

HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_KAFKA_HEARTBEAT_ID"
KAFKA_BOOTSTRAP="localhost:9092"
MAX_LAG=10000

# Check consumer group lag for Highlight consumers
LAG=$(kafka-consumer-groups.sh \
  --bootstrap-server "$KAFKA_BOOTSTRAP" \
  --group highlight-consumer \
  --describe 2>/dev/null \
  | awk 'NR>1 && $5 != "-" {sum += $5} END {print sum}')

if [ -z "$LAG" ] || [ "$LAG" -gt "$MAX_LAG" ]; then
  echo "WARNING: Kafka consumer lag is $LAG (threshold: $MAX_LAG)" >&2
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

The MAX_LAG=10000 threshold aligns with Highlight's own recommended alerting threshold for consumer group lag. A lag exceeding 10,000 messages indicates the Go backend consumers are unable to keep up with the ingestion rate — commonly caused by ClickHouse write latency or a Go consumer process crash.

Step 5: Monitor the Highlight Dashboard Availability

Your team accesses Highlight through its web dashboard to review session replays and error reports. A broken dashboard doesn't prevent data ingestion, but it prevents your team from acting on it. Add a dashboard availability monitor:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter: https://highlight.yourdomain.com
  3. Set Expected HTTP status to 200.
  4. Set Check interval to 5 minutes.
  5. Under Response time alerts, enable Alert when p95 response time exceeds 3000 ms (3 seconds).
  6. Click Save.

The Highlight frontend is served as a compiled React SPA. A slow dashboard typically indicates:

  • PostgreSQL query latency on dashboard data (projects, alerts, settings)
  • Redis connectivity failure causing session lookup to fall back to PostgreSQL for every request
  • ClickHouse query latency on session/error listing endpoints

Step 6: Configure Alert Channels and Escalation

  1. Go to Alert Channels in Vigilmon and add Slack and PagerDuty. For a production observability platform, PagerDuty is appropriate — losing your monitoring tool during an incident is exactly the scenario where you need an immediate page.
  2. Set Consecutive failures before alert to 1 on the session ingestion endpoint — a single ingestion failure represents actual data loss.
  3. Set Consecutive failures before alert to 2 on the dashboard monitor — a slow cold start or temporary overload may self-resolve.
  4. Set Consecutive failures before alert to 1 on both heartbeat monitors (Kafka lag and ClickHouse throughput).

Add a maintenance window during Highlight upgrades (which require ClickHouse migrations and Kafka consumer restarts):

# Before upgrading Highlight
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"monitor_id": "YOUR_INGESTION_MONITOR_ID", "duration_minutes": 20}'

# Upgrade Highlight
docker compose pull && docker compose up -d

# Maintenance window expires; Vigilmon resumes monitoring

Summary

| Monitor | Target | Alert Threshold | What It Catches | |---|---|---|---| | Backend health | /public (GraphQL ping) | Non-200 | Highlight Go backend down | | Dashboard | https://highlight.domain | Non-200 or p95 > 3 s | Frontend unavailable | | Ingestion throughput | ClickHouse heartbeat | Missed 10-min ping | Pipeline data loss | | Kafka consumer lag | kafka-consumer-groups check | Lag > 10,000 | Processing bottleneck | | SSL certificate | Highlight domain | Expiry < 21 days | TLS renewal failure |

Highlight.io is your window into what's happening in production — but that window needs its own maintenance. With Vigilmon watching the ingestion pipeline, Kafka consumer health, and dashboard availability, you'll know when Highlight itself needs attention before an incident leaves your team flying blind.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles