tutorial

How to Monitor Apache IoTDB with Vigilmon

IoTDB DataNode crashes, ConfigNode leader election failures, and WAL disk exhaustion silently drop time-series writes from thousands of IoT devices. Learn how to monitor Apache IoTDB health, write throughput, compaction status, and cluster synchronization with Vigilmon HTTP probes and heartbeat monitors.

Apache IoTDB is purpose-built for high-rate IoT time-series ingestion — factories writing sensor data at millions of points per second, smart grids logging power consumption across thousands of devices, transportation systems tracking vehicle telemetry in real time. When a DataNode goes down, every IoT device that writes to that node starts queuing data locally or dropping measurements. When the WAL disk fills, all writes to the cluster block. When compaction falls behind, TsFile count grows unboundedly until query performance degrades to unusable.

Vigilmon gives you external uptime monitoring for IoTDB DataNode and ConfigNode health, write path availability, compaction health, and cluster synchronization — through HTTP health endpoints and heartbeat monitors. This tutorial walks you through the full setup.

Why Apache IoTDB Needs External Monitoring

IoTDB provides JMX metrics (via Prometheus exporter), the show cluster SQL command, and internal log monitoring. But these have a fundamental gap: they tell you what the running cluster reports about itself. External monitoring with Vigilmon adds:

  • DataNode TCP reachability — is port 6667 (the client RPC port) actually accepting connections? A running JVM process can stop accepting connections while appearing healthy in logs
  • Write path end-to-end — does a write to the cluster succeed? A DataNode can be up while the WAL disk is full and all writes are rejected
  • Cluster member count drift — if a DataNode leaves the cluster, replication factor degrades silently; Vigilmon catches this before data loss occurs
  • Heartbeat-based compaction verification — compaction runs asynchronously; heartbeat monitors verify it's progressing on schedule
  • ConfigNode leader loss — if the ConfigNode cluster loses its leader, all schema management operations stall; external monitoring catches this before operators try to create a new timeseries and fail

What you'll need

  • Apache IoTDB 1.x or later (clustered or standalone)
  • Java runtime (IoTDB runs on JVM)
  • A lightweight HTTP health sidecar with access to IoTDB's RPC port 6667
  • A free Vigilmon account

Step 1: Build an IoTDB health endpoint

IoTDB exposes metrics via JMX and optionally Prometheus (enable with enable_metric_service=true in iotdb-system.properties). The health sidecar below uses the Prometheus scrape endpoint for metrics and the IoTDB JDBC driver for write-path checks.

Enable Prometheus metrics in IoTDB

Add to conf/iotdb-system.properties:

# Enable metrics collection
enable_metric_service=true
metric_reporter_list=PROMETHEUS
metric_prometheus_reporter_port=9091

Restart IoTDB for the change to take effect. Metrics will be available at:

http://<datanode-host>:9091/metrics

Python health sidecar

# iotdb_health.py
import os
import requests
from flask import Flask, jsonify

app = Flask(__name__)

IOTDB_METRICS_URL = os.getenv('IOTDB_METRICS_URL', 'http://localhost:9091/metrics')
IOTDB_HOST = os.getenv('IOTDB_HOST', 'localhost')
IOTDB_PORT = int(os.getenv('IOTDB_PORT', '6667'))


def scrape_metrics():
    try:
        resp = requests.get(IOTDB_METRICS_URL, timeout=5)
        resp.raise_for_status()
        metrics = {}
        for line in resp.text.splitlines():
            if line.startswith('#') or not line.strip():
                continue
            parts = line.rsplit(' ', 1)
            if len(parts) == 2:
                metrics[parts[0].strip()] = parts[1].strip()
        return metrics, None
    except Exception as e:
        return {}, str(e)


def parse_metric(metrics, name, default=0.0):
    for key, val in metrics.items():
        if key == name or key.startswith(name + '{') or key.startswith(name + ' '):
            try:
                return float(val)
            except ValueError:
                continue
    return default


@app.route('/health/iotdb/datanode')
def datanode_health():
    import socket
    # First check TCP port reachability
    try:
        sock = socket.create_connection((IOTDB_HOST, IOTDB_PORT), timeout=5)
        sock.close()
    except Exception as e:
        return jsonify(status='down', error=f'port {IOTDB_PORT} unreachable: {str(e)}'), 503

    metrics, err = scrape_metrics()
    if err:
        return jsonify(status='degraded', warning='metrics_unavailable', error=err, port='open')

    # Check JVM health via uptime metric
    uptime = parse_metric(metrics, 'jvm_uptime_seconds_total')

    return jsonify(
        status='ok',
        rpc_port=IOTDB_PORT,
        jvm_uptime_seconds=uptime,
    )


@app.route('/health/iotdb/write')
def write_health():
    metrics, err = scrape_metrics()
    if err:
        return jsonify(status='degraded', error=err), 503

    # Write throughput (points per second) and latency
    write_points = parse_metric(metrics, 'iotdb_session_execute_insert_records_total')
    write_latency_p95 = parse_metric(metrics, 'iotdb_stage_operator_latency_seconds{quantile="0.95",stage="write"}')

    issues = []
    if write_latency_p95 > 0.5:
        issues.append(f'write P95 latency {write_latency_p95:.3f}s exceeds 500ms threshold')

    if issues:
        return jsonify(
            status='degraded',
            issues=issues,
            write_points_total=write_points,
            write_latency_p95_seconds=write_latency_p95,
        ), 503

    return jsonify(
        status='ok',
        write_points_total=write_points,
        write_latency_p95_seconds=write_latency_p95,
    )


@app.route('/health/iotdb/compaction')
def compaction_health():
    metrics, err = scrape_metrics()
    if err:
        return jsonify(status='degraded', error=err), 503

    # Compaction queue depth — high values indicate compaction falling behind
    compaction_queue = parse_metric(metrics, 'iotdb_compaction_task_count{status="waiting"}')
    compaction_errors = parse_metric(metrics, 'iotdb_compaction_task_count{status="error"}')

    if compaction_errors > 0:
        return jsonify(
            status='critical',
            reason='compaction_errors',
            compaction_error_tasks=int(compaction_errors),
            compaction_queue_depth=int(compaction_queue),
        ), 503

    if compaction_queue > 100:
        return jsonify(
            status='degraded',
            reason='compaction_queue_high',
            compaction_queue_depth=int(compaction_queue),
        ), 503

    return jsonify(
        status='ok',
        compaction_queue_depth=int(compaction_queue),
        compaction_errors=int(compaction_errors),
    )


@app.route('/health/iotdb/storage')
def storage_health():
    import shutil
    data_dir = os.getenv('IOTDB_DATA_DIR', '/opt/iotdb/data')

    if not os.path.exists(data_dir):
        return jsonify(status='unknown', error=f'data dir {data_dir} not found')

    usage = shutil.disk_usage(data_dir)
    used_pct = round(usage.used / usage.total * 100, 1)

    if used_pct > 85:
        return jsonify(
            status='critical',
            reason='disk_over_85pct',
            used_pct=used_pct,
            free_gb=round(usage.free / 1024**3, 1),
            total_gb=round(usage.total / 1024**3, 1),
        ), 503

    return jsonify(
        status='ok',
        used_pct=used_pct,
        free_gb=round(usage.free / 1024**3, 1),
        total_gb=round(usage.total / 1024**3, 1),
    )


@app.route('/health/iotdb/cluster')
def cluster_health():
    metrics, err = scrape_metrics()
    if err:
        return jsonify(status='degraded', error=err), 503

    # DataNode member count
    datanode_count = parse_metric(metrics, 'iotdb_cluster_node_count{type="datanode",status="Running"}')
    confignode_count = parse_metric(metrics, 'iotdb_cluster_node_count{type="confignode",status="Running"}')

    expected_datanodes = int(os.getenv('EXPECTED_DATANODE_COUNT', '3'))
    expected_confignodes = int(os.getenv('EXPECTED_CONFIGNODE_COUNT', '3'))

    issues = []
    if datanode_count < expected_datanodes:
        issues.append(f'only {int(datanode_count)}/{expected_datanodes} DataNodes running')
    if confignode_count < expected_confignodes:
        issues.append(f'only {int(confignode_count)}/{expected_confignodes} ConfigNodes running')

    if issues:
        return jsonify(
            status='critical',
            reason='cluster_members_missing',
            issues=issues,
            datanode_count=int(datanode_count),
            confignode_count=int(confignode_count),
        ), 503

    return jsonify(
        status='ok',
        datanode_count=int(datanode_count),
        confignode_count=int(confignode_count),
    )


@app.route('/health/iotdb/wal')
def wal_health():
    import shutil
    wal_dir = os.getenv('IOTDB_WAL_DIR', '/opt/iotdb/data/wal')

    if not os.path.exists(wal_dir):
        return jsonify(status='ok', note='WAL dir not found — may be on same filesystem as data')

    usage = shutil.disk_usage(wal_dir)
    used_pct = round(usage.used / usage.total * 100, 1)

    if used_pct > 90:
        return jsonify(
            status='critical',
            reason='wal_disk_over_90pct',
            used_pct=used_pct,
            free_gb=round(usage.free / 1024**3, 2),
        ), 503

    return jsonify(
        status='ok',
        wal_disk_used_pct=used_pct,
        free_gb=round(usage.free / 1024**3, 2),
    )


if __name__ == '__main__':
    app.run(host='0.0.0.0', port=9083)

Install and start:

pip install flask requests
export IOTDB_HOST=localhost
export IOTDB_DATA_DIR=/opt/iotdb/data
export IOTDB_WAL_DIR=/opt/iotdb/data/wal
export EXPECTED_DATANODE_COUNT=3
python iotdb_health.py &

As a systemd unit:

# /etc/systemd/system/iotdb-health.service
[Unit]
Description=IoTDB Health Endpoint
After=network.target

[Service]
ExecStart=/usr/bin/python3 /opt/iotdb-health/iotdb_health.py
Restart=always
User=iotdb
Environment=IOTDB_HOST=localhost
Environment=IOTDB_DATA_DIR=/opt/iotdb/data
Environment=IOTDB_WAL_DIR=/opt/iotdb/data/wal
Environment=EXPECTED_DATANODE_COUNT=3
Environment=EXPECTED_CONFIGNODE_COUNT=3

[Install]
WantedBy=multi-user.target

systemctl enable --now iotdb-health

Verify:

curl http://localhost:9083/health/iotdb/datanode
# {"status": "ok", "rpc_port": 6667, "jvm_uptime_seconds": 84321.0}

curl http://localhost:9083/health/iotdb/cluster
# {"status": "ok", "datanode_count": 3, "confignode_count": 3}

Step 2: Monitor DataNode TCP reachability directly

Add a TCP monitor for IoTDB's RPC port — this is your fastest signal if a DataNode crashes:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose TCP Port
  3. Host: iotdb-datanode-01.example.com
  4. Port: 6667
  5. Check interval: 1 minute
  6. Save as "IoTDB DataNode 01 RPC Port"

Repeat for each DataNode in your cluster. Also add TCP monitors for the ConfigNode management port (default 10710):

  1. Monitors → New Monitor → TCP Port
  2. Host: iotdb-confignode-01.example.com
  3. Port: 10710
  4. Save as "IoTDB ConfigNode 01 Management Port"

Step 3: Add HTTP monitors via the health sidecar

DataNode application health

  1. Monitors → New Monitor → HTTP / HTTPS
  2. URL: http://iotdb-datanode-01.example.com:9083/health/iotdb/datanode
  3. Expected status: 200
  4. Check interval: 1 minute
  5. Save as "IoTDB DataNode Health"

Write path health

  1. Monitors → New Monitor → HTTP / HTTPS
  2. URL: http://iotdb-datanode-01.example.com:9083/health/iotdb/write
  3. Expected status: 200
  4. Check interval: 2 minutes
  5. Save as "IoTDB Write Path Health"

Compaction health

  1. Monitors → New Monitor → HTTP / HTTPS
  2. URL: http://iotdb-datanode-01.example.com:9083/health/iotdb/compaction
  3. Expected status: 200
  4. Check interval: 5 minutes
  5. Save as "IoTDB Compaction Health"

Storage (disk) health

  1. Monitors → New Monitor → HTTP / HTTPS
  2. URL: http://iotdb-datanode-01.example.com:9083/health/iotdb/storage
  3. Expected status: 200
  4. Check interval: 5 minutes
  5. Save as "IoTDB Storage Disk Usage"

WAL health

  1. Monitors → New Monitor → HTTP / HTTPS
  2. URL: http://iotdb-datanode-01.example.com:9083/health/iotdb/wal
  3. Expected status: 200
  4. Check interval: 5 minutes
  5. Save as "IoTDB WAL Disk Health"

Cluster member health

  1. Monitors → New Monitor → HTTP / HTTPS
  2. URL: http://iotdb-datanode-01.example.com:9083/health/iotdb/cluster
  3. Expected status: 200
  4. Check interval: 2 minutes
  5. Save as "IoTDB Cluster Member Count"

Step 4: Add a heartbeat monitor for TTL enforcement and scheduled tasks

IoTDB's TTL (time-to-live) enforcement runs asynchronously. Use a heartbeat monitor to verify it's completing:

#!/bin/bash
# iotdb-ttl-check.sh — run every hour via cron

HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_TOKEN"
IOTDB_CLI="/opt/iotdb/sbin/start-cli.sh"

# Run a lightweight query to verify cluster is accepting queries
result=$($IOTDB_CLI -h 127.0.0.1 -p 6667 -u root -pw root -e "SHOW DATABASES" 2>&1)

if echo "$result" | grep -q "Database"; then
    curl -s -X POST "$HEARTBEAT_URL" > /dev/null
    echo "IoTDB responsive, heartbeat sent"
else
    echo "IoTDB query failed: $result"
    echo "Heartbeat NOT sent — Vigilmon will alert"
fi

crontab -e
# Add:
0 * * * * /opt/iotdb-health/iotdb-ttl-check.sh

In Vigilmon:

  1. Monitors → New Monitor → Heartbeat
  2. Interval: 90 minutes (alert if hourly check misses by more than 30 minutes)
  3. Copy the heartbeat URL into the script above

Step 5: Configure alert channels

Email alerts

  1. Alert Channels → Add Channel → Email
  2. Enter your IoT platform or data engineering team's on-call email
  3. Assign to all IoTDB monitors

Webhook alerts

Vigilmon sends:

{
  "monitor_name": "IoTDB DataNode 01 RPC Port",
  "status": "down",
  "host": "iotdb-datanode-01.example.com",
  "port": 6667,
  "started_at": "2026-06-20T02:10:00Z",
  "duration_seconds": 75
}

When the DataNode RPC port goes down, every IoT device writing to that node starts failing. Route DataNode alerts to PagerDuty immediately — recovery time directly impacts data loss window.

Recommended alert priority

| Monitor | Severity | Immediate action | |---------|----------|-----------------| | DataNode RPC Port (6667) | Critical | Check JVM process; restart DataNode if crashed | | ConfigNode Port (10710) | Critical | Check ConfigNode process; leader election may be in progress | | DataNode Application Health | High | Check JVM heap; check gc logs | | Write Path Health | High | Check WAL disk; check connection pool saturation | | WAL Disk | Critical | Free disk immediately — WAL full blocks all writes | | Storage Disk | High | Expand volume or add data directory | | Cluster Member Count | Critical | Check failed DataNode; check network between nodes | | Compaction Health | Medium | Check compaction logs; restart if stuck | | TTL Heartbeat | Medium | Check cron job; verify IoTDB CLI access |

Full monitor summary

| Monitor | Type | Endpoint | What it catches | |---------|------|----------|-----------------| | DataNode 01 RPC | TCP | datanode-01:6667 | DataNode crash / port down | | ConfigNode 01 | TCP | confignode-01:10710 | ConfigNode unavailable | | DataNode Health | HTTP | :9083/health/iotdb/datanode | JVM health, port open | | Write Path | HTTP | :9083/health/iotdb/write | Write latency >500ms | | Compaction | HTTP | :9083/health/iotdb/compaction | Compaction errors/backlog | | Storage Disk | HTTP | :9083/health/iotdb/storage | Disk >85% | | WAL Disk | HTTP | :9083/health/iotdb/wal | WAL disk >90% | | Cluster Members | HTTP | :9083/health/iotdb/cluster | DataNode/ConfigNode missing | | TTL Heartbeat | Heartbeat | heartbeat URL | Hourly cluster query failing |

What's next

  • Grafana integration — IoTDB has an official Grafana plugin for TsFile queries; pair it with Vigilmon alerts to combine internal dashboards with external uptime alerting
  • Multi-DataNode coverage — deploy a health sidecar on each DataNode and add separate Vigilmon monitors per node, so you know exactly which node failed rather than just "the cluster is degraded"
  • Edge IoTDB monitoring — IoTDB supports edge deployment on resource-constrained devices; use Vigilmon's heartbeat monitor to verify that edge IoTDB instances are writing upstream to the main cluster on schedule

Get started free at vigilmon.online — no credit card required, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles