tutorial

Monitoring Apache Paimon with Vigilmon

Apache Paimon brings streaming upserts to your data lake — but LSM compaction lag and object storage failures are invisible without external monitoring. Here's how to monitor Paimon's health end-to-end with Vigilmon.

Apache Paimon (formerly Flink Table Store) solves one of the hardest problems in the modern data stack: making object storage (S3, HDFS, OSS) work efficiently for both high-frequency streaming writes and fast batch analytics. By applying an LSM-Tree (Log-Structured Merge Tree) format on top of object storage, Paimon supports real-time CDC ingestion, efficient upserts, schema evolution, and time travel — all on the same tables your Spark or Trino batch queries read. But with that power comes operational complexity: compaction lag, snapshot accumulation, manifest file health, and object storage connectivity are all failure modes that don't surface in application logs until they cause query failures or write stalls. Vigilmon gives you the external health monitoring that Paimon's storage layer doesn't provide out of the box.

What You'll Set Up

  • Cron heartbeat monitors for Flink streaming write jobs writing to Paimon tables
  • Compaction lag monitoring via heartbeats
  • Object storage reachability monitoring
  • Catalog service health monitoring
  • HTTP monitors for Flink cluster connectivity (required by Paimon Flink integration)

Prerequisites

  • Apache Paimon 0.8+ integrated with Apache Flink 1.17+ (or Spark/Trino for batch reads)
  • Paimon tables stored on S3, HDFS, or OSS
  • Flink cluster with Paimon Flink connector deployed
  • A free Vigilmon account

Why Monitoring Paimon Matters

Paimon's LSM-Tree storage model introduces operational concerns that traditional columnar formats (Parquet/ORC) do not have:

  • Compaction lag — Paimon writes new data as L0 files and merges them through compaction. If compaction falls behind, read queries must merge an unbounded number of small files at query time — causing severe read performance degradation
  • Snapshot accumulation — Paimon retains historical snapshots for time travel. If snapshot expiration fails (e.g., due to object storage permission errors), snapshots accumulate indefinitely and storage costs grow unboundedly
  • Manifest file corruption — Paimon uses manifest files to track data files per snapshot. A manifest write failure can corrupt the snapshot, making the entire table unreadable
  • Object storage connectivity — All Paimon data is stored remotely (S3/HDFS/OSS). A network partition or IAM permission change causes all Flink streaming write jobs to fail
  • Changelog freshness — If the Flink streaming job writing CDC changes to Paimon stalls, downstream Paimon readers see stale data — a problem that is invisible unless you monitor lag explicitly

Step 1: Monitor Flink Write Job Health with Heartbeats

The primary interface to Paimon in a streaming architecture is a Flink job that reads from a CDC source (MySQL/PostgreSQL binlog via Flink CDC) and writes to Paimon tables. Monitor these Flink jobs first — they are the entry point for all data into Paimon.

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Name the monitor: Flink CDC → Paimon write job.
  3. Set the expected interval to 5 minutes.
  4. Copy the heartbeat URL.

Add a cron job that checks the Flink job status and sends a heartbeat only when the job is RUNNING:

#!/bin/bash
# /etc/cron.d/paimon-flink-heartbeat
# Runs every 3 minutes
*/3 * * * * monitoring curl -sf "http://flink-jobmanager:8081/jobs/overview" | \
  python3 -c "
import sys, json
jobs = json.load(sys.stdin).get('jobs', [])
cdc_jobs = [j for j in jobs if 'paimon' in j.get('name','').lower() and j['status'] == 'RUNNING']
exit(0 if cdc_jobs else 1)
" && curl -fsS "https://vigilmon.online/heartbeat/YOUR_FLINK_KEY" > /dev/null 2>&1

Set the Vigilmon heartbeat interval to 10 minutes. If the Flink job stops running (FAILED, CANCELED, or FINISHED unexpectedly), the heartbeat expires and you get alerted.

Step 2: Monitor Flink JobManager API

Paimon's Flink integration requires the Flink cluster to be operational for all streaming write operations. Add an HTTP monitor for the Flink JobManager REST API:

  1. In Vigilmon, click Add Monitor → HTTP / HTTPS.
  2. Enter: http://flink-jobmanager:8081/overview.
  3. Set Check interval to 1 minute.
  4. Set Expected HTTP status to 200.
  5. Click Save.

The Flink /overview endpoint returns cluster health including task manager count and available slots:

{
  "taskmanagers": 4,
  "slots-total": 16,
  "slots-available": 2,
  "jobs-running": 3,
  "jobs-finished": 0,
  "jobs-failed": 0
}

Alert if "jobs-failed" is greater than 0 by setting Expected response body does NOT contain to "jobs-failed":1 (or use a script-based check for non-zero values).

Step 3: Monitor Object Storage Reachability

All Paimon data files, manifest files, and snapshot files are stored in object storage (S3, HDFS, or OSS). An object storage failure immediately stops all Flink streaming write jobs. Monitor object storage reachability with a lightweight script:

For S3

#!/bin/bash
# /etc/cron.d/paimon-s3-health
*/2 * * * * monitoring aws s3 ls s3://your-paimon-bucket/ --region your-region \
  --max-items 1 > /dev/null 2>&1 && \
  curl -fsS "https://vigilmon.online/heartbeat/YOUR_S3_KEY" > /dev/null 2>&1

Set the Vigilmon heartbeat interval to 5 minutes. Pair this with an IAM permission check — AWS can silently return 403 errors if policies change, so log the response code:

STATUS=$(aws s3 ls s3://your-paimon-bucket/ --region your-region 2>&1)
if echo "$STATUS" | grep -q "AccessDenied"; then
  # Don't ping — let the heartbeat expire to trigger alert
  exit 1
fi
curl -fsS "https://vigilmon.online/heartbeat/YOUR_S3_KEY"

For HDFS

#!/bin/bash
# /etc/cron.d/paimon-hdfs-health
*/2 * * * * hdfs hdfs dfs -ls /paimon/ > /dev/null 2>&1 && \
  curl -fsS "https://vigilmon.online/heartbeat/YOUR_HDFS_KEY" > /dev/null 2>&1

Step 4: Monitor Paimon Catalog Service

Paimon tables are registered in a catalog — either a Hive Metastore (for shared access with Spark, Trino, and Hive) or a filesystem-based catalog. Catalog failures prevent DDL operations (creating/altering Paimon tables) and can prevent query planning.

For Hive Metastore

Add a TCP port monitor for the Hive Metastore Thrift interface:

  1. In Vigilmon, click Add Monitor → TCP Port.
  2. Enter hive-metastore:9083.
  3. Set Check interval to 1 minute.
  4. Click Save.

For a deeper check, create a heartbeat that confirms the Metastore responds to a SHOW DATABASES query:

#!/bin/bash
# /etc/cron.d/hive-metastore-health
*/5 * * * * hive beeline -u "jdbc:hive2://hive-metastore:10000" \
  -e "SHOW DATABASES;" > /dev/null 2>&1 && \
  curl -fsS "https://vigilmon.online/heartbeat/YOUR_METASTORE_KEY" > /dev/null 2>&1

For Filesystem Catalog

For filesystem-based catalogs, verify the catalog root path is accessible and the catalog metadata file is readable:

#!/bin/bash
# /etc/cron.d/paimon-catalog-health
*/5 * * * * monitoring test -f /path/to/paimon/catalog.conf && \
  aws s3 ls "s3://your-bucket/paimon-catalog/" > /dev/null 2>&1 && \
  curl -fsS "https://vigilmon.online/heartbeat/YOUR_CATALOG_KEY" > /dev/null 2>&1

Step 5: Monitor Compaction Health

Paimon's LSM-Tree compaction merges Level-0 (L0) files written by streaming jobs into larger, more read-efficient files in higher levels. Compaction lag — measured by the number of L0 files awaiting compaction — is the most important operational metric for Paimon read performance.

Monitor compaction via Flink metrics exposed by the Paimon Flink connector. Create a script that queries the Flink metrics API for compaction-related counters:

#!/bin/bash
# /opt/monitoring/check-paimon-compaction.sh
FLINK_URL="http://flink-jobmanager:8081"
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_COMPACTION_KEY"
MAX_L0_FILES=50  # Alert threshold: more than 50 L0 files is a compaction lag warning

# Get the Paimon write job ID
JOB_ID=$(curl -sf "$FLINK_URL/jobs/overview" | \
  python3 -c "
import sys, json
jobs = json.load(sys.stdin).get('jobs', [])
paimon_jobs = [j for j in jobs if 'paimon' in j.get('name','').lower() and j['status'] == 'RUNNING']
print(paimon_jobs[0]['id'] if paimon_jobs else '')
" 2>/dev/null)

if [ -z "$JOB_ID" ]; then
  # No running Paimon job — heartbeat will expire
  exit 1
fi

# Check if the job has compaction-related metrics available
# Paimon exposes metrics via Flink's metric system
METRICS=$(curl -sf "$FLINK_URL/jobs/$JOB_ID/metrics?get=numCompactions")
if [ $? -eq 0 ]; then
  curl -fsS "$HEARTBEAT_URL"
fi

Run every 5 minutes with a 15-minute heartbeat interval. For Paimon tables with high write throughput, also monitor compaction via the dedicated Paimon compaction job (Paimon 0.8+ supports a standalone compaction job):

# Start a dedicated Paimon compaction job
flink run -c org.apache.paimon.flink.action.CompactAction \
  /path/to/paimon-flink-connector.jar \
  --warehouse s3://your-bucket/paimon \
  --database your_db \
  --table your_table

Monitor this compaction job's RUNNING status via the Flink API just like the write job in Step 1.

Step 6: Monitor Snapshot Expiration

Paimon retains multiple historical snapshots for time travel queries. The snapshot.num-retained.max configuration controls the maximum number of retained snapshots. When snapshot expiration runs successfully, old snapshots are deleted and storage is reclaimed.

Monitor snapshot health by counting snapshots directly in object storage:

#!/bin/bash
# /etc/cron.d/paimon-snapshot-health
MAX_SNAPSHOTS=100

SNAPSHOT_COUNT=$(aws s3 ls "s3://your-bucket/paimon/your_db/your_table/snapshot/" | \
  wc -l | tr -d ' ')

if [ "$SNAPSHOT_COUNT" -lt "$MAX_SNAPSHOTS" ]; then
  curl -fsS "https://vigilmon.online/heartbeat/YOUR_SNAPSHOT_KEY"
fi

Set the Vigilmon heartbeat interval to 30 minutes. If snapshot count grows beyond your retention policy (indicating expiration is failing), the heartbeat stops and you are alerted within 30 minutes.

Step 7: Monitor Changelog Freshness

For real-time analytics use cases, the lag between a source database change and the corresponding Paimon snapshot determines query freshness. Monitor changelog lag by comparing the Paimon table's latest snapshot timestamp against the current time:

#!/bin/bash
# /opt/monitoring/check-paimon-freshness.sh
MAX_LAG_SECONDS=300  # Alert if Paimon is more than 5 minutes behind
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_FRESHNESS_KEY"

# Read the latest snapshot metadata from S3
LATEST_SNAPSHOT=$(aws s3 ls "s3://your-bucket/paimon/your_db/your_table/snapshot/" \
  --recursive | sort | tail -1 | awk '{print $4}')

if [ -z "$LATEST_SNAPSHOT" ]; then
  exit 1  # No snapshot found — let heartbeat expire
fi

# Get snapshot commit timestamp from the snapshot JSON file
COMMIT_TIME=$(aws s3 cp "s3://your-bucket/paimon/$LATEST_SNAPSHOT" - 2>/dev/null | \
  python3 -c "import sys,json; print(json.load(sys.stdin).get('timeMillis',0))")

NOW_MS=$(date +%s%3N)
LAG_MS=$((NOW_MS - COMMIT_TIME))
LAG_SECONDS=$((LAG_MS / 1000))

if [ "$LAG_SECONDS" -lt "$MAX_LAG_SECONDS" ]; then
  curl -fsS "$HEARTBEAT_URL"
fi

Run every 2 minutes. Set the Vigilmon heartbeat interval to 10 minutes. If lag exceeds 5 minutes, the heartbeat expires and you are alerted.

Step 8: Configure Alerting

Create a dedicated Paimon alert channel in Vigilmon:

  1. Go to Alert Channels → Add Channel.
  2. Configure Slack, PagerDuty, or email.
  3. For storage and compaction failures (which affect data freshness but not uptime), use a lower-priority channel.
  4. For Flink write job failures and object storage failures (which cause data loss risk), use PagerDuty or high-priority Slack.

Recommended alert thresholds:

| Monitor | Alert Condition | Severity | |---|---|---| | Flink write job heartbeat | Missed for 10 min | Critical | | Flink JobManager API | Down for 2 checks | Critical | | Object storage heartbeat | Missed for 5 min | Critical | | Hive Metastore TCP | Down for 1 check | High | | Compaction heartbeat | Missed for 15 min | High | | Snapshot count heartbeat | Missed for 30 min | Medium | | Changelog freshness heartbeat | Missed for 10 min | High |

Conclusion

Apache Paimon's streaming lakehouse capabilities rest on a foundation of object storage, LSM-Tree compaction, and Flink streaming integration — all of which can fail independently and silently. With Vigilmon monitoring the Flink write jobs, object storage reachability, Hive Metastore, compaction health, and snapshot freshness, you have end-to-end visibility into your Paimon data pipeline. Problems that would previously surface only when an analyst reports stale data now trigger alerts within minutes.

Start with the Flink write job heartbeat and object storage check — these are the critical path for all Paimon data ingestion. Then add compaction and freshness monitoring for production tables with real-time SLAs. Sign up for a free Vigilmon account to get started.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles