Monitoring Apache SeaTunnel with Vigilmon

Apache SeaTunnel is the unified data integration platform for modern data stacks — 300+ connectors, support for Flink, Spark, and its own Zeta execution engine, CDC support, and schema evolution out of the box. When SeaTunnel is healthy, your ETL pipelines silently move terabytes between databases, data lakes, and SaaS systems. When a connector fails or the Zeta engine master crashes, pipelines stall and data stops flowing — often without any external alert. Vigilmon provides the external monitoring layer that SeaTunnel lacks by default: engine process health, pipeline success rates via heartbeats, REST API uptime, and connector throughput monitoring.

What You'll Set Up

• HTTP health monitor for the SeaTunnel REST API
• Cron heartbeats for each production data pipeline
• TCP port monitor for the SeaTunnel Zeta engine master
• HTTP monitor for the SeaTunnel web console (if deployed)
• SSL certificate alerts for SeaTunnel API endpoints

Prerequisites

• Apache SeaTunnel 2.3+ deployed with the Zeta engine, Flink, or Spark
• SeaTunnel REST API accessible (default port 5801 on the Zeta engine)
• A free Vigilmon account

Why Monitoring SeaTunnel Matters

SeaTunnel ETL pipelines run in the background — no user is watching the console when a MySQL CDC connector loses its binlog position or an Elasticsearch sink connector starts throwing 429 errors. The consequences of a silent failure depend on the pipeline:

• CDC pipelines — a stalled CDC connector means your data lake falls behind the source database; queries return stale data
• Batch ETL jobs — a failed nightly load leaves dashboards showing yesterday's numbers with no warning
• Streaming pipelines — Kafka consumer lag grows unboundedly until the consumer group offset is reset
• Schema evolution — a source schema change can break a connector silently, causing type mismatch errors that drop records

External monitoring with Vigilmon catches all of these before downstream consumers notice.

Step 1: Monitor the SeaTunnel REST API

The SeaTunnel Zeta engine exposes a REST API on port 5801 for job management (submit, stop, query status). This is the primary programmatic interface to SeaTunnel. Add a Vigilmon HTTP monitor:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the SeaTunnel REST API health endpoint:

http://seatunnel-master:5801/hazelcast/rest/cluster

4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

For the Zeta engine, the cluster info endpoint returns member count and cluster state:

{
  "members": [...],
  "connectionCount": 3,
  "allConnectionCount": 3
}

Alternatively, probe the job list endpoint which confirms the API is fully functional:

http://seatunnel-master:5801/v3/job/list/running

Step 2: Monitor the Zeta Engine Master Process

The SeaTunnel Zeta master coordinates job scheduling across worker nodes. A master crash causes all running jobs to fail and prevents new submissions. Add a TCP monitor for the Hazelcast cluster port:

1. In Vigilmon, click Add Monitor → TCP Port.
2. Enter seatunnel-master:5801.
3. Set Check interval to 1 minute.
4. Click Save.

For self-hosted deployments where the master runs as a service, add a cron heartbeat to confirm the process is alive:

# /etc/cron.d/seatunnel-master-heartbeat
*/2 * * * * seatunnel pgrep -f "SeaTunnel-Starter" && \
  curl -fsS "https://vigilmon.online/heartbeat/YOUR_MASTER_KEY" > /dev/null 2>&1

Set the Vigilmon heartbeat interval to 5 minutes. If the master process dies, the heartbeat expires and you get alerted within 5 minutes.

Step 3: Monitor Worker Node Health

SeaTunnel Zeta workers execute the actual pipeline tasks. Monitor worker connectivity via the Zeta REST API:

# Check worker node count via the Hazelcast API
curl -s http://seatunnel-master:5801/hazelcast/rest/cluster | \
  python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d['members']))"

Create a Vigilmon keyword monitor that checks for a minimum expected member count:

1. Add an HTTP / HTTPS monitor for http://seatunnel-master:5801/hazelcast/rest/cluster.
2. Set Expected response body contains to the number of members you expect (e.g., "memberCount":3 or match the members array length).
3. Alert if the count drops below your minimum cluster size.

Step 4: Monitor Individual Pipeline Success with Heartbeats

Each SeaTunnel pipeline is defined in a configuration file and executed as a job. Production pipelines should send a Vigilmon heartbeat on successful completion:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Name the monitor after your pipeline: MySQL → ClickHouse daily sync.
3. Set the expected ping interval to match your pipeline schedule (e.g., 25 hours for a daily pipeline with 1 hour buffer).
4. Copy the heartbeat URL.

Wrap your SeaTunnel job submission with a heartbeat ping on success:

#!/bin/bash
# run-pipeline.sh
PIPELINE_CONFIG="/opt/seatunnel/jobs/mysql-to-clickhouse.conf"
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_PIPELINE_KEY"
SEATUNNEL_API="http://seatunnel-master:5801"

# Submit the job and capture the job ID
JOB_ID=$(curl -sf -X POST "$SEATUNNEL_API/v3/job/submit" \
  -H "Content-Type: application/json" \
  -d "{\"jobConfig\": $(cat $PIPELINE_CONFIG)}" | \
  python3 -c "import sys,json; print(json.load(sys.stdin)['jobId'])")

# Poll for job completion
while true; do
  STATUS=$(curl -sf "$SEATUNNEL_API/v3/job/$JOB_ID/detail" | \
    python3 -c "import sys,json; print(json.load(sys.stdin)['jobStatus'])")
  
  if [ "$STATUS" = "FINISHED" ]; then
    curl -fsS "$HEARTBEAT_URL" -d "msg=pipeline_success"
    exit 0
  elif [ "$STATUS" = "FAILED" ] || [ "$STATUS" = "CANCELED" ]; then
    echo "Pipeline failed with status: $STATUS" >&2
    exit 1
  fi
  sleep 30
done

For streaming pipelines that run continuously, send a heartbeat every N minutes from within the application:

import requests
import time
import threading

def heartbeat_loop(url: str, interval_seconds: int = 300):
    """Send a Vigilmon heartbeat every `interval_seconds` to confirm the pipeline is running."""
    while True:
        try:
            requests.get(url, timeout=10)
        except Exception:
            pass
        time.sleep(interval_seconds)

# Start heartbeat in background thread
threading.Thread(
    target=heartbeat_loop,
    args=("https://vigilmon.online/heartbeat/YOUR_KEY", 300),
    daemon=True
).start()

Step 5: Monitor Kafka Consumer Lag for Streaming Pipelines

SeaTunnel streaming pipelines reading from Kafka can fall behind if the processing rate drops below the ingestion rate. Monitor consumer group lag to catch pipeline slowdowns before they become failures:

Add a cron job that checks consumer group lag and pings Vigilmon with the lag value:

#!/bin/bash
# /etc/cron.d/seatunnel-kafka-lag
# Runs every 5 minutes
*/5 * * * * kafka kafka-consumer-groups.sh \
  --bootstrap-server kafka:9092 \
  --group seatunnel-pipeline-group \
  --describe | \
  awk 'NR>1 {sum += $5} END {print sum}' | \
  xargs -I{} curl -fsS \
  "https://vigilmon.online/heartbeat/YOUR_LAG_KEY?lag={}" > /dev/null 2>&1

In Vigilmon, configure the heartbeat to alert if the ping stops — this catches both the monitoring script failing and the consumer group becoming inactive. For lag thresholds, add custom logic to skip the Vigilmon ping if lag exceeds your acceptable threshold, causing the heartbeat to expire as an indirect alert.

Step 6: Monitor the SeaTunnel Web Console

If you are running the SeaTunnel web UI (the optional management console), add a Vigilmon HTTP monitor:

1. In Vigilmon, add an HTTP / HTTPS monitor for the web console URL.
2. Default port is typically 8801 or behind a reverse proxy: https://seatunnel-ui.yourdomain.com.
3. Set Check interval to 2 minutes.
4. Set Expected HTTP status to 200.

Enable SSL certificate monitoring if the console is HTTPS:

1. Open the monitor settings.
2. Enable Monitor SSL certificate.
3. Set Alert when certificate expires in less than 21 days.

Step 7: Monitor Pipeline Retry Exhaustion

SeaTunnel retries failed tasks before marking a job as FAILED. When a pipeline exhausts all retries, it is a signal that the underlying connector issue is persistent (source unavailability, schema mismatch, sink permission error). Poll for retry-exhausted jobs:

#!/bin/bash
# Check for any FAILED jobs that exhausted retries
FAILED_COUNT=$(curl -sf "http://seatunnel-master:5801/v3/job/list/finished" | \
  python3 -c "
import sys, json
jobs = json.load(sys.stdin)
failed = [j for j in jobs.get('data', []) if j.get('jobStatus') == 'FAILED']
print(len(failed))
")

if [ "$FAILED_COUNT" = "0" ]; then
  curl -fsS "https://vigilmon.online/heartbeat/YOUR_FAILED_JOB_KEY"
fi

Run this check every 5 minutes via cron. Set the Vigilmon heartbeat interval to 10 minutes — if any jobs enter a FAILED final state, the heartbeat stops and you get alerted within 10 minutes.

Step 8: Configure Alerting

Create a dedicated alert channel for SeaTunnel:

1. In Vigilmon, go to Alert Channels → Add Channel.
2. Set up Slack, PagerDuty, or email alerts.
3. For high-severity pipeline failures, use PagerDuty or a dedicated Slack channel with on-call paging.

Recommended alert thresholds:

| Monitor | Alert Condition | Severity | |---|---|---| | SeaTunnel REST API | Down for 2 checks (2 min) | Critical | | Zeta master process | Heartbeat missed 5 min | Critical | | Production pipeline heartbeat | Missed for pipeline interval + buffer | High | | Kafka consumer lag | Heartbeat expired | High | | SeaTunnel web console | Down for 5 min | Medium | | Failed job count | Any FAILED job detected | High |

Conclusion

SeaTunnel data pipelines are designed to run silently in the background — which is exactly what makes silent failures so dangerous. With Vigilmon monitoring the SeaTunnel REST API, the Zeta engine master, individual pipeline completion heartbeats, and Kafka consumer lag, you have end-to-end coverage of your ETL infrastructure. Pipeline failures surface within minutes rather than being discovered when a downstream analyst notices stale data the next morning.

Start with the REST API monitor and a heartbeat for your most critical pipeline, then expand coverage to your full pipeline fleet. Sign up for a free Vigilmon account to get started.