Monitoring Apache StreamPark with Vigilmon

Apache StreamPark (formerly StreamX) is the unified management platform for Apache Flink and Spark streaming jobs — handling development, deployment, lifecycle management, and alerting for entire fleets of streaming applications. When StreamPark is healthy, your data pipelines run. When the console crashes or a Flink job quietly enters FAILED status, you need to know immediately. Vigilmon gives you the external monitoring layer that StreamPark's built-in alerts cannot provide: console uptime, Flink cluster reachability, database health, and cron heartbeats for critical background services.

What You'll Set Up

• HTTP uptime monitor for the StreamPark web console
• Cron heartbeat for the StreamPark Spring Boot application process
• Database connectivity monitor for the StreamPark MySQL/PostgreSQL backend
• Flink JobManager API reachability monitor
• HTTP monitors for each production Flink job's REST endpoint
• SSL certificate alerts for the StreamPark console domain

Prerequisites

• Apache StreamPark 2.x deployed on Kubernetes, YARN, or standalone
• StreamPark console accessible over HTTP/HTTPS
• A free Vigilmon account

Why Monitoring StreamPark Matters

StreamPark is the single control plane for all your Flink jobs. A console failure means operators cannot submit new jobs, cannot restart failed jobs, and cannot view checkpoint status — all job management is blocked. The built-in DingTalk/Slack alerts StreamPark sends for individual job failures are useless if the StreamPark console itself is down. External monitoring catches:

• Console crash — Spring Boot process terminated, Kubernetes pod CrashLoopBackOff, or nginx reverse proxy failure
• Database connectivity loss — StreamPark console loses its MySQL/PostgreSQL connection and cannot retrieve or store job configurations
• Flink cluster unreachable — StreamPark cannot submit jobs or query job status if the Flink JobManager REST API is down
• Checkpoint failures — Flink checkpointing failures are silent at the platform level unless you actively poll job metrics
• Build pipeline stalls — StreamPark compiles Flink application JARs before deployment; a stalled Maven/Gradle build blocks all new deployments

Step 1: Monitor the StreamPark Web Console

The StreamPark console is a Spring Boot application that serves the Vue.js frontend and the REST API on a single port (default 10000). Add a Vigilmon HTTP monitor:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the StreamPark console URL: https://streampark.yourdomain.com (or http://your-server-ip:10000 for direct access).
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

If StreamPark exposes an actuator health endpoint (enabled via Spring Boot Actuator), use it instead:

https://streampark.yourdomain.com/actuator/health

The actuator health endpoint returns a JSON body with detailed component status including database connectivity. Set Expected response body contains to "status":"UP" for a richer signal.

Step 2: Verify StreamPark Console Process with a Heartbeat

For deployments where StreamPark runs as a background service (systemd, supervisor, or a Kubernetes sidecar), configure a Vigilmon cron heartbeat so you know the process is alive even if the HTTP check passes through a cached response:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval to 5 minutes.
3. Copy the heartbeat URL: https://vigilmon.online/heartbeat/YOUR_KEY.
4. Add a cron job on the StreamPark host to ping Vigilmon after confirming the process is alive:

# /etc/cron.d/streampark-heartbeat
*/5 * * * * streampark curl -fsS --retry 3 \
  "https://vigilmon.online/heartbeat/YOUR_KEY" \
  --data-urlencode "msg=$(systemctl is-active streampark-console)" > /dev/null 2>&1

For Kubernetes deployments, add the liveness probe to the StreamPark console pod and pair it with a curl sidecar:

livenessProbe:
  httpGet:
    path: /actuator/health
    port: 10000
  initialDelaySeconds: 60
  periodSeconds: 30
  failureThreshold: 3

Step 3: Monitor StreamPark's Database Backend

StreamPark stores all job configurations, build history, alert rules, and team settings in MySQL or PostgreSQL. A database connectivity failure causes the console to reject all user interactions (job submissions, alert rule changes) even if the Spring Boot process is still running.

Add a TCP port monitor for the database:

1. In Vigilmon, click Add Monitor → TCP Port.
2. Enter the database host and port:
   • MySQL: your-db-host:3306
   • PostgreSQL: your-db-host:5432
3. Set Check interval to 1 minute.
4. Click Save.

For a deeper connectivity check, expose a custom database health endpoint from StreamPark (if using Spring Boot Actuator with management.endpoint.health.show-details=always):

https://streampark.yourdomain.com/actuator/health/db

Set the Vigilmon monitor to check for "status":"UP" in the response body. This confirms the JDBC connection pool is healthy, not just that the TCP port is open.

Step 4: Monitor Flink Cluster Reachability

StreamPark submits jobs to Flink via the Flink REST API (default port 8081 on the JobManager). If the Flink cluster is unreachable, StreamPark cannot submit new jobs, cannot cancel running jobs, and displays stale status for all jobs in the console. Add a Vigilmon HTTP monitor for each Flink cluster StreamPark manages:

1. In Vigilmon, click Add Monitor → HTTP / HTTPS.
2. Enter the Flink JobManager REST API URL: http://flink-jobmanager:8081/overview.
3. Set Check interval to 1 minute.
4. Set Expected HTTP status to 200.
5. Click Save.

The /overview endpoint returns cluster metadata including running jobs, task manager count, and available task slots. Check for a healthy response body:

{
  "taskmanagers": 3,
  "slots-total": 12,
  "slots-available": 4,
  "jobs-running": 2
}

If slots-available drops to 0, your cluster is at capacity and new job submissions will queue indefinitely. Configure a Vigilmon keyword check on "slots-available":0 to alert early.

Step 5: Monitor Individual Flink Job Status

StreamPark manages the lifecycle of individual Flink jobs. Use the Flink REST API to check the status of each production job. Each running Flink job has a unique Job ID visible in the StreamPark console and in the Flink dashboard.

Create a Vigilmon HTTP monitor per production job:

1. Find the Job ID in the StreamPark console or via the Flink REST API:

curl http://flink-jobmanager:8081/jobs/overview

2. In Vigilmon, add an HTTP / HTTPS monitor for each critical job:

http://flink-jobmanager:8081/jobs/YOUR_JOB_ID

3. Set Expected response body contains to "state":"RUNNING".
4. Set Check interval to 1 minute.

If the job transitions to FAILED, CANCELED, or FINISHED (unexpectedly), Vigilmon triggers an alert. For jobs where FINISHED is a valid terminal state (batch jobs), check for FAILED instead:

Expected response body does NOT contain: "state":"FAILED"

Step 6: Monitor Flink Checkpoint Health

Checkpointing is Flink's fault tolerance mechanism — periodic snapshots of operator state that enable recovery without data loss. Checkpoint failures are one of the most important signals in a production Flink deployment.

Poll checkpoint metrics via the Flink REST API:

# Check latest checkpoint status for a job
curl http://flink-jobmanager:8081/jobs/YOUR_JOB_ID/checkpoints

Create a lightweight health script that pings Vigilmon when checkpoints are completing:

#!/bin/bash
# /opt/monitoring/check-checkpoints.sh
JOB_ID="YOUR_JOB_ID"
FLINK_URL="http://flink-jobmanager:8081"
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_CHECKPOINT_KEY"

STATUS=$(curl -sf "$FLINK_URL/jobs/$JOB_ID/checkpoints" | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  print(d['counts']['completed'])" 2>/dev/null)

if [ -n "$STATUS" ] && [ "$STATUS" -gt "0" ]; then
  curl -fsS "$HEARTBEAT_URL" > /dev/null
fi

Run this script every 5 minutes via cron and set the Vigilmon heartbeat interval to 10 minutes. If checkpoints stop completing, the heartbeat expires and Vigilmon alerts you.

Step 7: Monitor the StreamPark Build Service

StreamPark compiles Flink applications (Maven or Gradle builds) before deployment. A stalled build service blocks all new job deployments silently — StreamPark will show builds as BUILDING indefinitely. Monitor the build service health via the StreamPark REST API (if your deployment exposes it):

# List recent build records
curl -H "Authorization: Bearer YOUR_TOKEN" \
  https://streampark.yourdomain.com/api/app/builds?limit=5

Add a Vigilmon cron heartbeat that confirms builds are completing within your expected build time (typically 2–5 minutes for a typical Flink application):

1. In your CI/CD pipeline or build trigger script, add a Vigilmon ping after each successful build:

# After a successful StreamPark build
curl -fsS "https://vigilmon.online/heartbeat/YOUR_BUILD_KEY" \
  -d "msg=build_completed"

2. Set the heartbeat interval to match your build frequency — if you deploy at least once per hour, set it to 2 hours so a stuck build pipeline is caught within two deployment cycles.

Step 8: Configure Alerting

Tie all StreamPark monitors to a single alert channel so the on-call engineer gets one notification with full context:

1. In Vigilmon, go to Alert Channels → Add Channel.
2. Configure your preferred channel (Slack, PagerDuty, email, or webhook).
3. For Slack, use a dedicated #streampark-alerts channel:

Webhook URL: https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK

4. Assign this channel to all StreamPark monitors.

Recommended alert thresholds:

| Monitor | Alert Condition | |---|---| | StreamPark console HTTP | Down for 2 consecutive checks (2 min) | | StreamPark process heartbeat | Missed for 10 minutes | | Database TCP port | Down for 1 check | | Flink JobManager API | Down for 2 consecutive checks | | Production job status | State ≠ RUNNING for 1 check | | Checkpoint heartbeat | Missed for 10 minutes | | Build service heartbeat | Missed for configured build interval |

Step 9: SSL Certificate Monitoring

If StreamPark is exposed over HTTPS (strongly recommended for any console with production job management), add SSL certificate expiry monitoring:

1. Open the StreamPark console HTTP monitor in Vigilmon.
2. Enable Monitor SSL certificate.
3. Set Alert when certificate expires in less than 21 days.
4. Click Save.

For deployments using cert-manager on Kubernetes, a 21-day alert window gives you time to investigate cert-manager failures (ACME challenge errors, DNS propagation issues) before the certificate actually expires.

Conclusion

StreamPark is the operations center for your entire Flink streaming infrastructure — when it goes down, all job management stops. With Vigilmon monitoring the StreamPark console, its database backend, the Flink cluster it manages, and individual production jobs, you have layered coverage that catches problems at every level: from the StreamPark process itself down to per-job checkpoint failures. The combination of HTTP monitors for reachability and cron heartbeats for process liveness gives you confidence that your streaming infrastructure is healthy — and fast alerts when it is not.

Start with the console HTTP monitor and the Flink JobManager API check, then add per-job status monitors for your most critical production pipelines. Sign up for a free Vigilmon account to get started.