Monitoring Apache Ballista with Vigilmon

Apache Ballista is a distributed query execution engine that scales DataFusion SQL analytics across a cluster of nodes. When you self-host a Ballista cluster, you're running a scheduler that coordinates all query plans, a fleet of executors that crunch Arrow RecordBatches, and an object-store shuffle layer that exchanges intermediate results between query stages. A silent scheduler crash halts all incoming queries; a single executor dropout reduces parallelism without any visible error to the client. Vigilmon gives you continuous visibility into every layer of the Ballista stack — from gRPC reachability to query latency percentiles.

What You'll Set Up

• Ballista scheduler process health monitor (gRPC port)
• Executor registration count heartbeat
• Query submission success rate monitor
• Shuffle data exchange health check
• Flight SQL endpoint reachability monitor
• Query latency alert on p99 regression
• Scheduler queue depth heartbeat
• Executor-to-scheduler heartbeat loss alert

Prerequisites

• A running Ballista cluster (scheduler + one or more executors), deployed via Docker Compose, Kubernetes, or bare metal
• Ballista scheduler gRPC port accessible (default: 50050)
• Ballista Flight SQL port accessible (default: 50060)
• A Prometheus or custom metrics endpoint if you want numeric metric monitors
• A free Vigilmon account

Step 1: Monitor the Ballista Scheduler (gRPC Port)

The Ballista scheduler is the single coordinator for all query execution. If the scheduler goes down, every SQL query submitted to the cluster is rejected immediately. Add a TCP port monitor on the scheduler gRPC port so you know the moment the process becomes unreachable.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to TCP Port.
3. Enter Host: ballista-scheduler.yourdomain.com (or the LAN IP of your scheduler node).
4. Enter Port: 50050 (the default Ballista scheduler gRPC port).
5. Set Check interval to 1 minute.
6. Click Save.

If your scheduler exposes an HTTP REST status endpoint (common in custom Ballista deployments), add a second HTTP monitor:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://ballista-scheduler.yourdomain.com:50050/metrics or the health URL your deployment exposes.
3. Expected HTTP status: 200.
4. Check interval: 1 minute.
5. Click Save.

Alert recommendation: Set this monitor to alert after 1 missed check and page immediately — a scheduler outage stops 100% of query traffic.

Step 2: Monitor Executor Registration via Heartbeat

Ballista executors register with the scheduler on startup. If executors fail to register (JVM OOM, network partition, misconfigured scheduler address), the cluster silently loses query parallelism. Use a Vigilmon cron heartbeat driven by a lightweight script that polls the scheduler's executor list and sends a ping only when the count meets your minimum.

Add a script to one of your scheduler or monitoring nodes:

#!/bin/bash
# Check that Ballista executor count meets minimum threshold
# Requires ballista-cli or a custom gRPC health check binary

SCHEDULER="ballista-scheduler.yourdomain.com:50050"
MIN_EXECUTORS=2
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"

# Replace with your actual executor count query method
EXECUTOR_COUNT=$(ballista-cli --scheduler "$SCHEDULER" executors list 2>/dev/null | grep -c "ACTIVE")

if [ "$EXECUTOR_COUNT" -ge "$MIN_EXECUTORS" ]; then
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

In Vigilmon:

1. Click Add Monitor → Cron Heartbeat.
2. Set Name to Ballista Executor Count.
3. Set Expected interval to 2 minutes.
4. Set Grace period to 3 minutes.
5. Copy the heartbeat URL into the script above.
6. Schedule the script via cron: */2 * * * * /opt/monitoring/check-ballista-executors.sh.
7. Click Save.

Alert recommendation: If the heartbeat goes silent, it means either the check script failed or executor count dropped below minimum — both warrant an alert.

Step 3: Monitor Query Submission Success Rate

SQL queries arrive at the Ballista scheduler via gRPC or Flight SQL. A high query rejection rate indicates scheduler overload, an exhausted executor pool, or a configuration error. If your Ballista deployment exposes Prometheus metrics (common in production setups), configure a Vigilmon HTTP monitor that checks the metrics endpoint for elevated rejection counts.

Add a keyword monitor on your Prometheus metrics scrape endpoint:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://ballista-scheduler.yourdomain.com:9090/metrics (your Prometheus scrape endpoint).
3. Expected HTTP status: 200.
4. Under Keyword check, enter ballista_query_submissions_total to verify the metric is being exported.
5. Check interval: 1 minute.
6. Click Save.

For a more direct check, write a small probe script that submits a trivial SQL query (SELECT 1) to the scheduler and exits non-zero on failure, then drive it with a Vigilmon heartbeat:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"

# Submit a trivial query via Flight SQL client
result=$(echo "SELECT 1" | ballista-cli --scheduler ballista-scheduler.yourdomain.com:50060 2>&1)

if echo "$result" | grep -q "1"; then
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

Alert recommendation: A query submission failure should alert within 2 minutes — clients experience immediate errors.

Step 4: Monitor Query Stage Execution Success Rate

Ballista breaks each SQL query into stages executed in parallel across executors. A stage failure causes the entire query to fail. Monitor stage health via a heartbeat script that queries the scheduler's active job list and checks for failed stages.

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"

# Query the Ballista scheduler for failed jobs in the past 5 minutes
# Adapt to your monitoring/metrics pipeline
FAILED=$(curl -s http://ballista-scheduler.yourdomain.com:9090/metrics \
  | grep 'ballista_query_stage_failures_total' \
  | awk '{print $2}')

# Send heartbeat only if no recent stage failures (or if metric is stable)
if [ -z "$FAILED" ] || [ "$FAILED" = "0" ]; then
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

Add a cron heartbeat in Vigilmon with a 2-minute expected interval and a 5-minute grace period to absorb transient stage failures during large query bursts.

Step 5: Monitor Shuffle Data Exchange Health

Ballista exchanges intermediate query results (shuffle data) between stages via object storage (S3, GCS, or local disk). A shuffle write or read failure causes cascading query stage failures. Monitor the shuffle layer by checking that your object storage endpoint is reachable.

For S3-backed shuffle storage:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: https://s3.amazonaws.com or your MinIO/local S3-compatible endpoint (e.g., http://minio.yourdomain.com:9000/minio/health/live).
3. Expected HTTP status: 200.
4. Check interval: 1 minute.
5. Click Save.

For local disk shuffle (single-node dev setups), monitor the executor disk fill level via a heartbeat:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"
SHUFFLE_DIR="/tmp/ballista/shuffle"
THRESHOLD=85  # percent full

USAGE=$(df "$SHUFFLE_DIR" | awk 'NR==2 {gsub(/%/,""); print $5}')
if [ "$USAGE" -lt "$THRESHOLD" ]; then
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

Step 6: Monitor the Flight SQL Endpoint

Ballista exposes Apache Arrow Flight SQL on a dedicated port, enabling external clients (DBeaver, Tableau, JDBC/ODBC drivers) to submit SQL queries. If Flight SQL goes down, these clients get connection-refused errors while the gRPC API may still work fine.

1. Click Add Monitor → TCP Port.
2. Host: ballista-scheduler.yourdomain.com.
3. Port: 50060 (the default Ballista Flight SQL port).
4. Check interval: 1 minute.
5. Click Save.

Alert recommendation: Alert on the first missed check — external clients have no retry path when the port is closed.

Step 7: Monitor Query Latency (p99 Regression)

End-to-end distributed query latency is your primary performance SLO. Ballista adds scheduling and shuffle overhead on top of DataFusion's single-node baseline. Set up a synthetic query latency probe that runs a representative benchmark query and sends a heartbeat only when latency stays within your SLO.

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"
MAX_LATENCY_MS=5000  # 5 seconds p99 SLO for your benchmark query

START=$(date +%s%3N)
echo "SELECT COUNT(*) FROM your_benchmark_table" \
  | ballista-cli --scheduler ballista-scheduler.yourdomain.com:50060 2>/dev/null
END=$(date +%s%3N)

ELAPSED=$((END - START))
if [ "$ELAPSED" -lt "$MAX_LATENCY_MS" ]; then
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

Schedule this probe every 5 minutes. If query latency exceeds your SLO, the heartbeat goes silent and Vigilmon alerts you.

Step 8: Monitor Scheduler Queue Depth

When executors can't keep up with incoming queries, the scheduler builds a backlog queue. A deep queue means clients wait longer and the cluster is undersupplied with executor capacity. Monitor queue depth via a Prometheus gauge if your scheduler exports it:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://ballista-scheduler.yourdomain.com:9090/metrics.
3. Expected HTTP status: 200.
4. Under Keyword check, enter ballista_pending_queries to verify the queue metric is exported.
5. Check interval: 1 minute.
6. Click Save.

For a threshold-based alert, add a heartbeat script:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"
MAX_QUEUE=10

QUEUE_DEPTH=$(curl -s http://ballista-scheduler.yourdomain.com:9090/metrics \
  | grep 'ballista_pending_queries ' | awk '{print $2}')

if [ -z "$QUEUE_DEPTH" ] || [ "${QUEUE_DEPTH%.*}" -le "$MAX_QUEUE" ]; then
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

Alert threshold: Queue depth > 10 is your signal to add executor nodes.

Step 9: Configure Alerting

In Vigilmon, set up an alert channel for your Ballista cluster:

1. Go to Alerts → Add Alert Channel.
2. Choose Email, Slack, PagerDuty, or Webhook.
3. Apply the channel to all Ballista monitors.

Recommended per-monitor alert thresholds:

| Monitor | Alert After | Severity | |---|---|---| | Scheduler gRPC port | 1 missed check | Critical | | Executor count heartbeat | 1 missed check | Critical | | Query submission probe | 2 missed checks | Critical | | Flight SQL port | 1 missed check | High | | Shuffle storage endpoint | 2 missed checks | High | | Query latency SLO | 1 missed check | High | | Scheduler queue depth | 1 missed check | Medium | | Stage execution heartbeat | 2 missed checks | High |

Conclusion

A Ballista cluster distributes failure modes across schedulers, executors, and shuffle storage — any of which can degrade independently and silently. With Vigilmon you get a heartbeat or port check on each layer: scheduler reachability, executor registration count, query submission health, Flight SQL availability, shuffle storage, and latency SLO compliance. When any layer degrades, you know before your users do.

Get started at vigilmon.online — free for up to 5 monitors.