tutorial

Monitoring Apache Nemo with Vigilmon

Apache Nemo is a runtime-agnostic data processing framework that compiles pipelines for Spark and Flink. Here's how to monitor its driver, executors, optimization passes, and job health with Vigilmon.

Apache Nemo is an open source data processing framework from Seoul National University, donated to the Apache Software Foundation, that lets you write data processing pipelines once and compile them to run on Apache Spark, Apache Flink, or other execution engines. Nemo's DAG-based intermediate representation (IR) is transformed by pluggable optimization passes before being handed off to the target runtime — which means a single Nemo driver coordinates job submission, and any crash there takes every running job down with it. Vigilmon gives you continuous visibility into the Nemo driver, executor fleet, optimization pipeline, and the running jobs themselves so you catch failures before your data pipelines miss their SLAs.

What You'll Set Up

  • HTTP health probe for the Nemo driver/master REST API
  • Cron heartbeat for background job submission health
  • TCP port monitors for executor connectivity
  • Optimization pass failure alerting via log-scraping heartbeat
  • Job completion time tracking via custom metrics endpoint
  • Nemo web UI uptime monitor
  • Alert channels for driver crash and executor fleet changes

Prerequisites

  • Apache Nemo deployed (standalone or as a Beam runner on top of Flink/Spark)
  • Nemo driver REST API accessible (default port 8080 or as configured)
  • Nemo web dashboard accessible
  • A free Vigilmon account

Step 1: Monitor the Nemo Driver REST API

The Nemo driver is the master coordinator — it compiles IR, applies optimization passes, submits jobs to the target runtime, and tracks executor state. A driver crash halts every running Nemo job.

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. URL: http://your-nemo-host:8080/health (or the REST endpoint your Nemo deployment exposes — check NemoConf.MASTER_PORT).
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Under Keyword check, enter ok or running to confirm the response body signals a live driver, not just that the port is open.
  7. Click Save.

If your Nemo driver does not expose a dedicated /health route, use the job list endpoint instead:

http://your-nemo-host:8080/api/v1/jobs

An empty 200 response ([]) is still a valid liveness signal — it confirms the driver is running and the REST server is accepting connections.

Step 2: Monitor the Nemo Web UI

Nemo ships a web dashboard that shows running jobs, executor topology, and real-time metrics. If the UI goes down, operators lose visibility into the pipeline even if underlying processing continues.

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: http://your-nemo-host:8080/ (or the UI port, often the same as the REST API).
  3. Check interval: 2 minutes
  4. Expected HTTP status: 200
  5. Under Keyword check, enter Nemo to confirm the dashboard HTML loads.
  6. Click Save.

Step 3: Monitor Executor Connectivity via TCP

Nemo task executors run the actual work — they receive tasks from the driver and send metrics back. Add a TCP monitor for each executor host to catch network partition events before they cascade into task failures:

  1. Click Add Monitor → TCP Port.
  2. Host: executor hostname or IP.
  3. Port: the executor RPC port (default 10020 in many Nemo configurations — confirm in your executorConf).
  4. Check interval: 1 minute
  5. Click Save.

Repeat for each executor host. If you run executors dynamically, monitor the fixed entry-point hosts (e.g., YARN NodeManagers or Kubernetes node IPs) rather than ephemeral executor processes.

Step 4: Heartbeat for Job Submission Pipeline Health

Nemo's optimization passes run synchronously during job submission — if a pass fails, the job never reaches the execution engine. Add a cron heartbeat from your job submission wrapper to confirm the end-to-end submission path is healthy:

  1. Click Add Monitor → Cron Heartbeat.
  2. Set the expected interval to match your job submission frequency (e.g., 10 minutes for a pipeline that runs every 10 minutes, or 60 minutes for hourly batch jobs).
  3. Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).
  4. Wrap your Nemo job submission script to ping the heartbeat on successful submission:
#!/bin/bash
set -e

# Submit the Nemo job
java -cp nemo-dist.jar \
  org.apache.nemo.client.JobLauncher \
  -job_id my_pipeline \
  -optimization_policy org.apache.nemo.compiler.optimizer.pass.compiletime.composite.DefaultCompositePass \
  "$@"

# Signal successful job submission to Vigilmon
curl -fsS --retry 3 "https://vigilmon.online/heartbeat/abc123" > /dev/null

If the job submission fails (optimization pass error, driver unreachable, target runtime rejection), the heartbeat never fires and Vigilmon alerts you within one missed interval.

Step 5: Track Optimization Pass Health

Nemo applies IR optimization passes (data skew handling, locality scheduling) before compilation. A failed optimization pass aborts job submission silently unless your logs capture it. Export a pass-health metric from your driver:

Add a small HTTP endpoint to your Nemo driver that reports recent optimization pass results. If you're using a custom CompositeOptimizationPass, instrument it:

// In your custom pass implementation
@Override
public IRDAG apply(IRDAG dag) {
    long start = System.currentTimeMillis();
    try {
        IRDAG result = runOptimization(dag);
        MetricsCollector.record("optimization_pass_success", 1);
        MetricsCollector.record("optimization_pass_duration_ms",
            System.currentTimeMillis() - start);
        return result;
    } catch (Exception e) {
        MetricsCollector.record("optimization_pass_failure", 1);
        throw e;
    }
}

Then monitor the metrics endpoint:

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: http://your-nemo-host:8080/metrics/optimization
  3. Expected HTTP status: 200
  4. Keyword check: pass_failure":0 (zero failures in the last window)
  5. Click Save.

Step 6: Monitor Skew Detection and Mitigation

Nemo includes data skew handling as an optimization pass. Persistent skew that the optimizer can't resolve will stall your pipeline. Add a monitor on the skew detection metric:

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: http://your-nemo-host:8080/metrics/skew
  3. Expected HTTP status: 200
  4. Keyword check: unmitigated_skew":0
  5. Check interval: 5 minutes
  6. Click Save.

Step 7: Monitor Job Completion Time

Long-running or stuck Nemo jobs are a silent failure mode — the job appears to be running but is actually stalled on a failed executor or blocked shuffle. Expose a job duration endpoint from your driver and monitor it:

// Expose via REST: GET /api/v1/jobs/{jobId}/duration
// Returns {"jobId": "my_pipeline", "durationMs": 45000, "status": "RUNNING"}
  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: http://your-nemo-host:8080/api/v1/jobs/my_pipeline/duration
  3. Expected HTTP status: 200
  4. Keyword check: RUNNING or COMPLETED — alert if FAILED appears in the response body.
  5. Check interval: 2 minutes
  6. Click Save.

For batch jobs where you know the expected runtime, set a separate alert: if the job is still RUNNING after 2× the expected duration, page on-call.

Step 8: Beam Pipeline Integration Health

If you're using Nemo as an Apache Beam runner (via NemoRunner), monitor pipeline submission at the Beam level too:

import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
import requests

options = PipelineOptions([
    '--runner=NemoRunner',
    '--nemoMasterAddress=your-nemo-host:8081',
])

with beam.Pipeline(options=options) as p:
    result = (p
              | 'Read' >> beam.io.ReadFromText('gs://my-bucket/input')
              | 'Process' >> beam.Map(my_transform)
              | 'Write' >> beam.io.WriteToText('gs://my-bucket/output'))

# Ping heartbeat on successful pipeline submission
requests.get('https://vigilmon.online/heartbeat/beam123', timeout=5)

Create a separate Cron Heartbeat monitor with a 30 minute interval for Beam pipeline jobs, distinct from the raw Nemo job submission heartbeat.

Step 9: Metric Ingestion Health

Nemo executors ship runtime metrics (task duration, bytes shuffled, memory usage) back to the driver for display in the web UI. If metric ingestion breaks, you lose visibility into running jobs even if processing continues.

Add a monitor on the metrics ingestion endpoint:

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: http://your-nemo-host:8080/metrics/ingestion-health
  3. Expected HTTP status: 200
  4. Keyword check: ingestion_ok":true
  5. Check interval: 2 minutes
  6. Click Save.

Step 10: Configure Alert Channels

Set alert routing so each failure class reaches the right responder:

Driver crash (P0 — page immediately):

  • Monitor: Driver REST API HTTP check
  • Alert condition: down for 1 check
  • Channel: PagerDuty or SMS
  • Reason: driver crash stops all running Nemo jobs immediately

Executor fleet reduction (P1):

  • Monitor: TCP port checks on executor hosts
  • Alert condition: any host unreachable for 2 consecutive checks
  • Channel: Slack #data-engineering
  • Reason: executor loss causes task re-execution and increased job latency

Job submission heartbeat missing (P1):

  • Monitor: Cron heartbeat
  • Alert condition: heartbeat missed by 1 interval
  • Channel: Slack #data-engineering
  • Reason: optimization pass failure or driver unreachability blocking the pipeline

Job completion SLA breach (P2):

  • Monitor: Job duration HTTP check
  • Alert condition: keyword FAILED present in response
  • Channel: Email to data team
  • Reason: failed jobs require investigation and replay

Web UI down (P3):

  • Monitor: Web UI HTTP check
  • Alert condition: down for 2 consecutive checks
  • Channel: Email
  • Reason: processing continues but operator visibility is lost

Summary

| What to monitor | Monitor type | Check interval | Alert condition | |---|---|---|---| | Nemo driver REST API | HTTP | 1 min | Down for 1 check | | Nemo web UI | HTTP | 2 min | Down for 2 checks | | Executor TCP connectivity | TCP Port | 1 min | Unreachable for 2 checks | | Job submission pipeline | Cron Heartbeat | Per job cadence | Heartbeat missed | | Optimization pass failures | HTTP / keyword | 5 min | Failure count > 0 | | Data skew unmitigated | HTTP / keyword | 5 min | Unmitigated skew > 0 | | Job completion status | HTTP / keyword | 2 min | FAILED in response | | Beam pipeline submission | Cron Heartbeat | 30 min | Heartbeat missed | | Metric ingestion health | HTTP / keyword | 2 min | ingestion_ok false |

Apache Nemo's strength — runtime portability via a single IR — is also its monitoring complexity: a failure can occur at the driver, the optimization layer, the target runtime, or the executor level. Vigilmon gives you visibility at every layer with one dashboard and one alert configuration, so your data pipelines stay observable even as they execute across multiple backends.

Get started with Vigilmon free →

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles