tutorial

Monitoring Hazelcast Jet Stream Processing with Vigilmon

Hazelcast Jet brings distributed stream and batch processing to the Hazelcast IMDG — here's how to monitor cluster health, Jet job status, distributed snapshots, IMap state, Kafka lag, and GC pauses with Vigilmon.

Hazelcast Jet is a distributed stream and batch processing engine embedded in the Hazelcast Platform. Since Hazelcast Platform 5.0, Jet is the stream processing layer of a unified in-memory computing platform — the same JVM cluster that serves your distributed caches and IMaps also runs your streaming pipelines. Jet processes data as DAGs of vertices, provides exactly-once guarantees via distributed snapshotting, and uses Hazelcast IMap for blazing-fast in-memory state. But when a cluster member leaves, a snapshot fails, or a Jet job enters a failure loop, you need to know before it cascades into downstream data loss. Vigilmon gives you the HTTP-based health checks, heartbeats, and alerting to cover every critical dimension of your Hazelcast Platform / Jet deployment.

What You'll Set Up

  • Hazelcast cluster member count monitoring
  • Jet job status health checks
  • Distributed snapshot success rate alerting
  • IMap memory usage monitoring
  • Kafka source consumer group lag alerts
  • Processing throughput tracking
  • JVM heap and GC pause monitoring
  • Split-brain quorum protection alerts
  • Hazelcast Management Center / REST API health
  • Jet job restart frequency tracking

Prerequisites

  • Hazelcast Platform 5.0+ running as a JVM cluster (bare metal, Docker, or Kubernetes)
  • Hazelcast REST API enabled (hazelcast.rest.enabled: true in your config)
  • Hazelcast Management Center (optional but recommended)
  • A free Vigilmon account

Why Monitor Hazelcast Jet?

Hazelcast Platform's distributed nature means that failures are often partial and silent. A single member leaving the cluster degrades processing throughput without stopping the pipeline — until the cluster drops below quorum and Jet suspends all jobs. Distributed snapshots are your exactly-once guarantee; if they start failing, you are silently downgrading to at-least-once. IMap state that fills up the JVM heap causes GC pressure that cascades into processing stalls and then member crashes. And a Jet job that enters a failure-restart loop drains cluster resources and may never recover without operator intervention. Proactive Vigilmon monitoring surfaces all of these before they become incidents.

Step 1: Monitor Hazelcast Cluster Member Count

Hazelcast's REST API exposes cluster state including member count. Enable the REST API in your hazelcast.yaml:

hazelcast:
  rest-api:
    enabled: true
    endpoint-groups:
      CLUSTER_READ:
        enabled: true
      CLUSTER_WRITE:
        enabled: true
      HEALTH_CHECK:
        enabled: true

The health check endpoint is available at:

http://<member-host>:5701/hazelcast/health

This returns a simple 200 OK with body Hazelcast::NodeState=ACTIVE when the member is healthy. Add a Vigilmon monitor:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter http://<member-host>:5701/hazelcast/health.
  3. Set Expected HTTP status to 200.
  4. Optionally set Expected body contains to ACTIVE.
  5. Set Check interval to 1 minute.
  6. Click Save.

Add one monitor per member. For the cluster member count, query the cluster state endpoint:

http://<member-host>:5701/hazelcast/rest/cluster

This returns JSON with {"members":["<uuid>","<uuid>"],"state":"ACTIVE"}. Write a small proxy that parses this and returns 503 when members.length < MIN_MEMBERS:

// Spring Boot probe example
@RestController
public class ClusterProbe {
    @Value("${hazelcast.rest.url}")
    private String hazelcastUrl;

    @Value("${cluster.min.members:3}")
    private int minMembers;

    @GetMapping("/probe/cluster")
    public ResponseEntity<Map<String,Object>> clusterHealth() throws Exception {
        String json = restTemplate.getForObject(hazelcastUrl + "/rest/cluster", String.class);
        JSONObject obj = new JSONObject(json);
        int count = obj.getJSONArray("members").length();
        String state = obj.getString("state");
        boolean healthy = count >= minMembers && "ACTIVE".equals(state);
        return ResponseEntity.status(healthy ? 200 : 503)
            .body(Map.of("member_count", count, "state", state, "min_members", minMembers));
    }
}

Monitor http://<probe-host>:8080/probe/cluster in Vigilmon. Alert on 503 immediately — member loss directly impacts processing capacity.

Step 2: Monitor Jet Job Health

Jet jobs run as long-lived distributed computations. A failed job stops processing; a suspended job stops accumulating results. Poll job status via the Hazelcast REST API:

GET http://<member>:5701/hazelcast/rest/jets/jobs

This returns a list of jobs with their statuses: RUNNING, SUSPENDED, FAILED, COMPLETED. Write a probe that returns 503 if any expected job is not RUNNING:

@GetMapping("/probe/jet-jobs")
public ResponseEntity<Map<String,Object>> jetJobHealth() throws Exception {
    String json = restTemplate.getForObject(hazelcastUrl + "/rest/jets/jobs", String.class);
    JSONArray jobs = new JSONArray(json);
    List<Map<String,Object>> failed = new ArrayList<>();
    for (int i = 0; i < jobs.length(); i++) {
        JSONObject job = jobs.getJSONObject(i);
        String status = job.getString("status");
        if (!"RUNNING".equals(status) && !"COMPLETED".equals(status)) {
            failed.add(Map.of("name", job.getString("name"), "status", status));
        }
    }
    boolean healthy = failed.isEmpty();
    return ResponseEntity.status(healthy ? 200 : 503)
        .body(Map.of("failed_jobs", failed, "total_jobs", jobs.length()));
}

Add a Vigilmon monitor at /probe/jet-jobs with a 1-minute interval. A FAILED job is an immediate alert; a SUSPENDED job warrants investigation.

Step 3: Monitor Distributed Snapshot Health

Hazelcast Jet uses distributed snapshotting (similar to Flink checkpointing) to provide exactly-once and at-least-once guarantees. Failed snapshots mean state is not persisted and a crash would cause reprocessing from the last successful snapshot.

Monitor snapshot events via the Management Center REST API or JMX. If you have Management Center:

GET http://<mc-host>:8080/api/rest/clusters/<cluster-name>/jet/snapshot-stats

For a lightweight alternative, expose a probe that checks the last snapshot timestamp per job:

// Use HazelcastInstance.getJet().getJob(jobId).getSuspensionCause() and
// metrics APIs to detect snapshot failures
@GetMapping("/probe/snapshots")
public ResponseEntity<Map<String,Object>> snapshotHealth() {
    // Query Hazelcast metrics for: jet.snapshotting.completed.count
    // and jet.snapshotting.failed.count via the /metrics endpoint
    long failed = getMetricValue("jet.snapshotting.failed.count");
    long completed = getMetricValue("jet.snapshotting.completed.count");
    boolean healthy = failed == 0;
    return ResponseEntity.status(healthy ? 200 : 503)
        .body(Map.of("snapshots_failed", failed, "snapshots_completed", completed));
}

Hazelcast exposes Prometheus-compatible metrics at:

http://<member>:8080/metrics

(requires hazelcast.metrics.prometheus.enabled: true). Add a Vigilmon monitor. Alert on any 503 — snapshot failures silently degrade your delivery guarantee.

Step 4: Monitor IMap State Memory Usage

Jet uses Hazelcast IMap for stateful processing. When IMap memory usage exceeds cluster heap, members start evicting entries or crashing with OutOfMemoryError.

Use the REST API to check IMap statistics:

GET http://<member>:5701/hazelcast/rest/maps/<map-name>/stats

Write a probe that computes used heap percentage:

@GetMapping("/probe/imap-memory")
public ResponseEntity<Map<String,Object>> imapMemoryHealth() throws Exception {
    // Get total heap across all members
    String clusterJson = restTemplate.getForObject(hazelcastUrl + "/rest/cluster", String.class);
    // Get IMap heap cost
    String mapJson = restTemplate.getForObject(hazelcastUrl + "/rest/maps/jet-state/stats", String.class);
    JSONObject stats = new JSONObject(mapJson);
    long heapCost = stats.getLong("heapCost");
    long maxHeap = Runtime.getRuntime().maxMemory();
    double pct = (double) heapCost / maxHeap * 100;
    boolean healthy = pct < 80.0;
    return ResponseEntity.status(healthy ? 200 : 503)
        .body(Map.of("heap_used_pct", pct, "heap_cost_bytes", heapCost));
}

Alert on 503 (heap > 80%) — IMap memory pressure cascades quickly into GC issues and member crashes.

Step 5: Monitor Kafka Source Consumer Group Lag

Jet's Kafka source connector maintains a consumer group. Growing lag means Jet is not keeping up with the Kafka topic's message rate.

Expose a Kafka lag endpoint alongside your Jet cluster (see the pattern from the Bytewax tutorial). Return 200 when lag is below threshold, 503 when it exceeds it:

@GetMapping("/probe/kafka-lag")
public ResponseEntity<Map<String,Object>> kafkaLagHealth() {
    long lag = kafkaLagService.getTotalLag("jet-consumer-group");
    long threshold = 10000L;
    boolean healthy = lag < threshold;
    return ResponseEntity.status(healthy ? 200 : 503)
        .body(Map.of("total_lag", lag, "threshold", threshold));
}

Monitor at /probe/kafka-lag with a 1-minute interval.

Step 6: Monitor Processing Throughput

Hazelcast Jet exposes a jet.items.out metric per vertex. Query the Prometheus metrics endpoint for throughput:

http://<member>:8080/metrics

Look for jet_items_out_total and compute a rate. Write a probe that compares the current rate against a baseline:

@GetMapping("/probe/throughput")
public ResponseEntity<Map<String,Object>> throughputHealth() {
    double itemsPerSec = metricsService.getJetItemsOutRate();
    double baselineItemsPerSec = 50_000; // set from historical average
    double threshold = baselineItemsPerSec * 0.7;
    boolean healthy = itemsPerSec >= threshold;
    return ResponseEntity.status(healthy ? 200 : 503)
        .body(Map.of("items_per_sec", itemsPerSec, "threshold", threshold));
}

Alert on 503 — a sudden throughput drop usually indicates a member leaving, a job being suspended, or Kafka lag building up.

Step 7: Monitor JVM Heap and GC Pause Duration

Hazelcast Platform runs on the JVM. Long GC pauses (>5 seconds) cause heartbeat timeouts between members and can trigger false split-brain detection. Expose JVM health via a Spring Boot Actuator endpoint:

management:
  endpoints:
    web:
      exposure:
        include: health,metrics
  metrics:
    export:
      prometheus:
        enabled: true

Or write a custom probe:

@GetMapping("/probe/jvm")
public ResponseEntity<Map<String,Object>> jvmHealth() {
    MemoryMXBean mem = ManagementFactory.getMemoryMXBean();
    long used = mem.getHeapMemoryUsage().getUsed();
    long max = mem.getHeapMemoryUsage().getMax();
    double pct = (double) used / max * 100;

    long maxGcPauseMs = getMaxRecentGcPauseMs(); // from GarbageCollectorMXBean
    boolean healthy = pct < 85 && maxGcPauseMs < 5000;
    return ResponseEntity.status(healthy ? 200 : 503)
        .body(Map.of("heap_used_pct", pct, "max_gc_pause_ms", maxGcPauseMs));
}

private long getMaxRecentGcPauseMs() {
    return ManagementFactory.getGarbageCollectorMXBeans().stream()
        .mapToLong(gc -> gc.getCollectionTime())
        .max()
        .orElse(0L);
}

Monitor at /probe/jvm with a 1-minute interval. Alert on 503 — a GC pause >5 seconds can cascade into cluster instability.

Step 8: Monitor Split-Brain Protection

Hazelcast uses quorum (split-brain protection) rules to prevent cluster partitions from operating independently. When a partition happens, Hazelcast freezes the minority partition.

Monitor the cluster state via the REST API:

GET http://<member>:5701/hazelcast/rest/cluster

Parse the state field: ACTIVE is normal; FROZEN or PASSIVE indicates split-brain protection has activated. Write a probe:

@GetMapping("/probe/split-brain")
public ResponseEntity<Map<String,Object>> splitBrainHealth() throws Exception {
    String json = restTemplate.getForObject(hazelcastUrl + "/rest/cluster", String.class);
    JSONObject obj = new JSONObject(json);
    String state = obj.getString("state");
    boolean healthy = "ACTIVE".equals(state);
    return ResponseEntity.status(healthy ? 200 : 503)
        .body(Map.of("cluster_state", state));
}

Alert on 503 immediately — a frozen cluster stops processing all Jet jobs.

Step 9: Monitor Hazelcast Management Center Health

Management Center is the primary operational interface for Hazelcast Platform. If it goes down, your team loses visibility into cluster health, job status, and IMap statistics.

Management Center exposes a health endpoint:

GET http://<mc-host>:8080/health

Add a Vigilmon monitor:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter http://<mc-host>:8080/health.
  3. Set Expected HTTP status to 200.
  4. Set Check interval to 1 minute.
  5. Click Save.

Also monitor the Hazelcast REST API directly to catch cases where Management Center is up but has lost cluster connectivity:

GET http://<member>:5701/hazelcast/health

Step 10: Track Jet Job Restart Frequency

A Jet job that restarts more than 3 times in one hour is in a failure loop. Instrument job restart events and expose a restart rate endpoint:

// Implement a JobStateListener (Hazelcast Jet API)
public class RestartTracker implements JobStatusListener {
    private final Deque<Long> restartTimestamps = new ConcurrentLinkedDeque<>();

    @Override
    public void jobStatusChanged(JobStatusEvent event) {
        if (event.getNewStatus() == JobStatus.RUNNING && event.getPreviousStatus() == JobStatus.STARTING) {
            long now = System.currentTimeMillis();
            restartTimestamps.addLast(now);
            // evict entries older than 1 hour
            while (!restartTimestamps.isEmpty() && restartTimestamps.peekFirst() < now - 3_600_000) {
                restartTimestamps.pollFirst();
            }
        }
    }

    public int restartsInLastHour() {
        return restartTimestamps.size();
    }
}

Expose a probe returning 503 when restartsInLastHour() > 3:

@GetMapping("/probe/job-restarts")
public ResponseEntity<Map<String,Object>> jobRestartHealth() {
    int restarts = restartTracker.restartsInLastHour();
    boolean healthy = restarts <= 3;
    return ResponseEntity.status(healthy ? 200 : 503)
        .body(Map.of("restarts_last_hour", restarts, "threshold", 3));
}

Monitor at /probe/job-restarts. A job in a restart loop needs manual intervention to resolve the root cause.

Alerting Configuration

| Monitor | Condition | Action | |---------|-----------|--------| | Cluster member health | Any 503 or non-ACTIVE state | Page on-call immediately | | Jet job status | Any 503 (job not RUNNING) | Page on-call immediately | | Snapshot health | Any 503 (snapshot failure) | Page on-call immediately | | IMap memory | 503 (heap > 80%) | Notify platform team | | Kafka source lag | 503 (lag > threshold) | Notify data engineering team | | Processing throughput | 503 (dropped > 30%) | Notify data engineering team | | JVM / GC health | 503 (GC > 5s or heap > 85%) | Page on-call immediately | | Split-brain | 503 (cluster not ACTIVE) | Page on-call immediately | | Management Center | Any 503 | Notify platform team | | Job restart rate | 503 (>3 restarts/hour) | Notify data engineering team |

Route alerts to PagerDuty, Slack, or email using Vigilmon's Notification Channels.

Conclusion

Hazelcast Jet's in-memory, JVM-based architecture is extremely fast — but it means failures are often abrupt and silent. A member crash, a snapshot failure, or a GC pause can cascade through the cluster in seconds. Vigilmon's HTTP health checks, applied to a lightweight probe sidecar alongside your Hazelcast members, give you real-time visibility into the ten critical health dimensions: cluster member count, job status, snapshot integrity, IMap memory pressure, Kafka source lag, throughput, JVM health, split-brain protection, Management Center reachability, and job restart loops. With the right alerts wired up, you catch degradation in seconds rather than minutes.

Get started free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles