tutorial

Monitoring Infinispan with Vigilmon

Infinispan is Red Hat's distributed in-memory data grid — here's how to monitor cache hit ratios, cluster topology, Hot Rod connectivity, eviction pressure, cross-site replication lag, and JVM heap with Vigilmon.

Infinispan is an open source, distributed in-memory data grid and caching platform developed by Red Hat — the technology behind JBoss Data Grid and deeply integrated into WildFly and Quarkus for distributed caching. Created in 2009 as an evolution of JBoss Cache, Infinispan supports multiple operating modes (embedded within the application JVM, or as a standalone Infinispan Server), multiple cache topologies (local, replicated, distributed, invalidation), and multiple access protocols (Hot Rod binary protocol, REST API, and Memcached). When you run Infinispan in production, the health of your cache directly impacts application performance and availability: a degraded hit ratio forces expensive database fallbacks, a partitioned cluster splits your data across disconnected nodes, and memory pressure triggers eviction that invalidates your caching strategy. Vigilmon gives you continuous monitoring across every critical dimension of your Infinispan deployment.

What You'll Set Up

  • Infinispan Server REST API health endpoint monitor
  • Cache hit ratio monitoring via heartbeat
  • Cache entry count and eviction rate monitoring
  • Cluster topology and JGroups member count health check
  • Hot Rod client connectivity monitoring
  • Data rebalancing state monitoring
  • Persistence store (RocksDB/JDBC) health check
  • Cross-site replication lag monitoring
  • Eviction and off-heap memory pressure alert
  • JVM heap usage monitoring

Prerequisites

  • Infinispan Server 14+ running in standalone or clustered mode, OR Infinispan embedded in WildFly/Quarkus
  • REST API accessible on port 11222 (default for Infinispan Server)
  • A free Vigilmon account

Step 1: Monitor Infinispan Server Health

Infinispan Server exposes a REST health endpoint that reports overall server and cache container health. This is your primary liveness check — if the REST API is unreachable, you've lost both metrics access and the data grid itself.

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. URL: http://your-server:11222/rest/v2/cache-managers/default/health.
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Under Keyword check, enter HEALTHY to verify the cache manager is in healthy state.
  7. Click Save.

The health endpoint returns a JSON response like:

{
  "cluster_health": {
    "cluster_name": "infinispan",
    "health_status": "HEALTHY",
    "number_of_nodes": 3,
    "node_names": ["node1", "node2", "node3"]
  },
  "cache_health": [
    {"cache_name": "sessions", "status": "HEALTHY"}
  ]
}

Set up a secondary TCP check for the Hot Rod port:

  1. Click Add Monitor → TCP Port.
  2. Host: your Infinispan Server hostname.
  3. Port: 11222 (REST + Hot Rod over TLS, default) or 11221 (plain Hot Rod).
  4. Check interval: 1 minute.
  5. Click Save.

Step 2: Monitor Cache Hit Ratio

The cache hit ratio — the fraction of read requests served from memory versus misses that require fetching from the origin data store — is the most important measure of cache effectiveness. A hit ratio below 0.5 (50%) means more than half of all cache reads are going to your database, negating the performance benefit of caching.

#!/bin/bash
CACHE="sessions"  # Replace with your cache name
BASE="http://localhost:11222/rest/v2/caches/$CACHE"

STATS=$(curl -s "$BASE?action=stats" -H 'Accept: application/json')

HITS=$(echo "$STATS" | jq -r '.hits // 0')
MISSES=$(echo "$STATS" | jq -r '.misses // 0')
TOTAL=$(( HITS + MISSES ))

if [ "$TOTAL" -gt 0 ]; then
  HIT_PCT=$(( HITS * 100 / TOTAL ))
  # Ping heartbeat when hit ratio is above 50%
  if [ "$HIT_PCT" -ge 50 ]; then
    curl -s "https://vigilmon.online/heartbeat/YOUR_HITRATIO_HEARTBEAT_ID" > /dev/null
  fi
else
  # No traffic yet — ping so the monitor doesn't alert during startup
  curl -s "https://vigilmon.online/heartbeat/YOUR_HITRATIO_HEARTBEAT_ID" > /dev/null
fi
  1. Click Add Monitor → Cron Heartbeat.
  2. Set expected interval to 5 minutes.
  3. Add to cron: */5 * * * * /opt/scripts/infinispan-hitratio.sh.
  4. A missed ping means the hit ratio has dropped below 50% — investigate whether your cache was invalidated, the cluster rebalanced, or your working set grew beyond capacity.

Step 3: Monitor Cache Entry Count and Capacity

Infinispan caches have configurable entry count limits. When the cache approaches its maximum capacity, Infinispan begins evicting older entries, which reduces the effective hit ratio. Monitoring entry count and eviction rate tells you whether your cache sizing is adequate for your workload.

#!/bin/bash
CACHE="sessions"
BASE="http://localhost:11222/rest/v2/caches/$CACHE"

STATS=$(curl -s "$BASE?action=stats" -H 'Accept: application/json')

ENTRIES=$(echo "$STATS" | jq -r '.current_number_of_entries // 0')
EVICTIONS=$(echo "$STATS" | jq -r '.evictions // 0')

# Fetch configured max entries
CONFIG=$(curl -s "$BASE?action=config" -H 'Accept: application/json')
MAX_ENTRIES=$(echo "$CONFIG" | jq -r '.memory.max_count // 0')

echo "Entries: $ENTRIES / $MAX_ENTRIES, Evictions total: $EVICTIONS"

# Alert if over 90% of max capacity
if [ "$MAX_ENTRIES" -gt 0 ] && [ $(( ENTRIES * 100 / MAX_ENTRIES )) -lt 90 ]; then
  curl -s "https://vigilmon.online/heartbeat/YOUR_CAPACITY_HEARTBEAT_ID" > /dev/null
fi

Create a Cron Heartbeat with 5-minute interval. A missed ping indicates the cache is at or above 90% of its configured max — entries are being aggressively evicted, which will reduce hit ratio and increase database load.

Step 4: Monitor Cluster Topology Health

Infinispan distributes data across cluster nodes using JGroups for cluster membership. A split-brain or node loss event can cause data loss (in non-replicated modes), increased rebalancing load, and incorrect routing of requests. Monitoring the cluster member count gives you early warning of node failures.

#!/bin/bash
BASE="http://localhost:11222/rest/v2/cache-managers/default/health"

HEALTH=$(curl -s "$BASE" -H 'Accept: application/json')

NODE_COUNT=$(echo "$HEALTH" | jq -r '.cluster_health.number_of_nodes // 0')
CLUSTER_STATUS=$(echo "$HEALTH" | jq -r '.cluster_health.health_status // "UNKNOWN"')

EXPECTED_NODES=3  # Set to your expected cluster size

if [ "$CLUSTER_STATUS" = "HEALTHY" ] && [ "$NODE_COUNT" -ge "$EXPECTED_NODES" ]; then
  curl -s "https://vigilmon.online/heartbeat/YOUR_CLUSTER_HEARTBEAT_ID" > /dev/null
fi
  1. Create a Cron Heartbeat with 2-minute interval.
  2. Set EXPECTED_NODES to your minimum viable cluster size (e.g., for a 3-node cluster where you can tolerate 1 failure, set to 2).
  3. A missed ping fires when the cluster reports DEGRADED, FAILED, or has fewer members than expected — catching node failures before they exhaust replication capacity.

Step 5: Monitor Hot Rod Client Connectivity

Hot Rod is Infinispan's native binary protocol for high-performance client-server access. Clients in your application tier connect via Hot Rod and maintain a persistent connection pool. A drop in active Hot Rod connections indicates client-side failures or network partitions between your application and Infinispan.

  1. Click Add Monitor → TCP Port.
  2. Host: your Infinispan Server hostname.
  3. Port: 11222 (unified port for REST and Hot Rod over TLS, default) or 11221 (legacy plain Hot Rod).
  4. Check interval: 1 minute.
  5. Click Save.

For connection count monitoring from the server side:

#!/bin/bash
BASE="http://localhost:11222/rest/v2/server/connections"

CONN=$(curl -s "$BASE" -H 'Accept: application/json')
ACTIVE=$(echo "$CONN" | jq -r '.active_connections // 0')

echo "Active Hot Rod connections: $ACTIVE"

# Alert if active connections drop below your expected minimum
EXPECTED_MIN=5  # Adjust based on your application's connection pool size
if [ "$ACTIVE" -ge "$EXPECTED_MIN" ]; then
  curl -s "https://vigilmon.online/heartbeat/YOUR_HOTROD_HEARTBEAT_ID" > /dev/null
fi

Set heartbeat interval to 2 minutes. A drop below the minimum expected Hot Rod connections means your application clients have disconnected or failed to reconnect after a cluster event.

Step 6: Monitor Data Distribution and Rebalancing

Infinispan distributes data segments across nodes in distributed cache mode. When a node joins or leaves the cluster, Infinispan triggers a rebalance operation to redistribute segments. A prolonged rebalance (typically more than a few minutes for normal-sized caches) indicates cluster instability or an overloaded node struggling to transfer data.

#!/bin/bash
BASE="http://localhost:11222/rest/v2/cache-managers/default"

HEALTH=$(curl -s "$BASE/health" -H 'Accept: application/json')

# Check if any cache is in REBALANCING state
REBALANCING=$(echo "$HEALTH" | jq -r \
  '[.cache_health[] | select(.status == "REBALANCING")] | length')

if [ "$REBALANCING" -eq 0 ]; then
  curl -s "https://vigilmon.online/heartbeat/YOUR_REBALANCE_HEARTBEAT_ID" > /dev/null
fi
  1. Create a Cron Heartbeat with 5-minute interval.
  2. A brief rebalance after a node join/leave is normal; the alert fires when rebalancing has been ongoing for more than one heartbeat interval (5+ minutes), signaling that the rebalance is stuck or taking unusually long.

Step 7: Monitor Infinispan Persistence Store Health

If you've configured Infinispan with a persistence store (RocksDB, JDBC, or file-based), cache entries survive server restarts and memory pressure by writing to disk. Persistence store failures cause write operations to fail silently or fall back to memory-only, risking data loss on restart.

#!/bin/bash
CACHE="persistent-cache"  # Replace with your cache name
BASE="http://localhost:11222/rest/v2/caches/$CACHE"

STATS=$(curl -s "$BASE?action=stats" -H 'Accept: application/json')

STORE_LOADS=$(echo "$STATS" | jq -r '.stores // 0')
STORE_MISS=$(echo "$STATS" | jq -r '.store_load_failures // 0')

# Alert if store load failure rate is non-zero
if [ "${STORE_MISS:-0}" -eq 0 ]; then
  curl -s "https://vigilmon.online/heartbeat/YOUR_STORE_HEARTBEAT_ID" > /dev/null
fi

For RocksDB stores, also monitor disk space on the persistence directory:

#!/bin/bash
STORE_DIR="/opt/infinispan/data/persistence"  # Adjust to your store path

DISK_FREE=$(df -P "$STORE_DIR" | awk 'NR==2 {print $5}' | tr -d '%')

if [ "$DISK_FREE" -lt 85 ]; then
  curl -s "https://vigilmon.online/heartbeat/YOUR_DISK_HEARTBEAT_ID" > /dev/null
fi

Step 8: Monitor Cross-Site Replication Lag

If you're using Infinispan cross-datacenter replication (RELAY2/JGroups-based or remote site replication), replication lag accumulates when the secondary site can't keep up with writes from the primary. Sustained lag risks the secondary site becoming stale, undermining your disaster recovery capability.

#!/bin/bash
CACHE="replicated-cache"  # Replace with your cross-site replicated cache
REMOTE_SITE="LON"  # Replace with your remote site name
BASE="http://localhost:11222/rest/v2/caches/$CACHE/x-site/backups/$REMOTE_SITE"

XSITE=$(curl -s "$BASE" -H 'Accept: application/json')
STATUS=$(echo "$XSITE" | jq -r '.status // "unknown"')
SEND_QUEUE=$(echo "$XSITE" | jq -r '.send_queue_size // 0')

# Alert if replication is broken or queue is growing
if [ "$STATUS" = "online" ] && [ "$SEND_QUEUE" -lt 1000 ]; then
  curl -s "https://vigilmon.online/heartbeat/YOUR_XSITE_HEARTBEAT_ID" > /dev/null
fi
  1. Create a Cron Heartbeat with 5-minute interval.
  2. Alert fires when cross-site status is not online or the send queue exceeds 1000 entries — both indicate replication is falling behind.

Step 9: Monitor Eviction and Memory Pressure

Infinispan evicts cache entries when memory is full (or when the configured entry count limit is reached). A spike in eviction rate means your working set has grown beyond cache capacity, and hit ratio will soon fall. Off-heap memory usage (when configured with -XX:MaxDirectMemorySize) can also exhaust without triggering standard JVM heap alerts.

#!/bin/bash
CACHE="sessions"
BASE="http://localhost:11222/rest/v2/caches/$CACHE"

STATS=$(curl -s "$BASE?action=stats" -H 'Accept: application/json')

EVICTIONS=$(echo "$STATS" | jq -r '.evictions // 0')

# Track eviction delta between runs (store last value)
STATE_FILE="/tmp/infinispan-evictions-last"
LAST=$(cat "$STATE_FILE" 2>/dev/null || echo 0)
DELTA=$(( EVICTIONS - LAST ))
echo "$EVICTIONS" > "$STATE_FILE"

# Alert if more than 100 evictions occurred in the last minute
if [ "$DELTA" -lt 100 ]; then
  curl -s "https://vigilmon.online/heartbeat/YOUR_EVICTION_HEARTBEAT_ID" > /dev/null
fi

Create a Cron Heartbeat with 1-minute interval, cron schedule * * * * *. A missed ping means eviction rate exceeded your threshold — cache is under memory pressure and hit ratio will degrade.

Step 10: Monitor JVM Heap Health

Whether running Infinispan embedded in your application JVM or as a standalone Infinispan Server process, JVM heap health is fundamental. Heap exhaustion causes OutOfMemoryError and crashes the node, triggering a cluster rebalance that adds load to the remaining members.

#!/bin/bash
BASE="http://localhost:11222/rest/v2/server/memory"

MEM=$(curl -s "$BASE" -H 'Accept: application/json')

HEAP_USED=$(echo "$MEM" | jq -r '.heap_used // 0')
HEAP_MAX=$(echo "$MEM" | jq -r '.heap_max // 1')

HEAP_PCT=$(( HEAP_USED * 100 / HEAP_MAX ))

if [ "$HEAP_PCT" -lt 85 ]; then
  curl -s "https://vigilmon.online/heartbeat/YOUR_JVM_HEARTBEAT_ID" > /dev/null
fi
  1. Create a Cron Heartbeat with 2-minute interval.
  2. Alert when heap exceeds 85% — above this threshold, GC pressure becomes severe and the risk of OutOfMemoryError and node crash rises sharply.
  3. If using off-heap memory (eviction.type=MEMORY), also monitor direct memory via JMX java.nio:type=BufferPool,name=direct.

Alerting Configuration

Configure alert channels in Vigilmon to notify your team when cache health degrades:

  1. Go to Alert Channels in Vigilmon.
  2. Add Email as the baseline for all monitors.
  3. Add Slack or PagerDuty for cluster topology and REST API monitors (highest severity).
  4. Set 2 consecutive failures before alerting to avoid noise from momentary check failures.

Recommended alert thresholds:

| Monitor | Alert Condition | |---|---| | REST API health endpoint | Not HEALTHY or unreachable for 2 min | | Cache hit ratio | Below 50% | | Cluster node count | Below minimum expected | | Cluster status | DEGRADED or FAILED | | JVM heap | > 85% for 4 minutes | | Eviction rate | > 100 evictions/minute | | Cross-site status | Not online | | Persistence store failures | Any store_load_failures > 0 |

Conclusion

Infinispan's REST API provides rich per-cache and per-cluster metrics that map directly to the operational questions that matter most: Is my cache actually serving data from memory? Is my cluster fully connected? Is my working set exceeding cache capacity? Is my secondary site keeping up with writes? By routing these signals into Vigilmon heartbeats and combining them with direct TCP checks on the Hot Rod port and REST API health endpoint, you get comprehensive observability across the full Infinispan stack. Set these monitors up before a cache degradation event, and you'll catch hit ratio drops, cluster splits, and memory pressure before they translate into database overload or application performance regressions.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles