tutorial

Monitoring Apache Ozone with Vigilmon

Apache Ozone replaces HDFS for cloud-native big data — but OM leader loss, under-replicated containers, and SCM failures are invisible without external monitoring. Here's how to monitor Ozone end-to-end with Vigilmon.

Apache Ozone (top-level Apache project, 2022) is the next-generation distributed object store for the Hadoop ecosystem, designed to solve HDFS's most fundamental limitations at scale. Where HDFS uses a block-based model with a centralized NameNode that becomes a memory bottleneck at billions of files, Ozone uses an object/key-value model with independently scalable metadata components: the Ozone Manager (OM) handles namespace (buckets and keys), the Storage Container Manager (SCM) handles container placement and DataNode cluster membership, and DataNodes store data in fixed-size containers (5GB units). Ozone supports native S3 API compatibility (allowing existing S3-compatible tools to work against Ozone without modification), an HDFS-compatible filesystem interface (for Hadoop ecosystem compatibility), and is optimized for hundreds of billions of small files — a workload that cripples HDFS. But Ozone's multi-component architecture means there are more failure surfaces to monitor: OM leader elections, SCM outages, DataNode stale counts, under-replicated containers, and S3 gateway crashes all require active monitoring. Vigilmon provides the external visibility that keeps your Ozone cluster operational.

What You'll Set Up

  • HTTP probes for Ozone Manager and Storage Container Manager health
  • DataNode count and stale DataNode monitoring via heartbeats
  • Container replication health monitoring
  • S3 gateway health and response time monitoring
  • Namespace operation success rate monitoring
  • ReconServer health monitoring
  • Pipeline health and certificate expiry monitoring

Prerequisites

  • Apache Ozone 1.3+ with Ozone Manager (OM), Storage Container Manager (SCM), and at least 3 DataNodes
  • Ozone Manager configured in HA mode (3-node Raft) for production
  • Ozone S3 gateway deployed if using S3-compatible clients
  • ReconServer deployed for cluster analytics
  • A free Vigilmon account

Why Monitoring Ozone Matters

Ozone's architecture distributes its responsibilities across several components, each with independent failure modes:

  • Ozone Manager (OM) leader loss — OM manages the namespace (all bucket and key metadata). If the OM loses its Raft leader — due to a network partition, node failure, or JVM crash — all namespace operations (key creation, reads, deletes) block until a new leader is elected. Clients experience timeouts, and Spark or Hive jobs fail at file I/O with no immediate indication that the cause is Ozone leadership rather than network connectivity.
  • Storage Container Manager (SCM) failure — SCM manages container placement (which DataNodes hold which containers), DataNode heartbeats, and replication decisions. An SCM failure halts container creation for new data and stops replication repair for under-replicated containers. Existing reads succeed, but new writes fail and replication degradation accumulates.
  • DataNode count below safe threshold — When enough DataNodes go stale (missed SCM heartbeats for too long), SCM marks them as DEAD and begins replicating their containers to surviving nodes. If more than 20% of DataNodes become stale simultaneously (e.g., after a rack-level network failure), container availability falls below the replication factor and read failures occur.
  • Under-replicated containers — Ozone stores data in 5GB containers. If a container's replication falls below the configured minimum (typically 3 replicas for 3-way replication), it is at risk of data loss if another DataNode holding a copy fails before SCM completes re-replication.
  • S3 gateway crash — Applications using Ozone via the S3 API depend on the S3 gateway process. A crash causes all S3 API calls to return connection errors, silently breaking data pipelines that use tools like Hadoop, Spark (via s3a connector), or standalone S3 clients.

Step 1: Monitor the Ozone Manager

The Ozone Manager exposes an HTTP status endpoint that reports OM health and Raft leadership state:

  1. In Vigilmon, click Add Monitor → HTTP / HTTPS.
  2. Enter: http://ozone-om:9874/api/v1/service/status.
  3. Set Check interval to 1 minute.
  4. Set Expected HTTP status to 200.
  5. Set Response body contains to "status":"RUNNING".
  6. Click Save.

For HA OM deployments (3-node Raft), add a separate monitor that checks the leader election state:

#!/bin/bash
# /opt/monitoring/check-ozone-om-leader.sh
# Verify that exactly one OM is reporting as leader in the OM HA group

OM_NODES="${OZONE_OM_NODES:-ozone-om1:9874,ozone-om2:9874,ozone-om3:9874}"
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_OM_LEADER_KEY"

LEADER_COUNT=0
for NODE in $(echo "$OM_NODES" | tr ',' ' '); do
  ROLE=$(curl -sf "http://$NODE/api/v1/service/status" | \
    python3 -c "
import sys, json
data = json.load(sys.stdin)
print(data.get('nodeRole', ''))
" 2>/dev/null)
  if [ "$ROLE" = "LEADER" ]; then
    LEADER_COUNT=$((LEADER_COUNT + 1))
  fi
done

# Exactly one leader is healthy; 0 means split-brain or all down
if [ "$LEADER_COUNT" -eq 1 ]; then
  curl -fsS "$HEARTBEAT_URL"
fi

Schedule every 1 minute. Set the Vigilmon heartbeat interval to 3 minutes. No leader for 3 minutes means all namespace operations are timing out.

Also add a TCP port monitor for the OM client RPC port (used by Hadoop filesystem and Ozone clients):

  1. Click Add Monitor → TCP Port.
  2. Enter ozone-om:9862.
  3. Set Check interval to 1 minute.
  4. Click Save.

Step 2: Monitor the Storage Container Manager

The SCM manages container placement and DataNode membership. Add an HTTP monitor for SCM health:

  1. In Vigilmon, click Add Monitor → HTTP / HTTPS.
  2. Enter: http://ozone-scm:9876/api/v1/service/status.
  3. Set Check interval to 1 minute.
  4. Set Expected HTTP status to 200.
  5. Set Response body contains to "status":"RUNNING".
  6. Click Save.

For HA SCM deployments, add the same leader election heartbeat pattern as for OM, targeting SCM nodes instead:

#!/bin/bash
# /opt/monitoring/check-ozone-scm-leader.sh
SCM_NODES="${OZONE_SCM_NODES:-ozone-scm1:9876,ozone-scm2:9876,ozone-scm3:9876}"
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_SCM_LEADER_KEY"

LEADER_COUNT=0
for NODE in $(echo "$SCM_NODES" | tr ',' ' '); do
  ROLE=$(curl -sf "http://$NODE/api/v1/service/status" | \
    python3 -c "
import sys, json
data = json.load(sys.stdin)
print(data.get('nodeRole', ''))
" 2>/dev/null)
  if [ "$ROLE" = "LEADER" ]; then
    LEADER_COUNT=$((LEADER_COUNT + 1))
  fi
done

if [ "$LEADER_COUNT" -eq 1 ]; then
  curl -fsS "$HEARTBEAT_URL"
fi

Schedule every 1 minute. Set the heartbeat interval to 3 minutes.

Step 3: Monitor DataNode Health

DataNode count dropping below threshold is an early warning of replication degradation. Create a heartbeat that monitors healthy DataNode count:

#!/bin/bash
# /opt/monitoring/check-ozone-datanode-health.sh
MIN_HEALTHY_PERCENT=80   # Alert if more than 20% of DataNodes are stale/dead
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_DATANODE_KEY"

SCM_URL="http://ozone-scm:9876"

NODE_STATS=$(curl -sf "$SCM_URL/api/v1/nodes" | python3 -c "
import sys, json
data = json.load(sys.stdin)
nodes = data.get('datanodes', [])
total = len(nodes)
healthy = sum(1 for n in nodes if n.get('opState') == 'IN_SERVICE' and n.get('state') == 'HEALTHY')
print(f'{healthy}/{total}')
" 2>/dev/null)

HEALTHY=$(echo "$NODE_STATS" | cut -d/ -f1)
TOTAL=$(echo "$NODE_STATS" | cut -d/ -f2)

if [ -n "$HEALTHY" ] && [ "${TOTAL:-0}" -gt 0 ]; then
  PCT=$(python3 -c "print(int($HEALTHY / $TOTAL * 100))")
  if [ "${PCT:-0}" -ge "$MIN_HEALTHY_PERCENT" ]; then
    curl -fsS "$HEARTBEAT_URL"
  fi
fi

Run every 2 minutes. Set the heartbeat interval to 5 minutes. Also add individual DataNode TCP monitors for the largest or most critical nodes:

  1. Click Add Monitor → TCP Port.
  2. Enter ozone-datanode1:9858 (DataNode XCeiver port).
  3. Set Check interval to 2 minutes.
  4. Click Save.

Step 4: Monitor Container Replication Health

Under-replicated containers are at risk of data loss if another DataNode fails before SCM repairs replication. Missing containers indicate potential data loss:

#!/bin/bash
# /opt/monitoring/check-ozone-container-health.sh
MAX_UNDER_REPLICATED=0   # Zero tolerance for under-replicated containers in production
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_CONTAINER_KEY"

SCM_URL="http://ozone-scm:9876"

CONTAINER_STATS=$(curl -sf "$SCM_URL/api/v1/containers/summary" | python3 -c "
import sys, json
data = json.load(sys.stdin)
print(data.get('underReplicated', 0), data.get('missingContainers', 0))
" 2>/dev/null)

UNDER_REPLICATED=$(echo "$CONTAINER_STATS" | awk '{print $1}')
MISSING=$(echo "$CONTAINER_STATS" | awk '{print $2}')

if [ "${MISSING:-1}" -eq 0 ] && [ "${UNDER_REPLICATED:-99}" -le "$MAX_UNDER_REPLICATED" ]; then
  curl -fsS "$HEARTBEAT_URL"
fi

Run every 5 minutes. Set the heartbeat interval to 10 minutes. Missing containers warrant immediate PagerDuty alerting — they indicate potential data loss risk. Under-replicated containers with zero missing is a high-priority issue (risk of data loss on next DataNode failure) but not yet data loss.

For clusters with large numbers of containers, a brief pulse of under-replication (1–5 containers) during DataNode restarts is expected. Set MAX_UNDER_REPLICATED=5 to suppress transient alerts and alert only when sustained replication degradation exceeds 5 containers.

Step 5: Monitor the S3 Gateway

The Ozone S3 gateway exposes a standard S3 API. Any crash or degradation blocks all S3-compatible clients:

  1. In Vigilmon, click Add Monitor → HTTP / HTTPS.
  2. Enter: http://ozone-s3g:9878/ (or your S3 gateway URL).
  3. Set Check interval to 1 minute.
  4. Set Expected HTTP status to 403 (S3 gateway returns 403 for unauthenticated root requests — this is normal and indicates the gateway is running).
  5. Click Save.

For a deeper check that validates S3 API functionality, add a heartbeat that performs a lightweight S3 operation:

#!/bin/bash
# /opt/monitoring/check-ozone-s3.sh
S3_ENDPOINT="http://ozone-s3g:9878"
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_S3_KEY"

# List buckets via S3 API — requires valid credentials
RESPONSE=$(AWS_ACCESS_KEY_ID="${OZONE_S3_ACCESS_KEY}" \
           AWS_SECRET_ACCESS_KEY="${OZONE_S3_SECRET_KEY}" \
           aws s3 ls \
           --endpoint-url "$S3_ENDPOINT" \
           --no-verify-ssl \
           2>&1)

if echo "$RESPONSE" | grep -qv "Error\|Failed\|error"; then
  curl -fsS "$HEARTBEAT_URL"
fi

Schedule every 2 minutes. Set the heartbeat interval to 5 minutes. Also add a response time threshold to the S3 gateway HTTP monitor — S3 gateway slowness (>2 seconds for root responses) often precedes crashes:

  1. Edit the S3 gateway monitor.
  2. Set Response time threshold to 2000ms.
  3. Save.

Step 6: Monitor Cluster Storage Capacity

Ozone cluster capacity at >85% causes write failures as DataNodes run out of space to accept new containers:

#!/bin/bash
# /opt/monitoring/check-ozone-capacity.sh
MAX_USED_PCT=85
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_CAPACITY_KEY"

SCM_URL="http://ozone-scm:9876"

CAPACITY=$(curl -sf "$SCM_URL/api/v1/cluster/capacity" | python3 -c "
import sys, json
data = json.load(sys.stdin)
used = data.get('used', 0)
total = data.get('total', 1)
print(int(used / total * 100))
" 2>/dev/null)

if [ "${CAPACITY:-100}" -lt "$MAX_USED_PCT" ]; then
  curl -fsS "$HEARTBEAT_URL"
fi

Run every 10 minutes. Set the heartbeat interval to 20 minutes. Capacity growth is gradual — a 20-minute detection window gives time to add DataNodes or reclaim space before writes start failing at 90%+.

Step 7: Monitor the ReconServer

Ozone's ReconServer provides cluster analytics and is responsible for detecting missing containers. If ReconServer goes down, missing container detection stops — under-replicated and missing containers accumulate silently:

  1. In Vigilmon, click Add Monitor → HTTP / HTTPS.
  2. Enter: http://ozone-recon:9888/api/v1/task/status.
  3. Set Check interval to 5 minutes.
  4. Set Expected HTTP status to 200.
  5. Set Response body contains to "lastTaskRunStatus":"SUCCESS".
  6. Click Save.

ReconServer failures are lower-priority than OM or SCM failures, but they represent a blind spot: if ReconServer is down and containers start going under-replicated, you won't detect it until a user reports a read failure.

Step 8: Monitor Certificate Expiry (Secure Ozone)

If your Ozone cluster is configured with mTLS (a requirement for production deployments), all components use certificates for mutual authentication. Certificate expiry causes cluster communication failures that are catastrophic and non-obvious:

#!/bin/bash
# /opt/monitoring/check-ozone-certs.sh
WARN_DAYS=30  # Alert 30 days before expiry
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_CERT_KEY"

# Check certificate expiry for OM, SCM, and a DataNode
CERTS=(
  "/etc/ozone/certs/om.crt"
  "/etc/ozone/certs/scm.crt"
  "/etc/ozone/certs/dn.crt"
)

SOONEST_EXPIRY=99999  # days

for CERT in "${CERTS[@]}"; do
  if [ -f "$CERT" ]; then
    EXPIRY_DATE=$(openssl x509 -enddate -noout -in "$CERT" | cut -d= -f2)
    EXPIRY_EPOCH=$(date -d "$EXPIRY_DATE" +%s 2>/dev/null || \
                   date -j -f "%b %d %H:%M:%S %Y %Z" "$EXPIRY_DATE" +%s 2>/dev/null)
    NOW_EPOCH=$(date +%s)
    DAYS_REMAINING=$(( (EXPIRY_EPOCH - NOW_EPOCH) / 86400 ))
    if [ "$DAYS_REMAINING" -lt "$SOONEST_EXPIRY" ]; then
      SOONEST_EXPIRY=$DAYS_REMAINING
    fi
  fi
done

if [ "$SOONEST_EXPIRY" -gt "$WARN_DAYS" ]; then
  curl -fsS "$HEARTBEAT_URL"
fi

Run daily. Set the Vigilmon heartbeat interval to 48 hours. Certificate expiry within 30 days stops the heartbeat and fires an alert with enough lead time to rotate certificates before they expire.

Alternatively, add a Vigilmon SSL certificate monitor for the OM HTTPS endpoint if OM is exposed with TLS:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enable Monitor SSL certificate.
  3. Set Alert when certificate expires in less than 30 days.
  4. Click Save.

Step 9: Configure Alert Routing

| Monitor | Alert Channel | Severity | Impact | |---|---|---|---| | OM leader heartbeat | PagerDuty + Slack | Critical | All namespace operations blocked | | SCM leader heartbeat | PagerDuty + Slack | Critical | Container placement and replication halted | | DataNode count heartbeat | Slack | High | Replication degradation starting | | Missing container heartbeat | PagerDuty + Slack | Critical | Data loss risk — immediate investigation required | | Under-replicated container heartbeat | Slack | High | Data loss risk on next DataNode failure | | S3 gateway HTTP | PagerDuty + Slack | Critical | S3-compatible clients blocked | | Cluster capacity heartbeat | Slack | High | New writes failing at >85% | | ReconServer HTTP | Email | Medium | Missing container detection blind spot | | Certificate expiry heartbeat | Slack | High | mTLS failure in <30 days |

Set consecutive failures before alert to 1 for OM leader, SCM leader, and missing container monitors. These represent active or imminent data loss risk. Set it to 3 for DataNode count (transient restarts are expected during maintenance) and capacity (slow growth doesn't need immediate response).

Configure escalation paths in Vigilmon: if OM or SCM alerts do not acknowledge within 15 minutes, escalate to phone call via PagerDuty. Ozone OM leader loss blocking all namespace operations for 15+ minutes indicates the HA election is stuck and requires manual intervention.

Conclusion

Apache Ozone's multi-component architecture gives you independence from HDFS's NameNode scalability ceiling — but it introduces multiple independent failure surfaces that all need monitoring. An OM leader loss blocks all key operations; an SCM failure halts new container creation and replication repair; a sudden increase in stale DataNodes risks under-replication; an S3 gateway crash silently breaks all S3-compatible data pipelines. With Vigilmon monitoring the OM, SCM, DataNodes, containers, S3 gateway, cluster capacity, and certificate expiry, you have the external visibility to catch Ozone failures before they cascade to data pipelines and users.

Start with the OM and SCM health probes and the S3 gateway HTTP monitor — these cover the highest-impact failure modes for Ozone production clusters. Then add DataNode count and container replication monitoring to detect gradual degradation. Sign up for a free Vigilmon account to get started.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles