OpenEBS Mayastor (now branded as Replicated PV Mayastor in OpenEBS 4.x) is a high-performance container-attached storage (CAS) engine for Kubernetes built on SPDK (Storage Performance Development Kit), NVMe-oF (NVMe over Fabrics), and io_uring. By running the entire I/O path in userspace and bypassing the kernel storage stack, Mayastor achieves sub-100 microsecond latency and high IOPS on NVMe hardware that is impossible to match with traditional kernel-based storage drivers. The tradeoff for that performance is a tightly coupled architecture: DiskPool health, Nexus replica sync, NVMe-oF target connectivity, and SPDK huge page allocation must all be healthy simultaneously for stateful pods to have functioning storage.
This tutorial shows you how to monitor OpenEBS Mayastor's critical health surfaces with Vigilmon so you detect storage degradation before it becomes application I/O failures.
Why Mayastor needs external monitoring
Mayastor surfaces metrics through Prometheus and a REST API, but internal metrics only capture what the control plane knows. External monitoring adds independent verification that catches failures the control plane may not report:
- io-engine pod crash — the SPDK-based io-engine DaemonSet exits on a storage node; all NVMe-oF targets on that node become unreachable; any Nexus using replicas on that node enters degraded state
- DiskPool entering degraded state — a DiskPool (raw NVMe device abstraction) goes degraded when the underlying disk has errors; volumes using that pool stop accepting new replicas
- Nexus degraded or faulted — the replicated volume (Nexus) loses enough replicas to drop below the minimum redundancy threshold; one more failure means data loss
- Rebuild stalled — a replica rebuild triggered by a failure stops making progress; the Nexus stays in degraded state indefinitely, vulnerable to a second failure
- NVMe-oF target unreachable — the NVMe-oF target can be unreachable even when the io-engine pod shows
Running, because the SPDK reactor may have crashed internally - Control plane crash — the Mayastor REST API server going down blocks all volume provisioning and management operations
- Huge page allocation failure — io-engine requires huge pages at startup; if the node doesn't have them allocated, io-engine crashes silently
What you'll need
- A Kubernetes cluster with OpenEBS Mayastor installed
kubectlaccess with permission to read Mayastor resources- A free Vigilmon account
Step 1: Expose the Mayastor REST API health endpoint
The Mayastor control plane exposes a REST API (rest deployment, typically in the mayastor namespace). Expose its health endpoint for external monitoring:
# Verify the REST API pod is running
kubectl get pods -n mayastor -l app=rest
# Check the REST API service
kubectl get svc -n mayastor | grep rest
# Expose via NodePort for external monitoring
kubectl expose deployment rest \
--name=rest-external \
--type=NodePort \
--port=8081 \
--target-port=8081 \
-n mayastor
Or use an Ingress limited to the health path:
# mayastor-health-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: mayastor-health
namespace: mayastor
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /v0/nodes
spec:
rules:
- host: mayastor.internal.example.com
http:
paths:
- path: /healthz
pathType: Prefix
backend:
service:
name: rest
port:
number: 8081
Verify the API responds:
# List nodes via REST API (good health probe)
NODE_IP=$(kubectl get svc rest-external -n mayastor \
-o jsonpath='{.spec.clusterIP}')
curl http://$NODE_IP:8081/v0/nodes
Step 2: Monitor the Mayastor control plane health
The REST API is both an operational interface and a health signal. If the control plane is down, you cannot provision volumes, check pool status, or trigger rebuilds.
- Log in to vigilmon.online and go to Monitors → New Monitor
- Choose HTTP / HTTPS
- URL:
http://mayastor.internal.example.com/v0/nodes - Check interval: 1 minute
- Expected response: status code
200 - Name:
mayastor-control-plane - Save
When the control plane goes down, Vigilmon fires within two minutes — well before most automated alerting pipelines that depend on Prometheus scraping the now-absent service.
Step 3: Monitor io-engine DaemonSet coverage
The io-engine DaemonSet runs on storage nodes with local NVMe SSDs. If it's missing from any storage node, that node's NVMe-oF targets are gone and any Nexus using replicas there enters degraded state.
# io-engine-coverage-check.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
name: io-engine-coverage-check
namespace: mayastor
spec:
schedule: "*/5 * * * *"
jobTemplate:
spec:
template:
spec:
serviceAccountName: mayastor-health-checker
restartPolicy: OnFailure
containers:
- name: checker
image: bitnami/kubectl:latest
env:
- name: VIGILMON_HEARTBEAT_URL
valueFrom:
secretKeyRef:
name: vigilmon-secrets
key: io-engine-coverage-heartbeat
command:
- /bin/sh
- -c
- |
set -e
DESIRED=$(kubectl get daemonset io-engine -n mayastor \
-o jsonpath='{.status.desiredNumberScheduled}')
READY=$(kubectl get daemonset io-engine -n mayastor \
-o jsonpath='{.status.numberReady}')
if [ "$READY" -lt "$DESIRED" ]; then
echo "ERROR: io-engine $READY/$DESIRED storage nodes running"
exit 1
fi
curl -fsS "$VIGILMON_HEARTBEAT_URL" > /dev/null
echo "io-engine $READY/$DESIRED storage nodes healthy."
Configure in Vigilmon:
- Go to Monitors → New Monitor → Heartbeat
- Name:
io-engine-daemonset-coverage - Expected interval: 5 minutes
- Grace period: 3 minutes
When this heartbeat stops firing, immediately check which storage node is missing io-engine:
# Find storage nodes
kubectl get nodes -l openebs.io/engine=mayastor --no-headers | awk '{print $1}'
# Find nodes with io-engine pods
kubectl get pods -n mayastor -l app=io-engine -o wide --no-headers | awk '{print $7}'
Step 4: Monitor DiskPool and Nexus health
DiskPool degradation and Nexus volume faults are the two most critical storage health signals. Set up a heartbeat that checks both:
# storage-health-check.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
name: mayastor-storage-health-check
namespace: mayastor
spec:
schedule: "*/5 * * * *"
jobTemplate:
spec:
template:
spec:
serviceAccountName: mayastor-health-checker
restartPolicy: OnFailure
containers:
- name: checker
image: curlimages/curl:latest
env:
- name: MAYASTOR_API
value: "http://rest.mayastor.svc.cluster.local:8081"
- name: VIGILMON_HEARTBEAT_URL
valueFrom:
secretKeyRef:
name: vigilmon-secrets
key: storage-health-heartbeat
command:
- /bin/sh
- -c
- |
set -e
# Check DiskPool health
DEGRADED_POOLS=$(curl -fsS "$MAYASTOR_API/v0/pools" | \
jq '[.[] | select(.state.status != "Online")] | length')
if [ "$DEGRADED_POOLS" -gt 0 ]; then
echo "ERROR: $DEGRADED_POOLS DiskPools not Online"
curl -fsS "$MAYASTOR_API/v0/pools" | \
jq '.[] | select(.state.status != "Online") | {name: .id, status: .state.status}'
exit 1
fi
# Check Nexus (volume) health
DEGRADED_VOLUMES=$(curl -fsS "$MAYASTOR_API/v0/volumes" | \
jq '[.entries[] | select(.state.status != "Online")] | length')
if [ "$DEGRADED_VOLUMES" -gt 0 ]; then
echo "ERROR: $DEGRADED_VOLUMES Nexus volumes degraded or faulted"
exit 1
fi
curl -fsS "$VIGILMON_HEARTBEAT_URL" > /dev/null
echo "All DiskPools Online. All volumes healthy."
Step 5: Monitor replica rebuild health
When a replica is lost (io-engine crash, disk failure), Mayastor starts a rebuild from a surviving replica. A stuck rebuild means the Nexus stays in degraded state — vulnerable to data loss if another replica is lost.
# replica-rebuild-check.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
name: mayastor-replica-rebuild-check
namespace: mayastor
spec:
schedule: "*/10 * * * *"
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: checker
image: curlimages/curl:latest
env:
- name: MAYASTOR_API
value: "http://rest.mayastor.svc.cluster.local:8081"
- name: MAX_REBUILD_MINUTES
value: "30"
- name: VIGILMON_HEARTBEAT_URL
valueFrom:
secretKeyRef:
name: vigilmon-secrets
key: replica-rebuild-heartbeat
command:
- /bin/sh
- -c
- |
set -e
NOW=$(date -u +%s)
MAX_AGE=$((MAX_REBUILD_MINUTES * 60))
STUCK=$(curl -fsS "$MAYASTOR_API/v0/volumes" | jq --argjson now "$NOW" --argjson max "$MAX_AGE" '
[.entries[] |
select(.state.rebuildProgress != null) |
select(
($now - (.state.rebuildStartTime // $now | strptime("%Y-%m-%dT%H:%M:%SZ") | mktime)) > $max
)
] | length' 2>/dev/null || echo 0)
if [ "$STUCK" -gt 0 ]; then
echo "ERROR: $STUCK volumes have rebuilds stuck >$MAX_REBUILD_MINUTES minutes"
exit 1
fi
curl -fsS "$VIGILMON_HEARTBEAT_URL" > /dev/null
echo "No stuck rebuilds. Heartbeat sent."
Step 6: Monitor PVC provisioning success
New PVCs backed by Mayastor StorageClass must go through control plane provisioning. Provisioning failures block stateful workload deployments.
# pvc-provisioning-probe.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
name: mayastor-pvc-probe
namespace: mayastor-test
spec:
schedule: "*/15 * * * *"
jobTemplate:
spec:
template:
spec:
serviceAccountName: mayastor-health-checker
restartPolicy: Never
containers:
- name: prober
image: bitnami/kubectl:latest
env:
- name: VIGILMON_HEARTBEAT_URL
valueFrom:
secretKeyRef:
name: vigilmon-secrets
key: pvc-provisioning-heartbeat
command:
- /bin/sh
- -c
- |
set -e
PROBE_NAME="probe-$(date +%s)"
START=$(date +%s)
# Create a 1Gi test PVC
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: $PROBE_NAME
namespace: mayastor-test
spec:
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 1Gi
storageClassName: mayastor-3
EOF
# Wait up to 30 seconds for Bound
for i in $(seq 1 30); do
STATUS=$(kubectl get pvc "$PROBE_NAME" -n mayastor-test \
-o jsonpath='{.status.phase}' 2>/dev/null || echo "Pending")
if [ "$STATUS" = "Bound" ]; then
ELAPSED=$(( $(date +%s) - START ))
echo "PVC bound in ${ELAPSED}s"
kubectl delete pvc "$PROBE_NAME" -n mayastor-test --ignore-not-found
curl -fsS "$VIGILMON_HEARTBEAT_URL" > /dev/null
exit 0
fi
sleep 1
done
echo "ERROR: PVC not bound within 30 seconds"
kubectl delete pvc "$PROBE_NAME" -n mayastor-test --ignore-not-found
exit 1
Step 7: Monitor I/O latency for storage SLOs
Mayastor's primary value is sub-100 microsecond latency. Monitor P99 write latency for your critical volumes using a write-latency probe:
# Check Mayastor volume I/O stats via REST API
curl http://rest.mayastor.svc.cluster.local:8081/v0/volumes | \
jq '.entries[] | {
name: .spec.uuid,
write_latency_us: .state.child_states[]?.stats?.write_latency_us
}'
For a simpler Vigilmon integration, expose an HTTP endpoint from a sidecar application that writes and reads a test block, measuring latency:
# storage-latency-probe.py — expose as /latency-check endpoint
import time, os, http.server, socketserver
PROBE_PATH = "/mnt/mayastor-probe/latency-test"
class Handler(http.server.BaseHTTPRequestHandler):
def do_GET(self):
if self.path == "/latency-check":
try:
start = time.perf_counter()
with open(PROBE_PATH, "wb") as f:
f.write(b"x" * 4096)
f.flush()
os.fsync(f.fileno())
elapsed_us = (time.perf_counter() - start) * 1_000_000
if elapsed_us > 500: # Alert if write > 500 microseconds
self.send_response(503)
msg = f"write latency {elapsed_us:.0f}us > 500us threshold"
else:
self.send_response(200)
msg = f"write latency {elapsed_us:.0f}us OK"
self.end_headers()
self.wfile.write(msg.encode())
except Exception as e:
self.send_response(500)
self.end_headers()
self.wfile.write(str(e).encode())
else:
self.send_response(404)
self.end_headers()
with socketserver.TCPServer(("", 8090), Handler) as httpd:
httpd.serve_forever()
Deploy this as a sidecar in a test pod with a Mayastor PVC mounted at /mnt/mayastor-probe/, then add a Vigilmon HTTP monitor targeting the /latency-check endpoint with a 1-second response time threshold.
Step 8: Configure alert channels
Mayastor failures are high-severity storage incidents. Route to your storage on-call path immediately.
Critical (page immediately — PagerDuty or equivalent):
mayastor-control-plane(HTTP)io-engine-daemonset-coverage(heartbeat)mayastor-storage-health(heartbeat — DiskPool/Nexus degraded)mayastor-pvc-provisioning(heartbeat)
Warning (notify — Slack #storage-ops):
mayastor-replica-rebuild-check(heartbeat — stuck rebuild)mayastor-latency-probe(HTTP — latency SLO degradation)
In Vigilmon, go to Alert Channels → New Channel for each destination and assign monitors accordingly.
Step 9: Create a storage infrastructure status page
- In Vigilmon, go to Status Pages → New Status Page
- Name: "Mayastor Storage Health"
- Add monitors grouped by layer:
- Control Plane:
mayastor-control-plane - Storage Nodes:
io-engine-daemonset-coverage - Volume Health:
mayastor-storage-health,mayastor-replica-rebuild-check - Provisioning:
mayastor-pvc-provisioning - Performance:
mayastor-latency-probe
- Control Plane:
- Share with SRE and storage engineering teams
Summary
| Monitor | Type | What it catches |
|---|---|---|
| mayastor REST API /v0/nodes | HTTP | Control plane crash, no volume provisioning |
| io-engine-daemonset-coverage | Heartbeat | io-engine missing on storage node |
| mayastor-storage-health | Heartbeat | DiskPool degraded, Nexus faulted |
| mayastor-replica-rebuild-check | Heartbeat | Rebuild stuck >30 min, lingering degraded state |
| mayastor-pvc-provisioning | Heartbeat | PVC creation failing or too slow |
| mayastor-latency-probe /latency-check | HTTP | Write latency >500μs, NVMe saturation |
Mayastor's performance advantage is only realized when the entire chain — io-engine pods, DiskPool health, Nexus replica count, NVMe-oF targets, and huge page allocation — stays healthy simultaneously. One degraded DiskPool or a missing io-engine can cascade from a single-replica loss to a data availability incident if a second failure hits before the rebuild completes. External monitoring with Vigilmon ensures you detect each failure mode independently and at the layer it occurs.
Get started free at vigilmon.online — no credit card required, first monitor running in under two minutes.