Monitoring Akri on Kubernetes with Vigilmon

Akri makes IoT devices first-class citizens in Kubernetes — but when the Akri controller crashes or an agent DaemonSet pod goes missing from a node, device discovery silently stops. Workloads requesting camera streams or OPC-UA endpoints get allocation failures with no obvious indication of why. Vigilmon fills that gap with cron heartbeat monitors that continuously verify Akri's control plane health, device discovery count, and broker pod status.

What You'll Set Up

• Akri controller pod health monitor
• Akri agent DaemonSet completeness check (all nodes covered)
• Device discovery count baseline monitor
• Device allocation success rate tracking
• Akri webhook health check
• Broker pod crash-loop detection

Prerequisites

• Kubernetes cluster with Akri installed (helm chart or YAML manifests)
• kubectl access to the cluster
• A free Vigilmon account
• A monitoring host with kubectl access (or a Kubernetes CronJob)

Step 1: Monitor the Akri Controller

The Akri controller manages the lifecycle of Instance CRDs — the Kubernetes objects that represent discovered devices. When the controller crashes, discovered devices become stale, workloads can't be scheduled to new devices, and Instances for disconnected devices are never cleaned up.

Create a script that checks controller pod health and sends a heartbeat to Vigilmon:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval to 2 minutes.
3. Copy the heartbeat URL (e.g. https://vigilmon.online/heartbeat/abc123).

#!/bin/bash
# akri-controller-health.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
NAMESPACE="akri"  # adjust if deployed to a different namespace

READY=$(kubectl get deployment akri-controller -n "$NAMESPACE" \
  -o jsonpath='{.status.readyReplicas}' 2>/dev/null)

if [ "$READY" -ge 1 ]; then
  curl -s "$HEARTBEAT_URL"
fi

Schedule with a Kubernetes CronJob so the check runs inside the cluster:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: akri-controller-health
  namespace: akri
spec:
  schedule: "*/2 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: akri-health-checker
          containers:
          - name: checker
            image: bitnami/kubectl:latest
            command:
            - /bin/sh
            - -c
            - |
              READY=$(kubectl get deployment akri-controller -n akri \
                -o jsonpath='{.status.readyReplicas}')
              if [ "${READY:-0}" -ge 1 ]; then
                wget -q -O- https://vigilmon.online/heartbeat/abc123
              fi
          restartPolicy: OnFailure

Step 2: Monitor Akri Agent DaemonSet Completeness

The Akri agent runs as a DaemonSet pod on every Kubernetes node. Each agent continuously scans for local devices matching configured discovery protocols (USB, ONVIF, OPC-UA, udev). If an agent pod is missing from a node — due to a resource eviction, taint change, or pod crash — that node's devices are invisible to the cluster.

#!/bin/bash
# akri-agent-health.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
NAMESPACE="akri"

DESIRED=$(kubectl get daemonset akri-agent -n "$NAMESPACE" \
  -o jsonpath='{.status.desiredNumberScheduled}' 2>/dev/null)
READY=$(kubectl get daemonset akri-agent -n "$NAMESPACE" \
  -o jsonpath='{.status.numberReady}' 2>/dev/null)

if [ "${DESIRED:-0}" -gt 0 ] && [ "${READY}" -eq "${DESIRED}" ]; then
  curl -s "$HEARTBEAT_URL"
fi

Set the Vigilmon heartbeat to 2 minutes. If any agent pod goes missing from any node, numberReady drops below desiredNumberScheduled and Vigilmon alerts.

Step 3: Monitor Device Discovery Count

Akri creates an Instance CRD for each discovered device. If the count drops below your fleet's baseline — say you normally see 8 ONVIF cameras and suddenly only see 5 — something is disconnected or a discovery protocol is failing.

#!/bin/bash
# akri-device-count.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
NAMESPACE="akri"
MIN_INSTANCES=5  # set to your expected minimum device count

INSTANCE_COUNT=$(kubectl get akrii -n "$NAMESPACE" --no-headers 2>/dev/null | wc -l)

if [ "${INSTANCE_COUNT:-0}" -ge "$MIN_INSTANCES" ]; then
  curl -s "$HEARTBEAT_URL"
fi

For finer-grained alerting by protocol type:

# Count ONVIF devices specifically
ONVIF_COUNT=$(kubectl get akrii -n "$NAMESPACE" \
  -l "akri.sh/configuration=akri-onvif" --no-headers 2>/dev/null | wc -l)

Run every 5 minutes. If device count drops below baseline, the heartbeat stops and Vigilmon alerts.

Step 4: Monitor Device Allocation Success Rate

When workloads request Akri devices (via akri.sh/configuration resource requests), the Akri controller allocates a specific Instance to the requesting pod. Allocation failures mean workloads start but can't access the devices they need.

Monitor allocation health by checking for Kubernetes events signaling allocation failures:

#!/bin/bash
# akri-allocation-health.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"
NAMESPACE="akri"

# Check for FailedMount or allocation error events in the last 5 minutes
FAILURES=$(kubectl get events -n "$NAMESPACE" \
  --field-selector reason=FailedMount \
  --sort-by='.lastTimestamp' 2>/dev/null | \
  awk -v cutoff="$(date -u -d '5 minutes ago' +%Y-%m-%dT%H:%M:%SZ)" \
  '$1 > cutoff' | wc -l)

if [ "${FAILURES:-0}" -eq 0 ]; then
  curl -s "$HEARTBEAT_URL"
fi

Set the Vigilmon heartbeat to 5 minutes. Any allocation failure event in the last 5 minutes pauses the heartbeat.

Step 5: Monitor the Akri Webhook

Akri registers a ValidatingWebhookConfiguration to validate Akri CRDs before they're admitted to the cluster. If the webhook pod becomes unhealthy, any attempt to create or update an Akri Configuration or Instance is rejected by the Kubernetes API server with a cryptic admission webhook error.

#!/bin/bash
# akri-webhook-health.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/mno345"
NAMESPACE="akri"

# Check webhook deployment
READY=$(kubectl get deployment akri-webhook-configuration -n "$NAMESPACE" \
  -o jsonpath='{.status.readyReplicas}' 2>/dev/null)

if [ "${READY:-0}" -ge 1 ]; then
  curl -s "$HEARTBEAT_URL"
fi

Alternatively, add an HTTP monitor directly to the webhook service if it exposes a health endpoint:

1. Add a monitor → HTTP / HTTPS.
2. Enter: https://akri-webhook.akri.svc.cluster.local/health (accessible from inside the cluster).
3. Set Expected HTTP status to 200.
4. Set Check interval to 2 minutes.

Step 6: Monitor Broker Pod Health

Akri schedules "broker" pods — workloads that the controller places on nodes with access to discovered devices. A broker pod crash-looping means the application trying to use the device (e.g., a video analytics pipeline) is continuously failing. Detect crash-loops across all Akri broker pods:

#!/bin/bash
# akri-broker-health.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/pqr678"
NAMESPACE="default"  # broker pods are typically in the workload namespace

# Find pods with akri broker label that are in CrashLoopBackOff
CRASHLOOPS=$(kubectl get pods -n "$NAMESPACE" \
  -l "akri.sh/instance" \
  --field-selector status.phase=Running 2>/dev/null | \
  grep -c CrashLoopBackOff || echo 0)

if [ "${CRASHLOOPS}" -eq 0 ]; then
  curl -s "$HEARTBEAT_URL"
fi

For a more robust check using JSON output:

CRASHLOOPS=$(kubectl get pods -n "$NAMESPACE" -l "akri.sh/instance" \
  -o jsonpath='{.items[*].status.containerStatuses[*].state.waiting.reason}' | \
  tr ' ' '\n' | grep -c CrashLoopBackOff || echo 0)

Step 7: ONVIF Camera Stream Reachability

If you're using Akri to discover ONVIF IP cameras, verify that the camera streams themselves are reachable independently of the Akri discovery layer:

1. Add a monitor → TCP Port for each camera.
2. Set Host to the camera's IP address.
3. Set Port to 80 (ONVIF uses HTTP for device management) or 554 (RTSP stream port).
4. Set Check interval to 2 minutes.

For cameras on a local network segment not directly reachable from Vigilmon's cloud probes, use a Vigilmon on-premise probe agent deployed on the Kubernetes cluster's host network.

Step 8: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and configure Slack, email, or PagerDuty.
2. Set Consecutive failures before alert:
   • Akri controller: 1 — controller crashes are never transient
   • Agent DaemonSet: 2 — allow one reconcile cycle before alerting
   • Device count: 3 — allow brief discovery gaps from device reboots
   • Broker pods: 2 — allow one restart before alerting on crash-loop
3. Label monitors clearly (e.g., "Akri Controller", "Akri ONVIF Agents", "Camera Brokers") so on-call responders know which component to investigate.

Summary

| Monitor | Target | What It Catches | |---|---|---| | Akri controller | Deployment readyReplicas | Device lifecycle management failure | | Akri agents | DaemonSet completeness | Per-node discovery loss | | Device count | Instance CRD count | Devices disconnecting or discovery failure | | Allocation health | Event failures | Workloads unable to access devices | | Akri webhook | Deployment readyReplicas | CRD admission failures | | Broker pods | CrashLoopBackOff check | Application-level device access failure | | ONVIF cameras | TCP :80 / :554 | Camera hardware offline |

Akri makes IoT devices schedulable Kubernetes resources — but its value depends on the controller, agents, and broker pods all staying healthy. With Vigilmon monitoring each layer, you'll catch discovery failures and device disconnects before your edge workloads start returning errors to end users.