Monitoring Kairos Edge Nodes with Vigilmon

Kairos is an open source, immutable Linux meta-distribution for edge Kubernetes deployments developed by Spectro Cloud. The OS is read-only at runtime and runs k3s (lightweight Kubernetes) out of the box. Edge nodes running Kairos may be in remote locations, operating on constrained hardware with intermittent WAN connectivity — which makes monitoring more important and more difficult than in a datacenter.

When a Kairos node goes offline, a bad OTA update triggers a rollback, or k3s crashes on an edge node, you need to know within minutes. Vigilmon gives you node reachability monitoring, A/B update health checks, k3s API server monitoring, filesystem integrity alerts, and fleet-wide visibility for your entire Kairos deployment.

What You'll Set Up

• Node reachability monitoring (SSH or k3s API server)
• A/B partition update health via heartbeat
• k3s API server health monitoring per node
• Read-only filesystem integrity alerts
• Node resource usage (CPU, memory, persistent storage)
• AuroraBoot PXE server health (if used)
• OS update rollback detection
• Edge fleet online/offline count monitoring

Prerequisites

• One or more Kairos nodes running k3s
• SSH access to Kairos nodes (or Kubernetes API access)
• AuroraBoot deployed (optional, if PXE provisioning is used)
• A free Vigilmon account

Step 1: Monitor Node Reachability

The most fundamental Kairos edge node check is whether the node itself is reachable. Use a TCP port monitor against the k3s API server port (6443) — it's available on every Kairos node running k3s and does not require exposing an HTTP endpoint.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to TCP Port.
3. Enter the node's public IP (or WAN IP) and port: 203.0.113.42:6443
4. Set Check interval to 2 minutes.
5. Set Alert after 2 consecutive failures.
6. Click Save.

Repeat for each production Kairos edge node. Use a descriptive name like kairos-node-001 (k3s API) to identify which physical node the monitor covers.

If your edge nodes are behind NAT without port forwarding, use the SSH port (22) instead:

1. Set Type to TCP Port.
2. Enter: edge-node.yourdomain.com:22
3. Set Check interval to 2 minutes.

Step 2: Monitor k3s API Server Health

Beyond raw TCP reachability, verify the k3s API server is responding to Kubernetes health checks. k3s exposes a /healthz endpoint on port 6443:

1. In Vigilmon, click Add Monitor → HTTP / HTTPS.
2. Enter: https://KAIROS_NODE_IP:6443/healthz
3. Set Expected HTTP status to 200.
4. Set Expected body to contain ok.
5. Set Check interval to 2 minutes.
6. Disable SSL certificate validation if using the self-signed cert that k3s generates by default.
7. Click Save.

For the k3s node status (whether the node itself is in Ready state), run a periodic check from a management host:

#!/bin/bash
# Run as Kubernetes CronJob on a management cluster
NODE_STATUS=$(kubectl get node kairos-node-001 -o jsonpath='{.status.conditions[-1].type}')
if [ "$NODE_STATUS" != "Ready" ]; then
  echo "Node not ready: $NODE_STATUS"
  exit 1
fi
curl -s https://vigilmon.online/heartbeat/YOUR_K3S_NODE_HB_TOKEN

Schedule this as a Kubernetes CronJob every 5 minutes. If it stops pinging, Vigilmon alerts after the heartbeat interval passes.

Step 3: Monitor A/B Partition Update Health

Kairos uses A/B partitions for safe OTA updates — the update writes to the passive partition and activates on reboot. If an update fails or triggers an automatic rollback, you need immediate notification.

Heartbeat from successful updates

Create a Kairos cloud-init hook that pings Vigilmon after a successful update and reboot:

# /oem/99-vigilmon-update-hook.yaml
name: "Vigilmon update success heartbeat"
stages:
  boot:
    - name: "Ping Vigilmon on successful Kairos boot"
      commands:
        - |
          # Only ping if this is the first boot after an update
          if [ -f /run/cos/upgrade_completed ]; then
            curl -sf https://vigilmon.online/heartbeat/YOUR_UPDATE_HB_TOKEN || true
            rm /run/cos/upgrade_completed
          fi

Set the Vigilmon heartbeat interval to 48 hours (or your typical update cadence). If the passive partition activates but the node never pings (indicating a failed post-update boot), Vigilmon alerts.

Rollback detection

Kairos activates the passive partition automatically when a new update fails to boot. Detect rollbacks by comparing the expected active partition version:

#!/bin/bash
# Check if running on an older-than-expected version (indicating rollback)
CURRENT_VERSION=$(kairos-agent state | grep 'kairos.version' | awk '{print $2}')
EXPECTED_VERSION=$(cat /etc/kairos/expected-version)
if [ "$CURRENT_VERSION" != "$EXPECTED_VERSION" ]; then
  echo "Rollback detected: running $CURRENT_VERSION, expected $EXPECTED_VERSION"
  # Do NOT ping heartbeat — let it expire to trigger alert
  exit 1
fi
curl -s https://vigilmon.online/heartbeat/YOUR_UPDATE_HB_TOKEN

Step 4: Monitor the Read-Only Filesystem

Kairos uses an immutable root filesystem mounted read-only at runtime. An unexpected rw remount of the root filesystem indicates tampering, a system misconfiguration, or a kernel/initrd issue.

Add a health script to each Kairos node that runs periodically via k3s CronJob:

#!/bin/bash
# Verify root filesystem is mounted read-only
ROOT_MOUNT=$(findmnt -n -o OPTIONS / | grep -c "ro,")
if [ "$ROOT_MOUNT" -eq 0 ]; then
  echo "ALERT: Root filesystem is NOT read-only"
  exit 1
fi
curl -s https://vigilmon.online/heartbeat/YOUR_FS_INTEGRITY_HB_TOKEN

Deploy as a k3s CronJob running every 10 minutes:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: kairos-fs-integrity-check
  namespace: monitoring
spec:
  schedule: "*/10 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          hostPID: true
          containers:
          - name: checker
            image: alpine:3
            command: ["/bin/sh", "-c"]
            args:
              - |
                ROOT_MOUNT=$(nsenter -m -t 1 findmnt -n -o OPTIONS / | grep -c "ro,")
                if [ "$ROOT_MOUNT" -eq 0 ]; then exit 1; fi
                wget -q https://vigilmon.online/heartbeat/YOUR_FS_INTEGRITY_HB_TOKEN -O /dev/null
            securityContext:
              privileged: true
          restartPolicy: OnFailure

Step 5: Monitor Node Resource Usage

Edge nodes are resource-constrained. Storage exhaustion on the persistent data partition is a common failure mode — the read-only root filesystem is fine, but writable partitions (the OEM partition, persistent state) can fill up.

Expose a lightweight metrics endpoint on each Kairos node using a k3s-deployed pod:

# node-resource-exporter.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: resource-health
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: resource-health
  template:
    metadata:
      labels:
        app: resource-health
    spec:
      hostNetwork: true
      volumes:
      - name: host-root
        hostPath:
          path: /
      containers:
      - name: exporter
        image: alpine:3
        ports:
        - containerPort: 9101
        command: ["/bin/sh", "-c"]
        args:
          - |
            while true; do
              STORAGE=$(df /host/var | tail -1 | awk '{print $5}' | tr -d '%')
              MEM=$(free | awk '/^Mem/{printf "%.0f", $3/$2*100}')
              STATUS="ok"
              if [ "$STORAGE" -gt 85 ] || [ "$MEM" -gt 90 ]; then STATUS="warn"; fi
              echo -e "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\n\r\nstorage=${STORAGE}% mem=${MEM}% status=${STATUS}" \
                | nc -l -p 9101 -q 1 || true
            done
        volumeMounts:
        - name: host-root
          mountPath: /host

Add an HTTP monitor in Vigilmon for this endpoint:

1. In Vigilmon, click Add Monitor → HTTP / HTTPS.
2. Enter: http://KAIROS_NODE_IP:9101/
3. Set Expected body to NOT contain warn.
4. Set Check interval to 5 minutes.
5. Click Save.

Step 6: Monitor AuroraBoot Server Health

If you use AuroraBoot for PXE provisioning of new Kairos nodes, monitor it as a standard HTTP service. An AuroraBoot crash means you cannot provision new nodes:

1. In Vigilmon, click Add Monitor → HTTP / HTTPS.
2. Enter the AuroraBoot server URL: http://auroraboot.yourdomain.com:8080
3. Set Expected HTTP status to 200.
4. Set Check interval to 5 minutes.
5. Click Save.

Also add a TCP monitor for the TFTP port used in PXE booting:

1. Click Add Monitor → TCP Port.
2. Enter: auroraboot.yourdomain.com:69
3. Set Check interval to 5 minutes.
4. Click Save.

Step 7: Monitor Fleet Online Percentage

For a fleet of Kairos edge nodes, track the percentage of nodes online. A drop below a threshold indicates widespread connectivity loss or a bad OTA update pushed to the fleet.

Deploy a fleet aggregator as a k3s CronJob on your management cluster:

#!/bin/bash
TOTAL_NODES=$(kubectl get nodes --no-headers | wc -l)
READY_NODES=$(kubectl get nodes --no-headers | grep -c " Ready ")
OFFLINE_NODES=$((TOTAL_NODES - READY_NODES))
ONLINE_PCT=$((READY_NODES * 100 / TOTAL_NODES))

echo "Fleet status: $READY_NODES/$TOTAL_NODES online ($ONLINE_PCT%)"

if [ "$ONLINE_PCT" -lt 90 ]; then
  echo "Fleet online percentage below 90% - not pinging heartbeat"
  exit 1
fi
curl -s https://vigilmon.online/heartbeat/YOUR_FLEET_HB_TOKEN

Set the Vigilmon heartbeat interval to 15 minutes. If more than 10% of the fleet goes offline, the heartbeat stops and Vigilmon alerts.

Step 8: Configure Alert Channels and Thresholds

1. Go to Alert Channels in Vigilmon and add Slack, email, or PagerDuty.
2. Configure per-monitor thresholds:
   • Node TCP reachability: alert after 2 failures (allow for transient WAN blips).
   • k3s API server: alert after 2 failures.
   • Filesystem integrity: alert after 1 failure (tampered root is always critical).
   • AuroraBoot: alert after 3 failures (less urgent than live nodes).
   • Fleet heartbeat: alert when heartbeat expires (configured in heartbeat interval).
3. Use Maintenance Windows before fleet-wide OTA updates:

# Suppress fleet alerts during planned OTA update window
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"monitor_id": "FLEET_MONITOR_ID", "duration_minutes": 60}'

Summary

| Monitor | Target | What It Catches | |---|---|---| | Node TCP reachability | Node IP:6443 or :22 | Node unreachable, network loss | | k3s API server | /healthz on each node | k3s crash, NotReady state | | A/B update heartbeat | Heartbeat URL | Failed OTA update or rollback | | Filesystem integrity | / mount mode check | Unexpected rw remount | | Node resource usage | Custom health endpoint | Storage > 85%, high memory | | AuroraBoot | HTTP :8080 + TCP :69 | New node provisioning blocked | | Fleet online % | k3s node count | Widespread outage or bad update |

Kairos edge nodes are designed to be autonomous and resilient — but when things go wrong at the edge, you often find out from end users, not from dashboards. With Vigilmon monitoring each node's reachability, k3s health, filesystem integrity, and OTA update success, you build the observability layer that Kairos's immutable design doesn't include out of the box.