tutorial

Monitoring Virtink VMs with Vigilmon

Virtink is a lightweight Kubernetes VM operator from SmartX that runs cloud-native virtual machines without the complexity of KubeVirt. Here's how to monitor the Virtink controller, daemon, VM power states, live migrations, and per-VM health with Vigilmon.

Virtink is an open source, lightweight Kubernetes operator for running virtual machines developed by SmartX. It uses QEMU with virtio and cloud-hypervisor as the VM backend, targeting cloud-native VMs that need only virtio drivers without full PC hardware emulation. Virtink VMs are defined as Kubernetes CRDs (VirtualMachine and VirtualMachineMigration) and managed by the virtink-controller and virtink-daemon DaemonSet.

When the Virtink controller crashes, VM lifecycle management stops — you can't create, start, or stop VMs via Kubernetes. When a daemon pod goes missing on a node, VMs can't be scheduled there. When a production VM silently powers off, your workload is gone. Vigilmon lets you monitor the Virtink control plane, per-node daemon health, individual VM power states, live migration status, and VM network connectivity in a single dashboard.

What You'll Set Up

  • Virtink controller deployment health monitoring
  • Virtink daemon DaemonSet completeness check
  • Per-VM power state and network reachability monitoring
  • Live migration success and duration alerts
  • VM disk (PVC) binding health
  • Validating webhook response time monitoring
  • cloud-hypervisor process health per VM

Prerequisites

  • Virtink installed on a Kubernetes cluster (v1.21+)
  • kubectl access to the virtink-system namespace
  • VMs defined as VirtualMachine CRDs
  • A free Vigilmon account

Step 1: Monitor the Virtink Controller

The virtink-controller is a Kubernetes Deployment that manages the lifecycle of VirtualMachine CRDs. If it crashes, no new VMs can be created, started, stopped, or migrated — all VM lifecycle operations halt.

Deploy a controller health checker as a Kubernetes CronJob:

#!/bin/bash
# Check virtink-controller deployment
READY=$(kubectl get deployment virtink-controller-manager -n virtink-system \
  -o jsonpath='{.status.readyReplicas}')
DESIRED=$(kubectl get deployment virtink-controller-manager -n virtink-system \
  -o jsonpath='{.spec.replicas}')

if [ "$READY" != "$DESIRED" ] || [ -z "$READY" ]; then
  echo "virtink-controller unhealthy: $READY/$DESIRED ready"
  exit 1
fi
curl -s https://vigilmon.online/heartbeat/YOUR_CONTROLLER_HB_TOKEN

Deploy as a Kubernetes CronJob running every 2 minutes:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: virtink-controller-health
  namespace: monitoring
spec:
  schedule: "*/2 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: monitoring-reader
          containers:
          - name: checker
            image: bitnami/kubectl:latest
            command: ["/bin/bash", "-c"]
            args:
              - |
                READY=$(kubectl get deployment virtink-controller-manager \
                  -n virtink-system -o jsonpath='{.status.readyReplicas}')
                DESIRED=$(kubectl get deployment virtink-controller-manager \
                  -n virtink-system -o jsonpath='{.spec.replicas}')
                if [ "$READY" = "$DESIRED" ] && [ -n "$READY" ]; then
                  wget -q https://vigilmon.online/heartbeat/YOUR_CONTROLLER_HB_TOKEN -O /dev/null
                fi
          restartPolicy: OnFailure

Set the Vigilmon heartbeat interval to 5 minutes. If the controller goes missing, the heartbeat expires and you get an alert.

Step 2: Monitor the Virtink Daemon DaemonSet

The virtink-daemon DaemonSet runs on every node and handles VM lifecycle operations on that node (starting VMs, managing QEMU processes). If a node is missing its daemon pod, VMs cannot be scheduled to that node.

Monitor daemon completeness with a CronJob that fails if any node is missing a daemon pod:

#!/bin/bash
DESIRED=$(kubectl get daemonset virtink-daemon -n virtink-system \
  -o jsonpath='{.status.desiredNumberScheduled}')
READY=$(kubectl get daemonset virtink-daemon -n virtink-system \
  -o jsonpath='{.status.numberReady}')

echo "virtink-daemon: $READY/$DESIRED ready"

if [ "$READY" != "$DESIRED" ] || [ -z "$READY" ]; then
  echo "ALERT: Not all nodes have a running virtink-daemon"
  exit 1
fi
curl -s https://vigilmon.online/heartbeat/YOUR_DAEMON_HB_TOKEN

Set the Vigilmon heartbeat interval to 5 minutes. A missing daemon pod on any node triggers the alert.

In Vigilmon:

  1. Click Add Monitor → Cron Heartbeat.
  2. Set the expected interval to 5 minutes.
  3. Copy the heartbeat URL and paste it into the script above.
  4. Click Save.

Step 3: Monitor VM Power States

Production VMs should always be in Running state. An unexpected power-off (due to a node failure, OOM kill, or QEMU crash) means your workload is gone until someone notices.

Create a per-VM heartbeat approach — each VM runs a simple HTTP server that Vigilmon probes:

Option A: HTTP endpoint inside the VM

If your VM runs a web service, add a health endpoint and monitor it directly:

  1. In Vigilmon, click Add Monitor → HTTP / HTTPS.
  2. Enter the VM's service URL: http://VM_IP/health (or via a LoadBalancer/NodePort service).
  3. Set Expected HTTP status to 200.
  4. Set Check interval to 1 minute.
  5. Click Save.

Option B: External power state check via Kubernetes API

For VMs not running web services, check the VirtualMachine CRD power state from outside:

#!/bin/bash
# Check all production VMs are in Running phase
FAILED_VMS=$(kubectl get virtualmachines -A \
  -l tier=production \
  -o jsonpath='{range .items[?(@.status.phase!="Running")]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}')

if [ -n "$FAILED_VMS" ]; then
  echo "Production VMs not running: $FAILED_VMS"
  exit 1
fi
curl -s https://vigilmon.online/heartbeat/YOUR_VM_POWER_HB_TOKEN

Label your production VMs with tier: production in their CRD metadata and run this check every 2 minutes.

Step 4: Monitor VM Network Connectivity

Virtink VMs use virtio-net for networking. Even if the VM is in Running state, it may have lost network connectivity due to a bridge misconfiguration, CNI plugin issue, or virtio-net driver problem.

Add a TCP port monitor for each production VM's primary service port:

  1. In Vigilmon, click Add Monitor → TCP Port.
  2. Enter the VM IP and port: VM_IP:22 (SSH) or VM_IP:80 (web).
  3. Set Check interval to 1 minute.
  4. Set Alert after 2 consecutive failures.
  5. Click Save.

For VMs accessed through a Kubernetes Service, monitor the Service's ClusterIP or LoadBalancer IP instead of the individual VM IP — this remains stable across VM restarts and migrations.

Step 5: Monitor Live Migration Health

Virtink supports live migration of VMs between nodes via VirtualMachineMigration CRDs. Failed migrations prevent node maintenance and can leave VMs in an undefined state.

Deploy a migration health checker:

#!/bin/bash
# Check for stuck or failed migrations
FAILED=$(kubectl get virtualmachinemigrationsources -A \
  -o jsonpath='{range .items[?(@.status.phase=="Failed")]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}')

# Check for migrations stuck in Running > 30 minutes
STUCK=$(kubectl get virtualmachinemigrationsources -A \
  -o json | python3 -c "
import sys, json
from datetime import datetime, timezone, timedelta
data = json.load(sys.stdin)
now = datetime.now(timezone.utc)
for item in data['items']:
    phase = item.get('status', {}).get('phase', '')
    if phase != 'Running':
        continue
    start = item.get('metadata', {}).get('creationTimestamp', '')
    if start:
        started = datetime.fromisoformat(start.replace('Z', '+00:00'))
        if now - started > timedelta(minutes=30):
            print(item['metadata']['namespace'] + '/' + item['metadata']['name'])
")

if [ -n "$FAILED" ] || [ -n "$STUCK" ]; then
  echo "Migration issues - Failed: $FAILED Stuck: $STUCK"
  exit 1
fi
curl -s https://vigilmon.online/heartbeat/YOUR_MIGRATION_HB_TOKEN

Set the heartbeat interval to 10 minutes. A failed or stuck migration triggers an alert.

Step 6: Monitor VM Disk (PVC) Health

Virtink VMs use PersistentVolumeClaims for disk storage. An unbound PVC prevents VM scheduling and startup.

Check all VM-associated PVCs are bound:

#!/bin/bash
# Find unbound PVCs used by VirtualMachine CRDs
UNBOUND=$(kubectl get pvc -A \
  -l app.kubernetes.io/managed-by=virtink \
  --no-headers \
  | grep -v "Bound" \
  | awk '{print $1"/"$2}')

if [ -n "$UNBOUND" ]; then
  echo "Unbound VM PVCs: $UNBOUND"
  exit 1
fi
curl -s https://vigilmon.online/heartbeat/YOUR_PVC_HB_TOKEN

Alternatively, query PVCs referenced in VM specs:

#!/bin/bash
# Extract PVC names from VirtualMachine specs and check binding
kubectl get virtualmachines -A -o json | python3 -c "
import sys, json, subprocess
data = json.load(sys.stdin)
pvc_names = []
for vm in data['items']:
    ns = vm['metadata']['namespace']
    for disk in vm.get('spec', {}).get('instance', {}).get('disks', []):
        pvc = disk.get('persistentVolumeClaim', {}).get('claimName')
        if pvc:
            pvc_names.append((ns, pvc))
for ns, pvc in pvc_names:
    result = subprocess.run(['kubectl', 'get', 'pvc', pvc, '-n', ns,
        '-o', 'jsonpath={.status.phase}'], capture_output=True, text=True)
    if result.stdout.strip() != 'Bound':
        print(f'UNBOUND: {ns}/{pvc}')
        sys.exit(1)
"
curl -s https://vigilmon.online/heartbeat/YOUR_PVC_HB_TOKEN

Step 7: Monitor the Virtink Webhook

Virtink installs a ValidatingWebhookConfiguration for VirtualMachine CRDs. If the webhook times out, kubectl apply of VM manifests fails with a timeout error, blocking all VM creation.

Monitor the webhook endpoint directly:

#!/bin/bash
# The webhook is served by the virtink-controller-manager on port 9443
WEBHOOK_RESPONSE=$(curl -sk -o /dev/null -w "%{http_code}" \
  --max-time 5 \
  https://virtink-controller-manager-webhook.virtink-system.svc:9443/validate-vm.virtink.smartx.com-v1alpha1-virtualmachine)

# Webhook returns 200 or 400 (missing body) — both mean it's responding
if [ "$WEBHOOK_RESPONSE" = "000" ]; then
  echo "Webhook timeout or unreachable"
  exit 1
fi
curl -s https://vigilmon.online/heartbeat/YOUR_WEBHOOK_HB_TOKEN

Set the heartbeat interval to 5 minutes. A webhook timeout blocks all VM creation.

Step 8: Configure Alert Channels and Thresholds

  1. Go to Alert Channels in Vigilmon and add Slack, email, or PagerDuty.
  2. Set priority by monitor:
    • Controller heartbeat: alert immediately on expiry — no controller means no VM management.
    • Daemon heartbeat: alert immediately — missing daemon blocks VM scheduling.
    • VM power state: alert after 1 heartbeat expiry for production VMs.
    • VM network (TCP): alert after 2 consecutive failures.
    • Migration health: alert on heartbeat expiry.
    • PVC health: alert immediately on unbound PVC.
  3. Create a Maintenance Window in Vigilmon before planned live migrations or cluster upgrades:
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"monitor_id": "VM_POWER_MONITOR_ID", "duration_minutes": 20}'

Summary

| Monitor | Target | What It Catches | |---|---|---| | Controller heartbeat | CronJob → k8s API | Controller crash, VM lifecycle blocked | | Daemon heartbeat | CronJob → k8s API | Missing daemon pod on any node | | VM power state | CronJob or HTTP endpoint | Production VM unexpected power-off | | VM network (TCP) | VM IP:port | Network connectivity loss | | Migration heartbeat | CronJob → k8s API | Failed or stuck live migration | | PVC health | CronJob → k8s API | Unbound disk blocking VM | | Webhook heartbeat | CronJob → webhook endpoint | VM creation blocked by webhook timeout |

Virtink's strength is running VMs as first-class Kubernetes objects with minimal overhead — but Kubernetes doesn't alert you when a controller crashes or a VM silently powers off. With Vigilmon watching each layer of the Virtink stack, from the controller and daemon to individual VM network reachability, you get the operational visibility needed to run production workloads on a lightweight VM platform.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles