How to Monitor Koordinator with Vigilmon

Koordinator is an open source QoS-aware co-location workload scheduling system for Kubernetes developed by Alibaba Cloud and donated to the CNCF as a Sandbox project. It enables efficient co-location of latency-sensitive online workloads and best-effort offline/batch workloads on the same nodes, increasing cluster resource utilization beyond what vanilla Kubernetes scheduling can achieve. The trade-off: when Koordinator's components fail silently, latency-sensitive workloads are no longer protected from batch job resource contention, and the utilization gains can turn into latency degradation.

This tutorial shows you how to monitor Koordinator's critical health surfaces with Vigilmon so you detect scheduler failures, Koordlet agent gaps, and co-location policy violations before they cause user-visible latency spikes.

Why Koordinator needs external monitoring

Koordinator runs two layers of components: the koordinator-manager controlling co-location policies cluster-wide, and the koordlet DaemonSet enforcing QoS per node. A failure in either layer can be silent — pods keep running, but the protection guarantees that make co-location safe are gone.

External monitoring catches what Kubernetes internal probes miss:

• Koordinator Manager crash — no co-location policy reconciliation; new pods are scheduled without QoS class annotations; LS workloads lose eviction protection
• Koordlet missing on a node — the DaemonSet agent is absent from a node; CPU suppression and memory reclaim are disabled there; any BE workload on that node can saturate resources unchecked
• LS workload latency degradation — P99 latency for latency-sensitive workloads climbs as BE workload contention increases; this is the primary SLO violation Koordinator is designed to prevent
• Excessive BE eviction rate — Koordinator is evicting BE workloads at an unusually high rate, indicating the co-location ratio is too aggressive and needs tuning
• Webhook failure — Koordinator's admission webhook injects QoS annotations at pod creation time; if the webhook times out, pods are created without their QoS class, defeating co-location guarantees
• Gang scheduling timeout — batch jobs using gang scheduling start failing to launch as a complete group, causing partial job starts and resource waste

What you'll need

• A Kubernetes cluster with Koordinator installed
• kubectl access with permission to read Koordinator resources
• A free Vigilmon account

Step 1: Expose the Koordinator Manager health endpoint

Koordinator Manager exposes a health endpoint. Expose it for external monitoring:

# Verify Koordinator Manager is running
kubectl get deployment koordinator-manager -n koordinator-system

# Check what ports are exposed
kubectl describe deployment koordinator-manager -n koordinator-system | grep -A10 Ports

# Expose health port externally
kubectl expose deployment koordinator-manager \
  --name=koordinator-manager-health \
  --type=NodePort \
  --port=8080 \
  --target-port=8080 \
  -n koordinator-system

Or route through an Ingress:

# koordinator-health-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: koordinator-health
  namespace: koordinator-system
spec:
  rules:
    - host: koordinator.internal.example.com
      http:
        paths:
          - path: /healthz
            pathType: Prefix
            backend:
              service:
                name: koordinator-manager
                port:
                  number: 8080

Verify it responds:

curl http://koordinator.internal.example.com/healthz
# ok

Step 2: Monitor Koordinator Manager health

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. URL: http://koordinator.internal.example.com/healthz
4. Check interval: 1 minute
5. Expected response: status code 200
6. Name: koordinator-manager health
7. Save

When the manager crashes, Vigilmon fires an alert within minutes. Without Koordinator Manager, new pods will still be scheduled by the native Kubernetes scheduler but without co-location QoS class annotations — your BE workloads will no longer respect LS resource priorities.

Step 3: Monitor Koordlet DaemonSet coverage

Koordlet is the per-node agent that actually enforces CPU suppression, memory reclaim, and eviction policies. If koordlet is missing from even one node, that node becomes unsafe for co-location — BE workloads on it can consume all CPU with no suppression.

Set up a Vigilmon heartbeat driven by a CronJob that checks DaemonSet coverage:

# koordlet-coverage-check.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: koordlet-coverage-check
  namespace: koordinator-system
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: koordinator-health-checker
          restartPolicy: OnFailure
          containers:
            - name: checker
              image: bitnami/kubectl:latest
              env:
                - name: VIGILMON_HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: koordlet-coverage-heartbeat
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  DESIRED=$(kubectl get daemonset koordlet -n koordinator-system \
                    -o jsonpath='{.status.desiredNumberScheduled}')
                  READY=$(kubectl get daemonset koordlet -n koordinator-system \
                    -o jsonpath='{.status.numberReady}')
                  if [ "$READY" -lt "$DESIRED" ]; then
                    echo "ERROR: Koordlet coverage $READY/$DESIRED nodes"
                    exit 1
                  fi
                  curl -fsS "$VIGILMON_HEARTBEAT_URL" > /dev/null
                  echo "Koordlet coverage $READY/$DESIRED. Heartbeat sent."

Configure the heartbeat in Vigilmon:

1. Go to Monitors → New Monitor → Heartbeat
2. Name: koordlet-daemonset-coverage
3. Expected interval: 5 minutes
4. Grace period: 3 minutes
5. Save and use the resulting heartbeat URL in your Secret

Check which nodes are missing koordlet:

# Nodes with koordlet pods
kubectl get pods -n koordinator-system -l app=koordlet \
  -o wide --no-headers | awk '{print $7}'

# All nodes
kubectl get nodes --no-headers | awk '{print $1}'

# Find the difference
comm -23 \
  <(kubectl get nodes --no-headers | awk '{print $1}' | sort) \
  <(kubectl get pods -n koordinator-system -l app=koordlet \
    -o wide --no-headers | awk '{print $7}' | sort)

Step 4: Monitor LS workload latency

Koordinator's core value proposition is protecting latency-sensitive (LS) workloads from BE workload interference. If you're not monitoring P99 latency for your LS workloads, you can't know whether the co-location guarantee is holding.

Expose your LS workloads' HTTP health or metrics endpoints and add Vigilmon response time thresholds:

1. For each critical LS workload, go to Monitors → New Monitor → HTTP / HTTPS
2. Set the URL to the workload's health or API endpoint
3. Under Response time threshold, set an alert if response time exceeds your SLO (e.g., 500ms for P50 proxy, 2000ms for P99 budget)
4. Name the monitors with the [LS] prefix: [LS] payment-api /health

When Koordinator's co-location enforcement degrades — koordlet crash, CPU suppression failure — you'll see response times climb on LS monitors before any other signal fires.

Step 5: Monitor the Koordinator admission webhook

Koordinator injects QoS class annotations via an admission webhook. If the webhook pod or service becomes unavailable, pod creation proceeds but with no QoS annotation — the pod is scheduled as a native Kubernetes pod with no Koordinator co-location guarantees.

# Verify the webhook is registered
kubectl get mutatingwebhookconfigurations | grep koordinator
kubectl describe mutatingwebhookconfiguration koordinator-admission

# Check webhook service
kubectl get svc -n koordinator-system | grep webhook

Add a TCP monitor for the webhook service:

1. In Vigilmon, go to Monitors → New Monitor → TCP Port
2. Host: your node IP or webhook service NodePort host
3. Port: 9443 (or the webhook port from your deployment)
4. Name: koordinator-webhook
5. Check interval: 1 minute

Step 6: Monitor gang scheduling health

Koordinator supports gang scheduling (all-or-nothing pod group scheduling) for batch jobs. Gang scheduling failures mean batch jobs either never start or start partially, wasting cluster resources and causing job failures.

Set up a heartbeat CronJob that submits a test gang-scheduled job and reports success:

# gang-scheduling-probe.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: gang-scheduling-probe
  namespace: koordinator-system
spec:
  schedule: "*/15 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: probe
              image: curlimages/curl:latest
              env:
                - name: VIGILMON_HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: gang-scheduling-heartbeat
              command:
                - /bin/sh
                - -c
                - |
                  # Check PodGroup objects for failures
                  FAILED=$(kubectl get podgroups -A -o json 2>/dev/null | \
                    jq '[.items[] | select(.status.phase == "Failed")] | length' || echo 0)
                  if [ "$FAILED" -gt 0 ]; then
                    echo "WARNING: $FAILED failed PodGroups"
                    exit 1
                  fi
                  curl -fsS "$VIGILMON_HEARTBEAT_URL" > /dev/null

Configure a 15-minute heartbeat in Vigilmon with a 10-minute grace period.

Step 7: Monitor node resource utilization targets

Koordinator's goal is to push node utilization above the native Kubernetes ceiling. If utilization is consistently below 60%, co-location policies are not working effectively — either BE workloads are being over-evicted or the resource allocation configuration is too conservative.

Set up a Vigilmon heartbeat that checks cluster-wide CPU utilization and alerts when it's unexpectedly low (indicating the co-location policy has become too restrictive):

# Check average node CPU utilization via metrics-server
kubectl top nodes --no-headers | \
  awk '{gsub(/%/,"",$3); sum+=$3; count++} END {print sum/count}'

Use this in a CronJob that pings the heartbeat only when utilization is within the expected range, letting the heartbeat timeout fire an alert if utilization falls too low.

Step 8: Configure alert channels

Co-location failures fall into two severity tiers:

Critical (page immediately):

• Koordinator Manager crash (koordinator-manager health HTTP monitor)
• Koordlet missing on a node (koordlet-daemonset-coverage heartbeat)
• Koordinator webhook down (koordinator-webhook TCP monitor)

Warning (notify, don't page):

• LS workload latency spike ([LS] HTTP monitors with response time thresholds)
• Gang scheduling failures (gang-scheduling-probe heartbeat)

Set up two alert channels in Vigilmon:

1. PagerDuty channel — assign to critical monitors
2. Slack #platform-eng channel — assign to all monitors

Go to Alert Channels → New Channel for each, then assign them to the appropriate monitors.

Step 9: Create a co-location health status page

1. In Vigilmon, go to Status Pages → New Status Page
2. Name: "Koordinator Co-location Health"
3. Add all Koordinator monitors grouped by layer:
   • Control Plane: koordinator-manager health, koordinator-webhook
   • Node Layer: koordlet-daemonset-coverage
   • Workload SLOs: [LS] monitors
   • Batch Scheduling: gang-scheduling-probe
4. Share with your platform and SRE teams

Summary

| Monitor | Type | What it catches | |---|---|---| | koordinator-manager /healthz | HTTP | Manager crash, no policy reconciliation | | koordinator-webhook :9443 | TCP | Webhook down, pods created without QoS | | koordlet-daemonset-coverage | Heartbeat | Koordlet missing on node(s) | | [LS] <workload> /health | HTTP | LS latency SLO violations from BE contention | | gang-scheduling-probe | Heartbeat | Gang scheduling failures | | node-utilization-check | Heartbeat | Co-location underutilization |

Koordinator's value is only realized if its enforcement layer is always running. One missing koordlet, one crashed manager, one timed-out webhook — and your co-location guarantees silently disappear while the cluster looks healthy from the outside.

Get started free at vigilmon.online — no credit card required, first monitor running in under two minutes.