How to Monitor XCP-ng Hypervisor Health with Vigilmon

XCP-ng is an enterprise-grade open source hypervisor running production VMs for thousands of organizations — but when the XAPI daemon crashes, all VM management on that host stops. When a shared storage repository fills past 85%, VMs start failing to write to disk. When Xen Orchestra goes offline, your team loses all visibility into the pool. None of these failures show up in logs you're already watching, and all of them cause cascading production outages.

Vigilmon gives you external uptime monitoring for XCP-ng host health endpoints, Xen Orchestra availability, storage repository health APIs, and heartbeat monitors for pool-level health checks. This tutorial covers the full stack.

Why XCP-ng Needs External Monitoring

XCP-ng provides extensive internal metrics via XAPI (the XenServer API), Xen Orchestra dashboards, and RRDD (the RRD daemon). But internal tooling has a fundamental blind spot: it only tells you what the XAPI daemon can see. External monitoring adds layers that internal tooling cannot provide:

• XAPI daemon crash detection — if the XAPI service on a host crashes, that host becomes unmanageable, but internal dashboards lose the connection silently
• Xen Orchestra availability — if Xen Orchestra's web service crashes, your operators can't see or manage any VMs in the pool
• Host network reachability — a host could be powered on with XAPI running, but unreachable due to bond degradation or switch failure
• Storage repository saturation — XAPI won't warn you until writes start failing; external health checks catch SR capacity before that point
• Cross-host perspective — a node in the pool monitoring another node catches split-brain and network partition scenarios that intra-host tooling cannot

Vigilmon's multi-region consensus probing means a single transient network blip won't page you at 3am, while genuine XAPI unavailability triggers an alert in seconds.

What you'll need

• One or more XCP-ng hosts (version 8.x or later)
• Xen Orchestra (XO) installed — Community Edition or XO Appliance
• A lightweight HTTP health sidecar (Node.js, Python, or shell script) running on the XCP-ng host or a management VM
• A free Vigilmon account

Step 1: Build an XCP-ng health endpoint

XCP-ng doesn't expose a native HTTP health endpoint, but you can create one that queries XAPI via the xe CLI. Run this as a small service on a management VM in the pool (not on the hypervisor host itself, so it survives a host crash).

Python health sidecar

# xcpng_health.py
from flask import Flask, jsonify
import subprocess
import json

app = Flask(__name__)

def run_xe(cmd):
    result = subprocess.run(
        ['xe'] + cmd,
        capture_output=True, text=True, timeout=10
    )
    return result.returncode == 0, result.stdout.strip(), result.stderr.strip()

@app.route('/health/xcpng')
def xcpng_health():
    # Check XAPI connectivity — this fails if XAPI is down
    ok, out, err = run_xe(['host-list', '--minimal'])
    if not ok:
        return jsonify(status='down', error=err), 503

    hosts = [h.strip() for h in out.split(',') if h.strip()]

    # Check each host's power state
    host_statuses = []
    for host_uuid in hosts:
        ok2, name, _ = run_xe(['host-param-get', f'uuid={host_uuid}', 'param-name=name-label'])
        ok3, enabled, _ = run_xe(['host-param-get', f'uuid={host_uuid}', 'param-name=enabled'])
        host_statuses.append({
            'uuid': host_uuid,
            'name': name if ok2 else 'unknown',
            'enabled': enabled == 'true' if ok3 else False,
        })

    degraded = [h for h in host_statuses if not h['enabled']]

    if degraded:
        return jsonify(
            status='degraded',
            reason='host_disabled',
            degraded_hosts=[h['name'] for h in degraded],
            all_hosts=host_statuses,
        ), 503

    return jsonify(status='ok', host_count=len(hosts), hosts=host_statuses)


@app.route('/health/xcpng/vms')
def vm_health():
    # Count running vs. halted VMs (excluding control domains)
    ok, out, _ = run_xe(['vm-list', '--minimal', 'is-control-domain=false'])
    if not ok:
        return jsonify(status='down', error='cannot reach XAPI'), 503

    all_vm_uuids = [u.strip() for u in out.split(',') if u.strip()]

    halted = []
    for uuid in all_vm_uuids:
        ok2, state, _ = run_xe(['vm-param-get', f'uuid={uuid}', 'param-name=power-state'])
        ok3, name, _ = run_xe(['vm-param-get', f'uuid={uuid}', 'param-name=name-label'])
        if ok2 and state == 'halted':
            halted.append(name if ok3 else uuid)

    return jsonify(
        status='ok',
        total_vms=len(all_vm_uuids),
        halted_vms=halted,
        halted_count=len(halted),
    )


@app.route('/health/xcpng/storage')
def storage_health():
    ok, out, _ = run_xe(['sr-list', '--minimal'])
    if not ok:
        return jsonify(status='down', error='cannot reach XAPI'), 503

    sr_uuids = [u.strip() for u in out.split(',') if u.strip()]
    critical = []
    srs = []

    for uuid in sr_uuids:
        ok_name, name, _ = run_xe(['sr-param-get', f'uuid={uuid}', 'param-name=name-label'])
        ok_used, used, _ = run_xe(['sr-param-get', f'uuid={uuid}', 'param-name=physical-utilisation'])
        ok_size, size, _ = run_xe(['sr-param-get', f'uuid={uuid}', 'param-name=physical-size'])

        if ok_used and ok_size and size.isdigit() and int(size) > 0:
            pct = round(int(used) / int(size) * 100, 1)
            entry = {'name': name if ok_name else uuid, 'used_pct': pct}
            srs.append(entry)
            if pct > 85:
                critical.append(entry)

    if critical:
        return jsonify(status='critical', reason='sr_over_85pct', critical_srs=critical, all_srs=srs), 503

    return jsonify(status='ok', storage_repositories=srs)


if __name__ == '__main__':
    app.run(host='0.0.0.0', port=9080)

Install dependencies and run:

pip install flask
python xcpng_health.py &

Or run it as a systemd unit on your management VM:

# /etc/systemd/system/xcpng-health.service
[Unit]
Description=XCP-ng Health Endpoint

[Service]
ExecStart=/usr/bin/python3 /opt/xcpng-health/xcpng_health.py
Restart=always
User=root

[Install]
WantedBy=multi-user.target

systemctl enable --now xcpng-health

Verify:

curl http://mgmt-vm.local:9080/health/xcpng
# {"status": "ok", "host_count": 3, "hosts": [...]}

curl http://mgmt-vm.local:9080/health/xcpng/storage
# {"status": "ok", "storage_repositories": [{"name": "NFS SR", "used_pct": 62.4}]}

Step 2: Monitor Xen Orchestra availability

Xen Orchestra exposes a web UI on port 80/443. Monitor its HTTP endpoint directly in Vigilmon — no sidecar needed.

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Set the URL to your Xen Orchestra instance: https://xo.internal.example.com
4. Set check interval to 1 minute
5. Set expected status code: 200
6. Save the monitor

If you run XO on a non-standard port:

https://xo.internal.example.com:443

Also add a TCP monitor for port 443 to catch cases where the web server crashes but the host is still up:

1. Monitors → New Monitor → TCP Port
2. Host: xo.internal.example.com
3. Port: 443
4. Save

When Xen Orchestra is unreachable, your team can't live-migrate VMs, can't check pool health, and can't respond to alerts — catching this early is critical.

Step 3: Monitor XCP-ng host XAPI health

Add the sidecar endpoints from Step 1 to Vigilmon:

Host and XAPI health

1. Monitors → New Monitor → HTTP / HTTPS
2. URL: http://mgmt-vm.local:9080/health/xcpng
3. Expected status: 200
4. Check interval: 1 minute
5. Save as "XCP-ng Pool XAPI Health"

VM power state health

1. Monitors → New Monitor → HTTP / HTTPS
2. URL: http://mgmt-vm.local:9080/health/xcpng/vms
3. Expected status: 200
4. Check interval: 2 minutes
5. Save as "XCP-ng VM Power States"

Storage repository health

1. Monitors → New Monitor → HTTP / HTTPS
2. URL: http://mgmt-vm.local:9080/health/xcpng/storage
3. Expected status: 200
4. Check interval: 5 minutes
5. Save as "XCP-ng Storage Repository Health"

Step 4: Monitor individual XCP-ng hosts via TCP

Even if the pool-level XAPI check passes, individual hosts can become network-unreachable. Add TCP monitors for each host:

1. Monitors → New Monitor → TCP Port
2. Host: xcpng-host-01.internal.example.com
3. Port: 22 (SSH — always open on a healthy XCP-ng host)
4. Save as "XCP-ng Host 01 Reachability"

Repeat for each host in your pool. Port 22 is ideal — it's always listening on a healthy XCP-ng host and is the first thing that goes away when a host is unreachable.

You can also monitor XAPI's management port directly:

| Port | Protocol | What it checks | |------|----------|----------------| | 22 | TCP | Host OS reachability | | 80 | TCP | XAPI HTTP management | | 443 | TCP | XAPI HTTPS management |

Step 5: Add a heartbeat monitor for periodic pool checks

For pool-level health checks that run on a schedule (backup verification, HA heartbeat, storage replication checks), use Vigilmon's heartbeat monitor. Your scheduled script pings Vigilmon every N minutes; if it stops reporting in, Vigilmon alerts.

#!/bin/bash
# /opt/xcpng-pool-check.sh — runs every 5 minutes via cron

HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_TOKEN"

# Check HA status
ha_status=$(xe pool-ha-compute-hypothetical-max-host-failures-to-tolerate 2>&1)

# Check XAPI is responding
xe host-list --minimal > /dev/null 2>&1
xapi_ok=$?

if [ $xapi_ok -eq 0 ]; then
    curl -s -X POST "$HEARTBEAT_URL" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"ok\", \"ha_check\": \"$ha_status\"}" \
        > /dev/null
fi

In Vigilmon:

1. Go to Monitors → New Monitor → Heartbeat
2. Set interval: 10 minutes (ping every 5 min; alert if 2 pings missed)
3. Copy the heartbeat URL and paste it into your script
4. Add the script to crontab:

crontab -e
# Add:
*/5 * * * * /opt/xcpng-pool-check.sh

Step 6: Configure alert channels

Email alerts

1. In Vigilmon, go to Alert Channels → Add Channel → Email
2. Enter your infrastructure team's on-call email
3. Assign to all XCP-ng monitors

Webhook alerts for Slack

{
  "monitor_name": "XCP-ng Pool XAPI Health",
  "status": "down",
  "url": "http://mgmt-vm.local:9080/health/xcpng",
  "started_at": "2026-01-15T03:22:00Z",
  "duration_seconds": 45
}

Route this to a Slack webhook to page your virtualization team immediately when XAPI goes dark.

Recommended alert escalation for XCP-ng

| Monitor | Alert within | Escalate to | |---------|-------------|-------------| | Xen Orchestra HTTP | 2 minutes | Operations team | | XCP-ng XAPI Health | 2 minutes | Infrastructure on-call | | Host TCP reachability | 3 minutes | Data center / senior infra | | Storage >85% | 10 minutes | Storage admin | | Heartbeat (pool checks) | 15 minutes | Infrastructure on-call |

Full monitor summary

| Monitor | Type | Endpoint | What it catches | |---------|------|----------|-----------------| | Xen Orchestra | HTTP | https://xo.example.com | XO web UI unavailability | | XAPI Pool Health | HTTP | :9080/health/xcpng | XAPI crash, host disabled | | VM Power States | HTTP | :9080/health/xcpng/vms | Unexpected VM halts | | Storage Repositories | HTTP | :9080/health/xcpng/storage | SR over 85% capacity | | Host 01 SSH | TCP | host-01:22 | Host network reachability | | Host 02 SSH | TCP | host-02:22 | Host network reachability | | Pool Heartbeat | Heartbeat | heartbeat URL | Pool check script failure |

What's next

• SSL certificate monitoring — if Xen Orchestra runs on HTTPS, Vigilmon will alert you before the certificate expires
• Multi-environment status pages — publish a private status page for your infrastructure team showing pool health at a glance
• Heartbeat-based backup verification — use heartbeat monitors to verify that XCP-ng backup scripts (via XAPI export or Xen Orchestra backup jobs) complete on schedule

Get started free at vigilmon.online — no credit card required, monitors live in under a minute.