Monitoring Pyrra SLO with Vigilmon

Pyrra makes it straightforward to define and track Service Level Objectives using Prometheus metrics, generating the multi-window, multi-burn-rate alerting rules described in the Google SRE Workbook without requiring manual PromQL expertise. It's the layer that turns raw Prometheus metrics into actionable SLO compliance data and error budget tracking.

But who monitors the monitor? If Pyrra's API server crashes, your SLO dashboards go dark. If its Prometheus recording rules stop syncing, the error budget calculations silently drift from reality. Vigilmon gives you an independent external check on Pyrra itself — so you know when the SLO management layer is healthy, not just whether your services are meeting their objectives.

What You'll Set Up

Pyrra API server HTTP health check
SLO compliance status endpoint
Error budget burn rate health check
Recording rule sync freshness monitor
Prometheus connectivity health check
Cron heartbeat for Pyrra's background reconciliation
Alert routing to Slack or PagerDuty

Prerequisites

Pyrra 0.6+ deployed (file-based mode or Kubernetes mode)
Prometheus running and scraping your services
At least one SLO defined in Pyrra
A free Vigilmon account

Step 1: Monitor the Pyrra API Server

The Pyrra API server is the core process that reads SLO definitions, generates recording rules, and serves the web UI. If it crashes, SLO tracking stops.

Pyrra exposes a health endpoint at /api/v1/objectives — a successful response confirms the server is running and can read SLO definitions:

Log in to vigilmon.online and click Add Monitor.
Set Type to HTTP / HTTPS.
Enter https://pyrra.yourdomain.com/api/v1/objectives.
Set Check interval to 1 minute.
Set Expected HTTP status to 200.
Enable Monitor SSL certificate and alert when less than 21 days remain.
Click Save.

If Pyrra is running without TLS (common in internal Kubernetes deployments), use the cluster-internal URL and monitor via a Vigilmon agent or internal probe:

http://pyrra.monitoring.svc.cluster.local:9099/api/v1/objectives

In Kubernetes mode, you can also monitor the /healthz endpoint if exposed.

Step 2: Monitor SLO Compliance Status

Pyrra's primary value is tracking whether each SLO is being met. Create a health endpoint that queries Pyrra's API and alerts when any SLO is breached:

#!/usr/bin/env python3
# /usr/local/bin/pyrra-slo-health (served via HTTP)
import urllib.request, json, sys, os

PYRRA_URL = os.environ.get('PYRRA_URL', 'http://localhost:9099')

try:
    with urllib.request.urlopen(f'{PYRRA_URL}/api/v1/objectives', timeout=10) as resp:
        objectives = json.loads(resp.read())

    breached = [o for o in objectives if o.get('status', {}).get('availability', 1) < o.get('target', 0.999)]

    if breached:
        print('Status: 503 Service Unavailable\r\nContent-Type: application/json\r\n')
        print(json.dumps({'slos_breached': len(breached), 'total': len(objectives),
                          'breached_names': [b.get('name','?') for b in breached]}))
    else:
        print('Status: 200 OK\r\nContent-Type: application/json\r\n')
        print(json.dumps({'slos_breached': 0, 'total': len(objectives), 'status': 'all_met'}))
except Exception as e:
    print('Status: 503 Service Unavailable\r\nContent-Type: application/json\r\n')
    print(json.dumps({'error': str(e)}))

Monitor this endpoint every 2 minutes. A 503 means at least one SLO is breached — an immediate alert is appropriate since SLO breaches represent degraded user experience.

Step 3: Monitor Error Budget Burn Rate

High error budget burn rate is the leading indicator that you'll breach an SLO before the window closes. Pyrra's multi-burn-rate alerts handle this in Prometheus, but you can also expose a burn rate health endpoint for Vigilmon:

#!/usr/bin/env python3
# /usr/local/bin/pyrra-burn-rate-health
import urllib.request, json, os

PYRRA_URL = os.environ.get('PYRRA_URL', 'http://localhost:9099')
FAST_BURN_THRESHOLD = 14.4   # 1-hour window — >14.4x burns budget in 1 day
SLOW_BURN_THRESHOLD = 6.0    # 6-hour window — >6x burns budget in ~4 days

try:
    with urllib.request.urlopen(f'{PYRRA_URL}/api/v1/objectives', timeout=10) as resp:
        objectives = json.loads(resp.read())

    # Check burn rates from Pyrra status fields
    critical = []
    for obj in objectives:
        burn = obj.get('status', {}).get('budget', {}).get('remaining', 1.0)
        if burn < 0.10:  # Less than 10% budget remaining
            critical.append({'name': obj.get('name', '?'), 'budget_remaining_pct': round(burn * 100, 1)})

    if critical:
        print('Status: 503 Service Unavailable\r\nContent-Type: application/json\r\n')
        print(json.dumps({'critical_slos': critical, 'count': len(critical)}))
    else:
        print('Status: 200 OK\r\nContent-Type: application/json\r\n')
        print(json.dumps({'status': 'ok', 'critical_slos': []}))
except Exception as e:
    print('Status: 503 Service Unavailable\r\nContent-Type: application/json\r\n')
    print(json.dumps({'error': str(e)}))

Monitor every 5 minutes. Error budget depletion is urgent but not instantaneous — a 5-minute check interval is appropriate.

Step 4: Monitor Recording Rule Sync Freshness

Pyrra generates Prometheus recording rules that pre-compute SLI metrics. In file-based mode, it writes rules to a directory; in Kubernetes mode, it creates PrometheusRule custom resources. If rule sync fails, Pyrra's calculations use stale data.

For file-based mode, check that rule files were recently written:

#!/bin/sh
# /usr/local/bin/pyrra-rules-health (CGI or HTTP wrapper)
RULES_DIR=/etc/prometheus/pyrra
MAX_AGE_MINUTES=10

OLDEST_MOD=$(find "$RULES_DIR" -name "*.yaml" -printf '%T@\n' 2>/dev/null | sort -n | head -1)
NOW=$(date +%s)
AGE_MINUTES=$(( (NOW - ${OLDEST_MOD%.*}) / 60 ))

if [ "$AGE_MINUTES" -gt "$MAX_AGE_MINUTES" ]; then
    printf 'Status: 503 Service Unavailable\r\nContent-Type: application/json\r\n\r\n'
    printf '{"rule_sync":"stale","oldest_file_age_minutes":%d}\n' "$AGE_MINUTES"
else
    printf 'Status: 200 OK\r\nContent-Type: application/json\r\n\r\n'
    printf '{"rule_sync":"ok","oldest_file_age_minutes":%d}\n' "$AGE_MINUTES"
fi

Monitor this every 5 minutes. If Pyrra crashes mid-reconciliation and rules haven't been written in 10+ minutes, something is wrong.

For Kubernetes mode, monitor the Pyrra operator's /metrics endpoint for reconciliation error counters:

http://pyrra-kubernetes.monitoring.svc.cluster.local:9443/metrics

Use Vigilmon's keyword check to alert if the response contains reconcile_errors_total{...} [^0].

Step 5: Monitor Prometheus Connectivity

Pyrra queries Prometheus to populate its SLO status dashboard. If Prometheus becomes unreachable, the Pyrra UI shows stale data with no warning. Add a health endpoint that tests the Prometheus connection from Pyrra's perspective:

#!/usr/bin/env python3
# /usr/local/bin/pyrra-prometheus-health
import urllib.request, json, os

PROMETHEUS_URL = os.environ.get('PROMETHEUS_URL', 'http://prometheus:9090')

try:
    with urllib.request.urlopen(f'{PROMETHEUS_URL}/-/healthy', timeout=5) as resp:
        if resp.status == 200:
            print('Status: 200 OK\r\nContent-Type: application/json\r\n')
            print(json.dumps({'prometheus': 'reachable'}))
        else:
            raise Exception(f'Prometheus returned {resp.status}')
except Exception as e:
    print('Status: 503 Service Unavailable\r\nContent-Type: application/json\r\n')
    print(json.dumps({'prometheus': 'unreachable', 'error': str(e)}))

Monitor every 1 minute. Prometheus unreachability causes Pyrra's SLO calculations to go stale — an immediate alert lets you investigate before dashboards become meaningless.

Step 6: Add a Cron Heartbeat for Pyrra Reconciliation

In file-based mode, Pyrra's reconciliation loop runs on an interval. Confirm it's running by sending a heartbeat after each successful reconciliation cycle. Add this to a sidecar or wrapper script:

#!/bin/bash
# /usr/local/bin/pyrra-heartbeat-wrapper
# Start Pyrra and emit a heartbeat after each rule write

# Watch for rule file changes and ping Vigilmon
inotifywait -m -e close_write /etc/prometheus/pyrra/ 2>/dev/null | while read -r; do
    curl -s --max-time 10 https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_ID
done

In Vigilmon:

Click Add Monitor → Cron Heartbeat.
Set the expected interval to 15 minutes (Pyrra's default reconciliation period is around 10 minutes; 15 gives headroom).
Copy the heartbeat URL into the script above.

If Pyrra's reconciliation stalls, the heartbeat stops, and Vigilmon alerts.

Step 7: Configure Alert Channels and Priorities

Go to Alert Channels in Vigilmon and add Slack and PagerDuty (or your on-call tool).
Route SLO breach alerts (Step 2) and error budget alerts (Step 3) to PagerDuty — these represent real user impact.
Route Pyrra API server health (Step 1) and recording rule sync (Step 4) to Slack initially, escalating to PagerDuty after 5 minutes if unacknowledged.
For the Prometheus connectivity monitor (Step 5), set Consecutive failures before alert to 2 — a single Prometheus probe failure can be transient.
Add maintenance windows in Vigilmon during Prometheus upgrades or Pyrra version updates to suppress expected gaps.

Summary

| Monitor | Target | What It Catches | |---|---|---| | Pyrra API server | /api/v1/objectives | Pyrra crash — SLO tracking stops | | SLO compliance | Custom /slo-health endpoint | Any SLO currently breached | | Error budget | Custom /burn-rate-health endpoint | Error budget <10% remaining | | Recording rule sync | Rule file freshness check | Stale SLI calculations | | Prometheus connectivity | Prometheus /-/healthy | Pyrra dashboards going stale | | Cron heartbeat | Vigilmon heartbeat URL | Pyrra reconciliation loop stalled |

Pyrra is your reliability accounting layer — it tells you how your services are performing against their SLOs. But that layer needs its own reliability monitoring. With Vigilmon watching Pyrra's health, SLO compliance, error budget burn rates, and recording rule freshness, you have an independent external check that ensures your SLO management infrastructure is as reliable as the services it's tracking.