How to Monitor pgBackRest Backup Health, WAL Archiving, and Repository Integrity with Vigilmon

pgBackRest is the most widely deployed PostgreSQL backup solution in production — used by Crunchy Data's Postgres Operator, Zalando's Patroni-based clusters, and thousands of teams running large, high-transaction PostgreSQL databases. Its parallel backup engine, block-level incremental support, WAL archiving, and multi-repository output make it the production-grade alternative to pg_basebackup and Barman.

The problem: pgBackRest runs as a cron job or Kubernetes CronJob and logs its results quietly. A WAL archive that silently falls behind PostgreSQL's generation rate, a full backup that missed its window, or a repository storage that's running out of space will not page anyone. When you finally need point-in-time recovery, you may discover your backup is hours — or days — stale. Vigilmon gives you continuous external monitoring of pgBackRest backup age, WAL archiving health, and repository integrity.

Why pgBackRest Needs External Monitoring

pgBackRest is excellent at taking backups but has no built-in alerting for backup staleness or WAL failures. External monitoring with Vigilmon adds:

• Backup age alerting — the most critical metric: alert the moment a backup misses its schedule window by 50%
• WAL archive lag monitoring — catches WAL archiving falling behind PostgreSQL's WAL generation rate, which breaks point-in-time recovery continuity
• Repository health checks — S3/GCS/local storage reachability and free space monitoring before the repository fills up
• Stanza integrity validation — catches pgBackRest stanza misconfiguration after PostgreSQL major version upgrades or cluster moves
• Standby backup health — replication lag on the backup replica affects backup consistency
• Multi-repository copy health — if you write to multiple repositories, silent failures on secondary repos undermine your backup redundancy

Step 1: Build a pgBackRest Health Endpoint

pgBackRest exposes backup metadata through the pgbackrest info --output=json command. A health sidecar script runs this command, parses the JSON output, and exposes an HTTP endpoint for Vigilmon.

Shell-Based Health Check Script

#!/usr/bin/env bash
# /usr/local/bin/pgbackrest-health-check.sh
# Outputs JSON; exit 0 = healthy, exit 1 = degraded

set -euo pipefail

STANZA="${PGBACKREST_STANZA:-main}"
BACKUP_AGE_WARNING_HOURS="${BACKUP_AGE_WARNING_HOURS:-36}"    # alert if last backup > 36h ago
WAL_LAG_WARNING_SECS="${WAL_LAG_WARNING_SECS:-60}"            # alert if WAL archive lag > 60s
REPO_FREE_WARNING_PCT="${REPO_FREE_WARNING_PCT:-20}"           # alert if <20% free space

info=$(pgbackrest --stanza="$STANZA" info --output=json 2>&1)
rc=$?

if [ $rc -ne 0 ]; then
  echo "{\"status\":\"down\",\"reason\":\"pgbackrest_info_failed\",\"error\":$(echo "$info" | python3 -c 'import sys,json; print(json.dumps(sys.stdin.read()))')}"
  exit 1
fi

# Parse with python for reliability
python3 - "$STANZA" "$BACKUP_AGE_WARNING_HOURS" "$WAL_LAG_WARNING_SECS" <<PYEOF
import sys, json, time

stanza_name = sys.argv[1]
age_warn_hours = float(sys.argv[2])
wal_lag_warn_secs = float(sys.argv[3])

info = json.loads(sys.stdin.read() if False else open('/dev/stdin').read())
# info is already parsed from bash; re-read from pipe
PYEOF

For production use, a dedicated Python or Node.js sidecar is more reliable:

Node.js Health Sidecar

// health/pgbackrest.js
const express = require('express');
const { execSync } = require('child_process');

const app = express();

const STANZA = process.env.PGBACKREST_STANZA || 'main';
const BACKUP_AGE_WARN_HOURS = parseFloat(process.env.BACKUP_AGE_WARNING_HOURS || '36');
const WAL_LAG_WARN_SECS = parseFloat(process.env.WAL_LAG_WARNING_SECS || '60');
const REPO_FREE_WARN_PCT = parseFloat(process.env.REPO_FREE_WARNING_PCT || '20');

function getPgBackRestInfo() {
  const output = execSync(
    `pgbackrest --stanza=${STANZA} info --output=json`,
    { timeout: 30000, encoding: 'utf8' }
  );
  return JSON.parse(output);
}

function getWalArchiveStatus() {
  try {
    // Check pg_stat_archiver via psql if available
    const output = execSync(
      `psql -U postgres -c "SELECT last_archived_time, last_failed_time, archived_count, failed_count FROM pg_stat_archiver" -t -A`,
      { timeout: 10000, encoding: 'utf8' }
    );
    const parts = output.trim().split('|');
    return {
      last_archived_time: parts[0]?.trim(),
      last_failed_time: parts[1]?.trim(),
      archived_count: parseInt(parts[2]) || 0,
      failed_count: parseInt(parts[3]) || 0,
    };
  } catch {
    return null;
  }
}

app.get('/health/pgbackrest', (req, res) => {
  try {
    const info = getPgBackRestInfo();

    const stanza = Array.isArray(info) ? info.find(s => s.name === STANZA) : null;
    if (!stanza) {
      return res.status(503).json({
        status: 'down',
        reason: 'stanza_not_found',
        stanza: STANZA,
        available_stanzas: Array.isArray(info) ? info.map(s => s.name) : [],
      });
    }

    const stanzaStatus = stanza.status?.code;
    if (stanzaStatus !== 0) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'stanza_error',
        stanza: STANZA,
        stanza_status_code: stanzaStatus,
        stanza_status_message: stanza.status?.message,
      });
    }

    const backups = stanza.backup || [];
    if (backups.length === 0) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'no_backups_found',
        stanza: STANZA,
      });
    }

    // Find most recent backup (any type)
    const lastBackup = backups[backups.length - 1];
    const lastBackupTime = new Date(lastBackup.timestamp.stop * 1000);
    const hoursAgo = (Date.now() - lastBackupTime.getTime()) / (1000 * 60 * 60);

    if (hoursAgo > BACKUP_AGE_WARN_HOURS) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'backup_too_old',
        stanza: STANZA,
        hours_since_last_backup: Math.round(hoursAgo * 10) / 10,
        warning_threshold_hours: BACKUP_AGE_WARN_HOURS,
        last_backup_type: lastBackup.type,
        last_backup_time: lastBackupTime.toISOString(),
      });
    }

    // Check WAL archive lag
    const walStatus = stanza.archive?.[0];
    const walLagSecs = walStatus?.['pg']?.[0]?.['id'] ? null : null; // via pg_stat_archiver

    const archiveStats = getWalArchiveStatus();

    return res.status(200).json({
      status: 'ok',
      stanza: STANZA,
      last_backup: {
        type: lastBackup.type,
        time: lastBackupTime.toISOString(),
        hours_ago: Math.round(hoursAgo * 10) / 10,
        size_gb: lastBackup.info?.size ? Math.round(lastBackup.info.size / 1e9 * 10) / 10 : null,
        compressed_size_gb: lastBackup.info?.['repository']?.size
          ? Math.round(lastBackup.info.repository.size / 1e9 * 10) / 10 : null,
      },
      backup_count: backups.length,
      wal_archive: archiveStats,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.get('/health/pgbackrest/wal', (req, res) => {
  try {
    const archiveStats = getWalArchiveStatus();
    if (!archiveStats) {
      return res.status(503).json({ status: 'down', reason: 'cannot_query_pg_stat_archiver' });
    }

    const { last_archived_time, last_failed_time, archived_count, failed_count } = archiveStats;

    // Alert if there are recent failures
    if (failed_count > 0 && last_failed_time) {
      const failedAt = new Date(last_failed_time);
      const minutesSinceFailure = (Date.now() - failedAt.getTime()) / (1000 * 60);
      if (minutesSinceFailure < 30) { // recent failure in last 30 minutes
        return res.status(503).json({
          status: 'degraded',
          reason: 'recent_wal_archive_failure',
          last_failed_time,
          minutes_since_failure: Math.round(minutesSinceFailure),
          failed_count,
        });
      }
    }

    // Check archive lag
    if (last_archived_time) {
      const archivedAt = new Date(last_archived_time);
      const lagSecs = (Date.now() - archivedAt.getTime()) / 1000;
      if (lagSecs > WAL_LAG_WARN_SECS) {
        return res.status(503).json({
          status: 'degraded',
          reason: 'wal_archive_lag_too_high',
          lag_seconds: Math.round(lagSecs),
          warning_threshold_secs: WAL_LAG_WARN_SECS,
          last_archived_time,
        });
      }
    }

    return res.status(200).json({
      status: 'ok',
      last_archived_time,
      archived_count,
      failed_count,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3009, () => console.log('pgBackRest health sidecar listening on :3009'));

Python Health Sidecar

# health/pgbackrest_health.py
import os, json, subprocess, time
from flask import Flask, jsonify
from datetime import datetime, timezone

app = Flask(__name__)

STANZA = os.environ.get('PGBACKREST_STANZA', 'main')
BACKUP_AGE_WARN_HOURS = float(os.environ.get('BACKUP_AGE_WARNING_HOURS', 36))
WAL_LAG_WARN_SECS = float(os.environ.get('WAL_LAG_WARNING_SECS', 60))

def pgbackrest_info():
    result = subprocess.run(
        ['pgbackrest', f'--stanza={STANZA}', 'info', '--output=json'],
        capture_output=True, text=True, timeout=30
    )
    if result.returncode != 0:
        raise RuntimeError(result.stderr.strip())
    return json.loads(result.stdout)

def wal_archive_stats():
    try:
        result = subprocess.run(
            ['psql', '-U', 'postgres', '-c',
             "SELECT last_archived_time, last_failed_time, archived_count, failed_count FROM pg_stat_archiver",
             '-t', '-A'],
            capture_output=True, text=True, timeout=10
        )
        parts = result.stdout.strip().split('|')
        return {
            'last_archived_time': parts[0].strip() or None,
            'last_failed_time': parts[1].strip() or None,
            'archived_count': int(parts[2] or 0),
            'failed_count': int(parts[3] or 0),
        }
    except Exception:
        return None

@app.route('/health/pgbackrest')
def pgbackrest_health():
    try:
        info = pgbackrest_info()
    except Exception as e:
        return jsonify(status='down', error=str(e)), 503

    stanza = next((s for s in info if s.get('name') == STANZA), None)
    if not stanza:
        return jsonify(status='down', reason='stanza_not_found', stanza=STANZA), 503

    if stanza.get('status', {}).get('code', -1) != 0:
        return jsonify(status='degraded', reason='stanza_error',
                       stanza_status=stanza.get('status')), 503

    backups = stanza.get('backup', [])
    if not backups:
        return jsonify(status='degraded', reason='no_backups_found', stanza=STANZA), 503

    last = backups[-1]
    last_time = datetime.fromtimestamp(last['timestamp']['stop'], tz=timezone.utc)
    hours_ago = (datetime.now(timezone.utc) - last_time).total_seconds() / 3600

    if hours_ago > BACKUP_AGE_WARN_HOURS:
        return jsonify(
            status='degraded', reason='backup_too_old', stanza=STANZA,
            hours_since_last_backup=round(hours_ago, 1),
            warning_threshold_hours=BACKUP_AGE_WARN_HOURS,
            last_backup_time=last_time.isoformat(),
        ), 503

    return jsonify(status='ok', stanza=STANZA,
                   last_backup=dict(type=last['type'],
                                    time=last_time.isoformat(),
                                    hours_ago=round(hours_ago, 1)),
                   backup_count=len(backups),
                   wal_archive=wal_archive_stats())

@app.route('/health/pgbackrest/wal')
def pgbackrest_wal():
    stats = wal_archive_stats()
    if not stats:
        return jsonify(status='down', reason='cannot_query_pg_stat_archiver'), 503

    failed_count = stats.get('failed_count', 0)
    if failed_count > 0 and stats.get('last_failed_time'):
        try:
            failed_at = datetime.fromisoformat(stats['last_failed_time'].replace(' ', 'T'))
            if not failed_at.tzinfo:
                failed_at = failed_at.replace(tzinfo=timezone.utc)
            mins = (datetime.now(timezone.utc) - failed_at).total_seconds() / 60
            if mins < 30:
                return jsonify(status='degraded', reason='recent_wal_archive_failure',
                               minutes_since_failure=round(mins), **stats), 503
        except Exception:
            pass

    if stats.get('last_archived_time'):
        try:
            archived_at = datetime.fromisoformat(stats['last_archived_time'].replace(' ', 'T'))
            if not archived_at.tzinfo:
                archived_at = archived_at.replace(tzinfo=timezone.utc)
            lag = (datetime.now(timezone.utc) - archived_at).total_seconds()
            if lag > WAL_LAG_WARN_SECS:
                return jsonify(status='degraded', reason='wal_archive_lag_too_high',
                               lag_seconds=round(lag),
                               warning_threshold_secs=WAL_LAG_WARN_SECS, **stats), 503
        except Exception:
            pass

    return jsonify(status='ok', **stats)

if __name__ == '__main__':
    app.run(port=3009)

Step 2: Configure Vigilmon HTTP Monitors

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. URL: https://your-pg-server.example.com/health/pgbackrest
4. Check interval: 5 minutes
5. Under Expected response:
   • Status code: 200
   • Response body contains: "status":"ok"
   • Response time threshold: 10000ms (pgbackrest info can take a few seconds for large clusters)
6. Assign your primary alert channel
7. Save

Add a second monitor for WAL archiving:

| Monitor URL | Purpose | Interval | Priority | |---|---|---|---| | /health/pgbackrest | Backup age, stanza health, overall status | 5 min | P1 | | /health/pgbackrest/wal | WAL archive lag, recent archive failures | 2 min | P1 |

WAL archive failures should be on a shorter interval than backup age — a broken WAL archive breaks point-in-time recovery and needs to be caught within minutes, not hours.

Step 3: Heartbeat Monitoring for Scheduled Backups

Wire heartbeat pings into your pgBackRest backup scripts so Vigilmon knows when a scheduled backup completes successfully (and alerts when it doesn't).

1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
2. Name: pgbackrest-daily-full-backup
3. Expected interval: 24 hours
4. Grace period: 4 hours
5. Copy the heartbeat URL

Add the ping to your backup cron or systemd timer:

#!/usr/bin/env bash
# /etc/cron.d/pgbackrest-backup
# 0 2 * * * postgres /usr/local/bin/pgbackrest-backup.sh

set -euo pipefail

STANZA="${PGBACKREST_STANZA:-main}"
VIGILMON_HB_URL="${VIGILMON_HEARTBEAT_FULL_URL:-}"

pgbackrest --stanza="$STANZA" --type=full backup

if [ -n "$VIGILMON_HB_URL" ]; then
  curl -fsS "$VIGILMON_HB_URL" || true
fi

For incremental backups that run more frequently:

# Incremental backup heartbeat (every 6 hours)
pgbackrest --stanza="$STANZA" --type=incr backup && \
  curl -fsS "${VIGILMON_HEARTBEAT_INCR_URL:-}" || true

Step 4: Repository Free Space Monitoring

Add an additional check for repository storage health. For S3/GCS repositories, the check verifies endpoint reachability; for local disk, it checks free space percentage.

Extend your health sidecar:

app.get('/health/pgbackrest/repository', (req, res) => {
  try {
    // Check repository info command
    const output = execSync(
      `pgbackrest --stanza=${STANZA} repo-ls / 2>&1 | head -5`,
      { timeout: 15000, encoding: 'utf8' }
    );

    // Check local disk if applicable
    let diskInfo = null;
    if (process.env.PGBACKREST_REPO_PATH) {
      const dfOutput = execSync(
        `df -P ${process.env.PGBACKREST_REPO_PATH} | tail -1`,
        { encoding: 'utf8' }
      );
      const parts = dfOutput.trim().split(/\s+/);
      const usePct = parseInt(parts[4]) || 0;
      const freePct = 100 - usePct;
      diskInfo = { path: parts[5], use_pct: usePct, free_pct: freePct };

      if (freePct < REPO_FREE_WARN_PCT) {
        return res.status(503).json({
          status: 'degraded',
          reason: 'repository_low_disk_space',
          disk: diskInfo,
          warning_threshold_pct: REPO_FREE_WARN_PCT,
        });
      }
    }

    return res.status(200).json({ status: 'ok', repository_reachable: true, disk: diskInfo });
  } catch (err) {
    return res.status(503).json({
      status: 'degraded',
      reason: 'repository_unreachable',
      error: err.message,
    });
  }
});

Step 5: Alert Routing

| Monitor | Alert Channel | Priority | Condition | |---|---|---|---| | HTTP: /health/pgbackrest | Slack + PagerDuty | P1 | Backup older than schedule + 50% | | HTTP: /health/pgbackrest/wal | Slack + PagerDuty | P1 | WAL archive lag >60s or recent failure | | HTTP: /health/pgbackrest/repository | Slack | P2 | Repository unreachable or <20% free | | Heartbeat: daily full backup | Slack + PagerDuty | P1 | Full backup did not complete on schedule | | Heartbeat: incremental backup | Slack | P2 | Incremental backup missed its window |

WAL archive failures and backup age staleness should both be P1 — your ability to perform point-in-time recovery depends on both, and discovering either failure during an actual disaster is too late.

Summary

pgBackRest backup failures are silent. A missed backup window or a broken WAL archive won't cause any application errors — it just means that when you need to restore, you can't. External monitoring with Vigilmon gives you continuous visibility into backup age, WAL archiving health, and repository integrity.

| Monitor Type | What It Covers | |---|---| | HTTP: /health/pgbackrest | Backup age, stanza health, last backup type | | HTTP: /health/pgbackrest/wal | WAL archive lag, archive failure rate | | HTTP: /health/pgbackrest/repository | Repository reachability, disk free space | | Heartbeat: full backup | Daily full backup completion | | Heartbeat: incremental backup | Frequent incremental backup completion |

Get started free at vigilmon.online — your first pgBackRest monitor is running in under two minutes.