tutorial

Monitoring SSSD with Vigilmon

SSSD (System Security Services Daemon) is the Linux daemon that authenticates users against Active Directory, LDAP, and FreeIPA. Here's how to monitor SSSD health, identity provider connectivity, authentication success rates, cache performance, and offline mode events with Vigilmon.

SSSD (System Security Services Daemon) is the daemon that handles authentication and identity resolution on enterprise Linux servers — it's how your Linux hosts connect to Active Directory, OpenLDAP, FreeIPA, and Kerberos. When SSSD fails or goes offline, users can't log in (or can only log in with stale cached credentials), sudo lookups fail, and SSH key lookups break. The failure is often silent: the service appears running in systemctl status, but the identity provider backend has silently disconnected. Vigilmon monitors SSSD daemon health, identity provider connectivity, authentication success rates, cache performance, and offline mode events — giving you early warning before users start getting locked out.

What You'll Set Up

  • SSSD daemon process health via cron heartbeat
  • Identity provider connectivity check per configured domain
  • Authentication success rate monitoring
  • NSS lookup latency alert (threshold: >2 seconds)
  • SSSD cache hit rate monitoring (threshold: <70%)
  • SSSD offline mode entry detection
  • Error log rate spike detection
  • Alert channels for system administrators

Prerequisites

  • SSSD installed and configured on one or more Linux hosts (RHEL/CentOS/Fedora/Debian/Ubuntu)
  • sssd service running with at least one configured domain
  • sssctl command available (SSSD 1.14+ for sssctl domain-status)
  • A free Vigilmon account

Step 1: Monitor the SSSD Daemon Process

SSSD runs as a main process plus child responder and provider processes. If the main sssd process crashes, all authentication and NSS lookups fail. Systemd will attempt to restart it, but the restart may fail if the identity provider is unreachable during startup.

Create a heartbeat monitor in Vigilmon:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to Cron / Heartbeat.
  3. Name it SSSD Daemon.
  4. Set Expected interval to 60 seconds.
  5. Set Alert after missing to 2 heartbeats.
  6. Copy the generated heartbeat URL.

Create the cron job:

# /etc/cron.d/sssd-heartbeat
* * * * * root systemctl is-active --quiet sssd && curl -fsS --retry 3 "https://hb.vigilmon.online/YOUR_HB_ID" > /dev/null 2>&1

This only fires the heartbeat if systemd reports the service as active. If SSSD crashes and fails to restart, Vigilmon alerts after 2 minutes.

Step 2: Monitor Identity Provider Connectivity Per Domain

SSSD can be configured with multiple domains (e.g., corp.example.com on AD and app.example.com on FreeIPA). Each domain has an independent backend provider process. A single domain going offline is less obvious than a full SSSD crash — the daemon stays running, but users in that domain can't authenticate against the live directory.

Use sssctl domain-status to check each domain:

#!/bin/bash
# /usr/local/bin/sssd-domain-check.sh

HB_URL="https://hb.vigilmon.online/YOUR_DOMAIN_HB_ID"

# Get list of configured domains
DOMAINS=$(sssctl domain-list 2>/dev/null)
if [ -z "$DOMAINS" ]; then
  # Can't list domains — SSSD may be down
  exit 1
fi

ALL_ONLINE=1
while IFS= read -r DOMAIN; do
  [ -z "$DOMAIN" ] && continue
  STATUS=$(sssctl domain-status "$DOMAIN" 2>/dev/null | grep -i "Online status" | awk '{print $NF}')
  if [ "$STATUS" != "Online" ]; then
    ALL_ONLINE=0
    break
  fi
done <<< "$DOMAINS"

if [ "$ALL_ONLINE" -eq 1 ]; then
  curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
fi

Create a heartbeat monitor SSSD Domains Online with a 60-second interval. Set it to alert after 3 missed heartbeats — a brief network blip may cause one or two offline status checks before SSSD recovers.

Schedule:

* * * * * root /usr/local/bin/sssd-domain-check.sh

Step 3: Monitor Authentication Success Rate

SSSD processes all PAM authentication requests for system logins. A drop in authentication success rate below 95% indicates problems with the Kerberos KDC, LDAP bind failures, or certificate validation errors — even when SSSD itself appears healthy.

#!/bin/bash
# /usr/local/bin/sssd-auth-rate-check.sh
# Parse SSSD PAM log for recent authentication failures

LOG_DIR="/var/log/sssd"
LOOKBACK_MINUTES=5
FAILURE_THRESHOLD_PCT=5  # alert if >5% failure rate (= <95% success)
HB_URL="https://hb.vigilmon.online/YOUR_AUTH_HB_ID"

SINCE=$(date -d "-${LOOKBACK_MINUTES} minutes" "+%Y-%m-%d %H:%M" 2>/dev/null || \
        date -v-${LOOKBACK_MINUTES}M "+%Y-%m-%d %H:%M" 2>/dev/null)

# Count auth attempts and failures in SSSD PAM log
PAM_LOG="${LOG_DIR}/sssd_pam.log"
if [ ! -f "$PAM_LOG" ]; then
  # No PAM log — SSSD may not be logging (not necessarily a failure)
  curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
  exit 0
fi

TOTAL=$(awk -v since="$SINCE" '$0 >= since && /pam_authenticate/' "$PAM_LOG" 2>/dev/null | wc -l)
FAILURES=$(awk -v since="$SINCE" '$0 >= since && /pam_authenticate.*failed/' "$PAM_LOG" 2>/dev/null | wc -l)

if [ "$TOTAL" -eq 0 ]; then
  # No auth activity in window — send heartbeat (idle is not a failure)
  curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
  exit 0
fi

FAILURE_PCT=$(awk "BEGIN {printf \"%d\", ($FAILURES / $TOTAL) * 100}")
if [ "$FAILURE_PCT" -lt "$FAILURE_THRESHOLD_PCT" ]; then
  curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
fi

Create a heartbeat SSSD Auth Success Rate with a 5-minute interval and schedule the script every 5 minutes.

Step 4: Monitor NSS Lookup Latency

SSSD responds to getpwnam and getgrnam lookups — every id username, ls -l, and SSH login triggers NSS lookups. When the SSSD cache is cold or the identity provider is slow, these lookups can take seconds, causing login delays and sudo hangs.

#!/bin/bash
# /usr/local/bin/sssd-nss-latency-check.sh

LATENCY_THRESHOLD_MS=2000
HB_URL="https://hb.vigilmon.online/YOUR_NSS_HB_ID"

# Look up a known user and measure response time
# Use a low-privileged service account or the first domain user you know exists
TEST_USER="${SSSD_TEST_USER:-nobody}"  # override via environment or /etc/sssd-monitor.env

START=$(date +%s%3N)
getent passwd "$TEST_USER" > /dev/null 2>&1
STATUS=$?
END=$(date +%s%3N)

LATENCY=$((END - START))

# getent returning non-zero for a known user is also a failure
if [ "$STATUS" -eq 0 ] && [ "$LATENCY" -lt "$LATENCY_THRESHOLD_MS" ]; then
  curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
fi

Set SSSD_TEST_USER to a domain user account you know exists in your directory (a service account works well). Create a heartbeat SSSD NSS Latency OK with a 60-second interval.

Step 5: Monitor SSSD Cache Hit Rate

SSSD's ldb cache allows authentication even when the identity provider is unreachable. A low cache hit rate (below 70%) means SSSD is repeatedly querying the identity provider for lookups it should be serving from cache — causing higher latency and more load on your AD/LDAP infrastructure.

#!/bin/bash
# /usr/local/bin/sssd-cache-check.sh

MIN_HIT_RATE=70
HB_URL="https://hb.vigilmon.online/YOUR_CACHE_HB_ID"

# Use sssctl to get cache stats if available (SSSD 1.16+)
CACHE_INFO=$(sssctl cache-expire --help 2>&1 | grep -i "expire")

# Parse SSSD debug logs for cache hit/miss counts
# Debug level 6+ logs cache lookups; adjust if you use a different debug level
LOG="/var/log/sssd/sssd_nss.log"
if [ ! -f "$LOG" ]; then
  curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
  exit 0
fi

HITS=$(grep -c "Returning cached" "$LOG" 2>/dev/null || echo 0)
MISSES=$(grep -c "not found in cache\|Cache expired" "$LOG" 2>/dev/null || echo 0)
TOTAL=$((HITS + MISSES))

if [ "$TOTAL" -eq 0 ]; then
  curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
  exit 0
fi

HIT_RATE=$(awk "BEGIN {printf \"%d\", ($HITS / $TOTAL) * 100}")
if [ "$HIT_RATE" -ge "$MIN_HIT_RATE" ]; then
  curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
fi

Create a heartbeat SSSD Cache Hit Rate with a 5-minute interval. Note: this script requires SSSD debug logging at level 6 or higher in sssd.conf. If you're not logging at debug level, check the SSSD log for alternative cache-related messages in your version.

Step 6: Detect SSSD Offline Mode Entry

When SSSD loses connectivity to the identity provider, it enters "offline mode" and starts serving cached credentials. Offline mode entry is not a crash — SSSD keeps running — but it's a critical event: users can only log in with stale cached passwords, and any users whose cache has expired are locked out.

#!/bin/bash
# /usr/local/bin/sssd-offline-check.sh

STATE_FILE="/tmp/sssd_offline_last_check"
HB_URL="https://hb.vigilmon.online/YOUR_OFFLINE_HB_ID"
LOG="/var/log/sssd/sssd.log"

if [ ! -f "$LOG" ]; then
  curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
  exit 0
fi

# Get timestamp of last check
LAST_CHECK=$(cat "$STATE_FILE" 2>/dev/null || echo "1970-01-01 00:00")
date "+%Y-%m-%d %H:%M" > "$STATE_FILE"

# Check if any domain entered offline mode since last check
OFFLINE_EVENTS=$(awk -v since="$LAST_CHECK" '$0 >= since && /Going offline/' "$LOG" 2>/dev/null | wc -l)

if [ "$OFFLINE_EVENTS" -eq 0 ]; then
  # No offline events — send heartbeat
  curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
fi

This check deliberately skips the heartbeat when any offline mode event is detected since the last run. Create a heartbeat SSSD Stays Online with a 5-minute interval. Unlike other monitors, this one alerts on ANY offline event — even one transient offline-then-online cycle within the check window is worth knowing about.

Step 7: Monitor SSSD Error Log Rate

SSSD logs errors to /var/log/sssd/ with separate log files per responder and provider. A spike in error log entries often precedes a full failure — Kerberos ticket renewal errors, LDAP bind timeouts, and TGT acquisition failures all appear here before authentication breaks.

#!/bin/bash
# /usr/local/bin/sssd-error-rate-check.sh

LOG_DIR="/var/log/sssd"
ERROR_THRESHOLD=50  # alert if >50 errors in the past 10 minutes
LOOKBACK_MINUTES=10
HB_URL="https://hb.vigilmon.online/YOUR_ERRLOG_HB_ID"

SINCE=$(date -d "-${LOOKBACK_MINUTES} minutes" "+%Y-%m-%d %H:%M" 2>/dev/null || \
        date -v-${LOOKBACK_MINUTES}M "+%Y-%m-%d %H:%M" 2>/dev/null)

TOTAL_ERRORS=0
for LOG in "${LOG_DIR}"/*.log; do
  [ -f "$LOG" ] || continue
  COUNT=$(awk -v since="$SINCE" '$0 >= since && /\[(ERROR|CRIT|FATAL)\]/' "$LOG" 2>/dev/null | wc -l)
  TOTAL_ERRORS=$((TOTAL_ERRORS + COUNT))
done

if [ "$TOTAL_ERRORS" -lt "$ERROR_THRESHOLD" ]; then
  curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1
fi

Create a heartbeat SSSD Error Rate OK with a 10-minute interval and schedule this script every 10 minutes.

Step 8: Configure Alert Channels

In Vigilmon, navigate to Settings → Alert Channels and configure:

  1. PagerDuty / on-call — for SSSD daemon down and domain offline events. These directly prevent user logins.
  2. Slack — for authentication rate, NSS latency, and error log spikes (leading indicators that need investigation but aren't yet causing outages).
  3. Email — for cache hit rate and offline mode alerts (needs review but not immediate action).

Alert channel assignment:

| Monitor | Channel | |---|---| | SSSD Daemon | PagerDuty + Slack | | SSSD Domains Online | PagerDuty + Slack | | SSSD Auth Success Rate | Slack + Email | | SSSD NSS Latency OK | Slack | | SSSD Cache Hit Rate | Email | | SSSD Stays Online | PagerDuty + Slack | | SSSD Error Rate OK | Slack |

Step 9: Multi-Host Deployment

If you're running SSSD on multiple Linux hosts (common in enterprise environments), create a separate set of heartbeat monitors per host — or at minimum per critical host group. A few patterns:

Per-host monitors: Name monitors with the hostname prefix (web01 SSSD Daemon, db01 SSSD Daemon) for clear attribution in alerts.

Aggregated cron: On hosts that share a jump host, aggregate the check results and send a single heartbeat per group only if all hosts are healthy:

#!/bin/bash
# Run on jump host — check SSSD on all managed servers
SERVERS="web01 web02 db01"
HB_URL="https://hb.vigilmon.online/YOUR_GROUP_HB_ID"

ALL_OK=1
for SERVER in $SERVERS; do
  ssh -o ConnectTimeout=5 "$SERVER" "systemctl is-active --quiet sssd" 2>/dev/null || ALL_OK=0
done

[ "$ALL_OK" -eq 1 ] && curl -fsS --retry 3 "$HB_URL" > /dev/null 2>&1

Conclusion

SSSD is one of those services that's invisible when it works and catastrophic when it doesn't — a silent failure means every user trying to SSH into your Linux servers gets an authentication error. The Vigilmon monitors in this guide cover the full SSSD failure surface: daemon process health, identity provider connectivity per domain, authentication success rates, NSS lookup latency, cache performance, and the particularly insidious offline mode transition. With these in place, your operations team gets alerted within 1–2 minutes of any SSSD failure — long before the support queue fills up with "can't log in" tickets.

Get started at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles