tutorial

Monitoring Apache Polaris Catalog with Vigilmon

Apache Polaris is an open source Iceberg REST Catalog server. Here's how to monitor Polaris server health, REST API availability, vended credentials, catalog metadata DB, commit rates, and admin API with Vigilmon.

Apache Polaris (incubating) is an open source implementation of the Apache Iceberg REST Catalog API — the catalog server that your query engines (Spark, Flink, Trino, DuckDB, StarRocks) use to discover, commit, and manage Iceberg tables. When you self-host Polaris, every table operation flows through it: engine startup, snapshot commits, metadata reads, and storage credential vending. If Polaris goes down, every connected engine loses catalog access and all table writes fail. Vigilmon monitors the full Polaris stack — server process, REST API, metadata database, vended credentials, and admin API — so you know the moment any layer degrades.

What You'll Set Up

  • Polaris server process health monitor (HTTP port)
  • REST Catalog API health probe
  • Vended credential success rate heartbeat
  • Catalog metadata database connectivity monitor
  • Iceberg table commit health check
  • Principal authentication success rate monitor
  • Namespace and table listing latency probe
  • Admin API availability monitor

Prerequisites

  • Apache Polaris deployed (Docker, bare metal, or Kubernetes) with the Quarkus HTTP server running
  • Polaris REST Catalog API port accessible (default: 8181)
  • Polaris admin API port accessible (default: 8182)
  • PostgreSQL or H2 database for catalog metadata storage
  • A free Vigilmon account

Step 1: Monitor the Polaris Server Process

Polaris runs as a Quarkus Java application. A JVM crash or OOM kill removes catalog access for every connected engine simultaneously. Add an HTTP uptime monitor on the Polaris health endpoint.

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. URL: http://polaris.yourdomain.com:8181/q/health (the Quarkus built-in health endpoint) or http://polaris.yourdomain.com:8181/healthcheck depending on your deployment.
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Under Keyword check, enter UP to verify the Quarkus health response body.
  7. Click Save.

If your Polaris deployment uses a reverse proxy (nginx, Caddy), add a second monitor on the external HTTPS URL with SSL certificate monitoring enabled:

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: https://polaris.yourdomain.com/q/health.
  3. Check interval: 1 minute.
  4. Enable Monitor SSL certificate → set expiry alert to 21 days.
  5. Click Save.

Alert recommendation: Alert after 1 missed check — a Polaris outage is a full catalog blackout for all connected engines.

Step 2: Monitor the Iceberg REST Catalog API

The Iceberg REST Catalog API is what query engines actually call — to load namespaces, fetch table metadata, and commit new snapshots. Monitor the catalog API endpoint directly to catch routing errors, authentication middleware failures, or API layer bugs that may not surface on the generic health check.

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: http://polaris.yourdomain.com:8181/api/catalog/v1/config.
  3. Expected HTTP status: 200.
  4. Check interval: 1 minute.
  5. Click Save.

The /api/catalog/v1/config endpoint returns the catalog configuration without requiring table-level credentials — it's a lightweight, unauthenticated (or lightly authenticated) probe that confirms the REST layer is routing correctly.

For a deeper API health check that exercises authentication:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"

# Use a service principal token to list namespaces — exercises auth + REST API
TOKEN=$(curl -s -X POST http://polaris.yourdomain.com:8181/api/catalog/v1/oauth/tokens \
  -d "grant_type=client_credentials&client_id=YOUR_CLIENT_ID&client_secret=YOUR_CLIENT_SECRET&scope=PRINCIPAL_ROLE:ALL" \
  | jq -r '.access_token')

STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
  -H "Authorization: Bearer $TOKEN" \
  http://polaris.yourdomain.com:8181/api/catalog/v1/YOUR_CATALOG/namespaces)

if [ "$STATUS" = "200" ]; then
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

Schedule this as a cron heartbeat with a 2-minute interval.

Step 3: Monitor Vended Credential Health

Polaris vends short-lived storage credentials (AWS STS tokens, GCS service account tokens, ADLS SAS tokens) to query engines per-operation. If credential vending fails, the engine can list table metadata but cannot read or write the actual Parquet/Avro data files. This failure mode is silent at the catalog level — the REST API stays healthy while engines get storage access-denied errors.

Create a heartbeat that exercises the credential vending path end-to-end:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"

# Get a token and request load credentials for a known table
TOKEN=$(curl -s -X POST http://polaris.yourdomain.com:8181/api/catalog/v1/oauth/tokens \
  -d "grant_type=client_credentials&client_id=YOUR_CLIENT_ID&client_secret=YOUR_CLIENT_SECRET&scope=PRINCIPAL_ROLE:ALL" \
  | jq -r '.access_token')

CRED_STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
  -H "Authorization: Bearer $TOKEN" \
  "http://polaris.yourdomain.com:8181/api/catalog/v1/YOUR_CATALOG/namespaces/YOUR_NS/tables/YOUR_TABLE/credentials")

if [ "$CRED_STATUS" = "200" ]; then
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

Schedule this heartbeat every 5 minutes. If Polaris's IAM role, storage integration configuration, or credential broker connectivity fails, the heartbeat goes silent and Vigilmon alerts you.

Step 4: Monitor the Catalog Metadata Database

Polaris stores all catalog metadata (catalog definitions, namespaces, table metadata locations, principal grants) in a relational database. A database connectivity loss causes every catalog operation to fail with a 500 error. Monitor the database independently of Polaris to distinguish a DB failure from a Polaris application failure.

For PostgreSQL:

  1. Click Add Monitor → TCP Port.
  2. Host: postgres.yourdomain.com (your PostgreSQL server).
  3. Port: 5432.
  4. Check interval: 1 minute.
  5. Click Save.

For a deeper connection pool health check, add a Polaris-specific database probe heartbeat:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"

# Check Polaris metrics endpoint for DB connection pool health
POOL_STATUS=$(curl -s http://polaris.yourdomain.com:8181/q/metrics \
  | grep 'agroal_pool_available_count' | awk '{print $2}')

if [ -n "$POOL_STATUS" ] && [ "${POOL_STATUS%.*}" -gt 0 ]; then
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

Alert recommendation: DB connectivity loss is a P0 incident — alert immediately.

Step 5: Monitor Iceberg Table Commit Health

Iceberg table commits (appends, overwrites, schema evolution, partition spec changes) are ACID operations that go through Polaris. A high commit conflict rate (HTTP 409) indicates concurrent write contention between engines. Monitor the commit path via a lightweight test commit heartbeat:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"

TOKEN=$(curl -s -X POST http://polaris.yourdomain.com:8181/api/catalog/v1/oauth/tokens \
  -d "grant_type=client_credentials&client_id=YOUR_CLIENT_ID&client_secret=YOUR_CLIENT_SECRET&scope=PRINCIPAL_ROLE:ALL" \
  | jq -r '.access_token')

# Fetch table metadata to verify commit path is accessible (read-only probe)
STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
  -H "Authorization: Bearer $TOKEN" \
  "http://polaris.yourdomain.com:8181/api/catalog/v1/YOUR_CATALOG/namespaces/YOUR_NS/tables/YOUR_TABLE")

if [ "$STATUS" = "200" ]; then
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

For production commit monitoring, integrate your Spark or Flink job to push a heartbeat ping after each successful checkpoint commit.

Step 6: Monitor Namespace and Table Listing Latency

Query engines list namespaces and tables during startup and periodically during operation. High listing latency indicates database query slowness, index degradation, or catalog metadata growth hitting performance limits.

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"
MAX_LATENCY_MS=1000  # 1-second p95 SLO for listing

TOKEN=$(curl -s -X POST http://polaris.yourdomain.com:8181/api/catalog/v1/oauth/tokens \
  -d "grant_type=client_credentials&client_id=YOUR_CLIENT_ID&client_secret=YOUR_CLIENT_SECRET&scope=PRINCIPAL_ROLE:ALL" \
  | jq -r '.access_token')

START=$(date +%s%3N)
curl -s -o /dev/null \
  -H "Authorization: Bearer $TOKEN" \
  "http://polaris.yourdomain.com:8181/api/catalog/v1/YOUR_CATALOG/namespaces"
END=$(date +%s%3N)

ELAPSED=$((END - START))
if [ "$ELAPSED" -lt "$MAX_LATENCY_MS" ]; then
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

Schedule every 5 minutes. Latency exceeding 1 second against a healthy DB indicates catalog metadata growth that needs index tuning.

Step 7: Monitor the Polaris Admin API

Polaris provides a management API for creating catalogs, principals, roles, and grants. If the admin API goes down, you lose the ability to provision new catalogs or rotate service principal credentials — a significant operational blocker.

  1. Click Add Monitor → TCP Port.
  2. Host: polaris.yourdomain.com.
  3. Port: 8182 (the default Polaris admin API port).
  4. Check interval: 1 minute.
  5. Click Save.

Add an HTTP check to verify the admin API layer:

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: http://polaris.yourdomain.com:8182/api/management/v1/catalogs.
  3. Expected HTTP status: 200 or 401 (both confirm the API is responding; 401 means the admin auth layer is enforcing credentials correctly).
  4. Check interval: 1 minute.
  5. Click Save.

Step 8: Configure Alerting

Set up an alert channel and apply it to all Polaris monitors:

  1. Go to Alerts → Add Alert Channel.
  2. Choose Email, Slack, PagerDuty, or Webhook.
  3. Apply the channel to every Polaris monitor.

Recommended thresholds:

| Monitor | Alert After | Severity | |---|---|---| | Polaris server health | 1 missed check | Critical | | REST Catalog API health | 1 missed check | Critical | | Metadata DB connectivity | 1 missed check | Critical | | Vended credential heartbeat | 1 missed check | High | | Admin API port | 2 missed checks | High | | Listing latency SLO | 1 missed check | Medium | | Table commit probe | 2 missed checks | Medium |

Conclusion

Apache Polaris is the single point of truth for every Iceberg table your engines know about. A server crash, database failure, or credential vending outage can silently break all data pipeline reads and writes long before anyone notices a missing dashboard metric. With Vigilmon you get layered monitoring across the Polaris HTTP server, the Iceberg REST API, metadata database, credential vending, and the admin API — each with independent alerting so you can triage the exact failure layer in seconds.

Get started at vigilmon.online — free for up to 5 monitors.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles