tutorial

Monitoring Akka Actor Systems with Vigilmon

Akka's actor model distributes failure across hierarchical supervision trees — but mailbox backlogs, dead letters, and cluster member splits are invisible without dedicated monitoring. Here's how to monitor your Akka ActorSystem, Cluster, Streams, and Persistence with Vigilmon.

Akka is an open source toolkit for building concurrent, distributed, and fault-tolerant event-driven applications on the JVM, originally developed by Jonas Boner and maintained by Lightbend. It implements the Actor Model: every unit of computation is an Actor — a lightweight, stateful, message-processing entity. Akka applications are hierarchical trees of actors communicating via asynchronous message passing, with built-in supervision strategies that restart failed actors automatically. The actor model's resilience is its strength, but it also makes failure modes subtle: a crashed actor gets restarted silently, but a stalled actor with a growing mailbox causes a slow degradation that's hard to detect. Vigilmon gives you external visibility into ActorSystem health, cluster membership, mailbox backlogs, and dead letter rates before those subtle failures become customer-visible incidents.

What You'll Set Up

  • ActorSystem health check via HTTP endpoint
  • Akka Cluster membership monitor
  • Akka HTTP server port and connection monitoring
  • Mailbox backlog and dead letter rate alerting
  • Akka Persistence journal write health check
  • Akka Streams backpressure detection
  • JVM heap and GC pause alerting

Prerequisites

  • Akka 2.8+ (Apache 2.0 license version, formerly Akka) or Apache Pekko
  • Akka HTTP or an existing HTTP endpoint for health exposure
  • Akka Management (recommended for Cluster health API)
  • A free Vigilmon account

Step 1: Monitor ActorSystem Health

The Akka ActorSystem is the runtime root for all actors. If the ActorSystem terminates — due to a fatal error or unhandled exception in a root actor — your entire application goes down. Expose an ActorSystem health endpoint:

import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.server.Directives._

class HealthRoutes(system: ActorSystem) {
  val routes = path("health") {
    get {
      if (!system.whenTerminated.isCompleted) {
        complete(200, s"""{"status":"UP","actors":${system.deadLetters.path.root}}""")
      } else {
        complete(503, """{"status":"DOWN","reason":"ActorSystem terminated"}""")
      }
    }
  }
}
  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter: http://your-akka-app:8080/health.
  4. Set Expected HTTP status to 200.
  5. Set Check interval to 1 minute.
  6. Click Save.

Step 2: Monitor Akka Cluster Membership

Akka Cluster tracks which JVM nodes are members of the distributed cluster. An unreachable member triggers split-brain risk — the cluster may partition into two independent groups each believing the other is down. Use the Akka Management HTTP endpoint which exposes cluster state:

  1. Add Akka Management to your project (akka-management dependency).
  2. Start the management endpoint: Akka Management binds to port 8558 by default.
  3. In Vigilmon, click Add Monitor.
  4. Set Type to HTTP / HTTPS with Keyword check.
  5. Enter: http://your-akka-node:8558/cluster/members.
  6. Set Keyword must NOT contain to "Unreachable".
  7. Set Check interval to 1 minute.
  8. Click Save.

The Akka Management /cluster/members endpoint returns JSON with member states. The keyword check fails (triggering an alert) as soon as any member enters Unreachable state.

For multi-node clusters, monitor each seed node:

http://akka-node-1:8558/cluster/members
http://akka-node-2:8558/cluster/members
http://akka-node-3:8558/cluster/members

Step 3: Monitor Akka HTTP Server Health

Akka HTTP's server processes all inbound REST requests. A binding failure — caused by port conflicts, address exhaustion, or ActorSystem problems — stops the HTTP server entirely. Monitor the server port directly:

  1. Click Add Monitor.
  2. Set Type to TCP Port.
  3. Enter host your-akka-app and port 8080 (or your configured HTTP port).
  4. Set Check interval to 1 minute.
  5. Click Save.

Also monitor active connection count by exposing it via your health endpoint:

path("health" / "http") {
  get {
    val stats = Http().serverSettings // gather connection stats from Akka HTTP
    complete(200, s"""{"connections":${activeConnections.get()}}""")
  }
}

Step 4: Monitor Mailbox Backlog and Dead Letters

In Akka, actors have mailboxes that queue incoming messages. An unboundedly growing mailbox indicates a stalled actor — it's receiving messages faster than it processes them, or it's blocked waiting on a future or I/O. Dead letters are messages sent to actors that have terminated — a sustained dead letter rate indicates misconfigured routing or actor lifecycle issues.

Expose mailbox sizes and dead letter counts via a monitoring endpoint:

class MailboxMonitorActor extends Actor {
  var deadLetterCount = 0L
  
  override def preStart(): Unit = {
    context.system.eventStream.subscribe(self, classOf[DeadLetter])
  }
  
  def receive: Receive = {
    case _: DeadLetter => deadLetterCount += 1
    case GetStats =>
      sender() ! MailboxStats(
        deadLetters = deadLetterCount,
        // Add per-actor mailbox sizes if using bounded mailboxes
      )
  }
}

Push a heartbeat when metrics are within bounds:

class AkkaHealthReporter(system: ActorSystem) {
  
  def reportHealth(): Unit = {
    val deadLetterRate = deadLetterTracker.getRate() // per minute
    val maxMailboxSize = mailboxMonitor.getMaxMailboxSize()
    
    if (deadLetterRate < 10 && maxMailboxSize < 1000) {
      vigilmon.fireHeartbeat("YOUR_MAILBOX_HEARTBEAT_ID")
    }
  }
  
  system.scheduler.scheduleAtFixedRate(0.seconds, 60.seconds)(
    () => reportHealth()
  )(system.dispatcher)
}

Create a Vigilmon Heartbeat monitor with a 3-minute timeout. Sustained dead letter spikes or growing mailboxes will stop the heartbeat and trigger an alert.

Step 5: Monitor Akka Cluster Sharding Health

Akka Cluster Sharding distributes stateful actors across cluster nodes. During shard rebalancing (triggered by node joins or departures), some entity actors become temporarily unavailable. Monitor shard region health via the Akka Management API:

  1. Click Add Monitor.
  2. Set Type to HTTP / HTTPS with Keyword check.
  3. Enter: http://your-akka-node:8558/cluster/shards/YOUR_SHARD_REGION_NAME.
  4. Set Keyword to "shards" to confirm the shard region is reporting state.
  5. Set Check interval to 2 minutes.
  6. Click Save.

You can also expose a custom shard health indicator:

path("health" / "shards") {
  get {
    val regionRef = ClusterSharding(system).shardRegion("MyEntity")
    onComplete(regionRef ? GetShardRegionState) {
      case Success(state: CurrentShardRegionState) =>
        val shardCount = state.shards.size
        complete(200, s"""{"status":"UP","shards":$shardCount}""")
      case _ =>
        complete(503, """{"status":"DOWN"}""")
    }
  }
}

Step 6: Monitor Akka Persistence Journal Health

Event-sourced actors write to an Akka Persistence journal (Cassandra, PostgreSQL, or another backend). Journal write failures cause event-sourced actors to fail and potentially lose state. Add a journal health check:

@Singleton
class PersistenceHealthIndicator(system: ActorSystem) {
  
  def reportJournalHealth(): Unit = {
    val writeLatency = journalMetrics.getP95WriteLatency()
    val errorRate = journalMetrics.getWriteErrorRate()
    
    if (errorRate < 0.01 && writeLatency < 100) { // <1% errors, <100ms p95
      vigilmon.fireHeartbeat("YOUR_JOURNAL_HEARTBEAT_ID")
    }
  }
}

Create a Vigilmon Heartbeat monitor with a 3-minute timeout. Journal write failures will stop the heartbeat and trigger an alert before event-sourced actors start losing state.

Step 7: Monitor Akka Streams Backpressure

Akka Streams apply backpressure when downstream processing is slower than upstream production. Sustained backpressure indicates a bottleneck that will eventually exhaust upstream buffers. Monitor stream throughput via a heartbeat:

class StreamHealthMonitor {
  
  def monitorStream[In, Out](source: Source[In, _], sink: Sink[Out, _])
      (implicit system: ActorSystem): Source[In, _] = {
    
    var elementsLastMinute = 0L
    var backpressureEvents = 0L
    
    source
      .map { elem =>
        elementsLastMinute += 1
        elem
      }
      .throttle(/* your expected rate */)
      // Backpressure is automatically applied by Akka Streams
  }
  
  def reportStreamHealth(): Unit = {
    val throughput = streamMetrics.getThroughput()
    val baseline = streamMetrics.getBaselineThroughput()
    
    // Alert if throughput drops >50% from baseline (sustained backpressure)
    if (throughput > baseline * 0.5) {
      vigilmon.fireHeartbeat("YOUR_STREAM_HEARTBEAT_ID")
    }
  }
}

Step 8: Monitor JVM Heap and GC Pause Health

Akka runs on the JVM and is sensitive to GC pauses — a long GC pause can stall the actor dispatcher, causing mailbox processing to freeze and timeouts to cascade. Monitor heap usage and GC pause duration:

import java.lang.management.ManagementFactory
import scala.concurrent.duration._

class JvmHealthReporter(system: ActorSystem) {
  
  def reportJvmHealth(): Unit = {
    val mem = ManagementFactory.getMemoryMXBean
    val heapUsed = mem.getHeapMemoryUsage.getUsed
    val heapMax = mem.getHeapMemoryUsage.getMax
    val heapPct = heapUsed.toDouble / heapMax * 100
    
    val maxGcPause = ManagementFactory.getGarbageCollectorMXBeans.asScala
      .map(gc => if (gc.getCollectionCount > 0) gc.getCollectionTime / gc.getCollectionCount else 0L)
      .maxOption.getOrElse(0L)
    
    if (heapPct < 85 && maxGcPause < 2000) {
      vigilmon.fireHeartbeat("YOUR_JVM_HEARTBEAT_ID")
    }
  }
  
  system.scheduler.scheduleAtFixedRate(0.seconds, 60.seconds)(
    () => reportJvmHealth()
  )(system.dispatcher)
}

Create a Vigilmon Heartbeat monitor with a 3-minute timeout. Heap above 85% or GC pauses above 2 seconds will stop the heartbeat and alert you before actor scheduling stalls cascade into user-visible failures.

Alerting Configuration

Configure alert channels for each Akka monitoring signal:

| Alert | Condition | Recommended Channel | |-------|-----------|---------------------| | ActorSystem terminated | HTTP 503 on /health | PagerDuty / SMS | | Cluster member unreachable | Keyword "Unreachable" detected | PagerDuty | | HTTP server down | TCP port check fails | PagerDuty | | Mailbox backlog / dead letters | Heartbeat missing > 3 min | Slack | | Shard region unhealthy | HTTP check fails | Slack | | Journal write failures | Heartbeat missing > 3 min | PagerDuty | | Stream throughput drop | Heartbeat missing > 3 min | Slack | | JVM heap / GC pause | Heartbeat missing > 3 min | Slack + Email |

In Vigilmon, go to Alerts → Notification Channels and add your Slack webhook, email, or PagerDuty integration key. Assign each monitor to its appropriate channel.

Conclusion

Akka's supervision-based resilience is powerful — but it makes failure modes subtle. Actors restart silently, mailboxes grow quietly, and dead letters accumulate invisibly until a stalled actor causes cascading timeouts. Vigilmon's combination of HTTP checks on Akka Management endpoints, TCP port monitors, and heartbeat monitors driven from within your ActorSystem gives you early warning on every failure mode: cluster splits, journal write failures, mailbox backlogs, and JVM heap saturation. You'll know before the split-brain protection kicks in, not after.

Start with a free Vigilmon account and add your first Akka cluster health monitor in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles