Kafka operations require evidence across brokers/controllers, producers, consumers, stream processors, networks, storage, and downstream systems. One dashboard or consumer-lag number cannot establish health.
Define service objectives
Set availability, durability, end-to-end freshness, throughput, p95/p99 latency, recovery time, recovery point, and cost objectives by workload. Record partition ordering, retention, replay, and data-loss tolerances.
Monitor layers
| Layer | Signals |
|---|---|
| Cluster | Controller/quorum health, under-replicated/offline partitions, ISR changes, disk, network, request latency |
| Producer | Errors, retries, record latency, batching, compression, throttling |
| Consumer | Lag with input rate, poll/commit errors, rebalances, processing freshness |
| Streams | Task state, restoration, dropped/late records, store and processing metrics |
| Outcome | Reconciliation, missing/duplicate events, downstream availability and freshness |
Test capacity and failure
Use representative record sizes, keys, partitions, compression, replication, retention, consumer groups, and bursts. Exercise broker/controller loss, disk pressure, network partition, zone loss, rebalance, rolling upgrade, credential rotation, throttling, and restore.
Clarify cloud responsibility
Managed services can operate infrastructure while customers retain responsibility for data classification, identity, network access, topic/ACL design, clients, schemas, retention, monitoring, and application recovery. Validate provider-specific availability, quotas, encryption, logs, regions, backup/export, egress cost, and exit plan.
Connect Streams monitoring, advanced configuration, and security and multi-cluster planning.
Reviewed against current Apache Kafka documentation September 4, 2026. Original publication date preserved.
Completion guide. A healthy broker is not the same thing as a healthy event-driven product. Monitor from the producer request through topic durability and consumer progress to the business outcome the event was supposed to create.
The observability signal path
Instrument every boundary with rate, errors, and duration, then add Kafka-specific saturation and correctness signals. End-to-end freshness—a timestamp or synthetic event observed at the destination—is the strongest check that the whole path is working.
Map symptoms to Kafka signals
| User-visible symptom | Kafka evidence | Correlate with | First question |
|---|---|---|---|
| Events arrive late | Consumer lag by partition; fetch and processing rates | Ingress rate, handler latency, rebalances | Is lag growing everywhere or on one partition? |
| Writes are slow | Produce request latency, timeouts, retries | ISR changes, disk/network saturation | Did acknowledgement wait or broker service time grow? |
| Missing results | Offset movement and error/DLQ counts | Schema failures, transaction aborts, sink errors | Was the record never produced, not consumed, or rejected? |
| Unstable throughput | Rebalance frequency, partition skew | Deployments, GC, autoscaling, hot keys | Does instability align with membership changes? |
A runbook for rising consumer lag
- Scope. Compare partitions. One hot partition suggests key skew; broad growth suggests insufficient capacity or downstream slowness.
- Rate. Compare arrival rate with consume and processing rates. Lag alone has no time dimension.
- Stability. Check group membership and rebalance activity around deploys, autoscaling, or timeouts.
- Handler. Inspect external calls, retries, GC pauses, and record-size outliers inside the consumer.
- Recoverability. Estimate drain time at safe capacity before adding workers; partitions cap useful parallelism.
Failure drill and release gate
Broker interruption
Confirm leaders recover, producers retry within their delivery contract, ISR returns to normal, and no silent loss is observed.
Poison record
Verify deserialization and business-rule failures follow a bounded retry and quarantine policy with enough context to replay safely.
Slow dependency
Throttle a sink and validate backpressure, lag alerts, resource ceilings, and drain-time estimates.
Alert on customer-impacting conditions and sustained trends, not every transient metric movement. Each alert should link to a runbook, name an owner, and state the safe first action.
Primary references and next reading
Use version-matched documentation for configuration defaults. The links below point to maintained upstream documentation rather than copied defaults.
Historical comments from Datanizant
No public comments on this article
No approved public comments were included in the WordPress export for this article.