Kafka at scale is an operational and risk-management problem, not a fixed broker count. Define throughput, latency, durability, availability, recovery, retention, security, sovereignty, and cost objectives before selecting topology.
Secure every connection and identity
Use supported TLS and authentication mechanisms, protect credentials and keys, and authorize producers, consumers, transactional IDs, groups, topics, cluster actions, and administrative APIs through least privilege. Test certificate rotation, revocation, bootstrap behavior, failed authorization, and emergency access.
Network encryption does not replace application-level data classification, minimization, retention, schema controls, or consumer authorization. Apply broader network security controls.
Use current KRaft architecture
Current Apache Kafka uses KRaft metadata mode rather than Apache ZooKeeper. Separate or combined broker/controller roles depend on deployment scale and current project guidance. Protect controller quorum availability, credentials, networks, storage, monitoring, upgrades, and backups of required configuration and security material.
Choose multi-cluster topology from failure domains
Clusters may be separated for regions, environments, business units, compliance boundaries, blast radius, or capacity. Separation adds duplicated infrastructure, governance, data movement, ordering, failover, and consistency challenges.
MirrorMaker 2 and vendor replication services replicate selected records and metadata with documented limits; they do not create synchronous multi-region transactions or zero-loss failover. Define topic selection, prefixes/naming, offsets, ACL handling, loop prevention, lag, duplicates, conflict policy, recovery point, recovery time, failover authority, and failback.
Treat “serverless Kafka” as a managed-service contract
Managed offerings can reduce infrastructure tasks but retain customer responsibilities for schemas, identities, authorization, data, clients, retention, monitoring, cost, incident response, and architecture. “Serverless” does not mean no servers, unlimited scaling, zero operations, or automatic portability.
Capacity and reliability tests
- Measure producer/consumer throughput, end-to-end latency, batch/compression effects, partition skew, replication, storage, and network saturation.
- Use replication,
min.insync.replicas, producer acknowledgments, idempotence, and transactions according to loss/duplication requirements. - Test broker/controller loss, rack/zone loss, network partition, disk pressure, credential rotation, quotas, reassignments, upgrades, and replication outages.
- Monitor under-replicated and offline partitions, ISR changes, request latency/errors, consumer lag with context, controller health, disk, network, JVM, quotas, and replication lag.
Govern data and schemas
Assign topic ownership, purpose, classification, schema compatibility, retention/deletion, access review, lineage, quality objectives, and decommissioning. Compaction is not a guaranteed immediate deletion mechanism; privacy erasure may require architecture and lifecycle controls beyond a Kafka topic.
Continue with advanced Kafka configuration and Kafka Streams monitoring. Validate every property against the exact Kafka and client releases.
Originally published April 5, 2018; technically reviewed and substantially updated September 4, 2026.
Completion guide. Scaling Kafka is an architecture exercise: define failure domains, trust boundaries, recovery objectives, and ownership before adding brokers or regions.
The multi-cluster decision map
| Pattern | Use when | Hidden cost | Proof required |
|---|---|---|---|
| Active / passive | Recovery site with a clear primary | Replication delay, client cutover, offset translation | Measured RPO/RTO in a failover drill |
| Regional isolation | Locality and bounded blast radius matter | Global views need deliberate aggregation | Behavior during cross-region partition |
| Hub / spoke | Central analytics consumes selected regional streams | Hub bottlenecks and data-governance complexity | Backfill and schema-compatibility plan |
| Active / active | Independent writes are truly required | Conflicts, duplicate effects, ordering and ownership | Deterministic conflict and reconciliation rules |
Cross-cluster replication copies records; it does not automatically preserve every operational property of the source environment. Validate topic configuration, ACLs, schemas, consumer cutover behavior, and the meaning of offsets at the destination.
Secure every trust boundary
Identity
Give workloads distinct principals. Prefer short-lived or centrally rotated credentials and remove shared human/service identities.
Transport
Encrypt client, inter-broker, and administrative traffic where the threat model requires it; verify hostname and certificate behavior.
Authorization
Grant topic, group, transactional ID, and cluster permissions narrowly. Test denied operations as part of CI or staging.
Evidence
Audit administrative changes and authentication failures without logging secrets or sensitive payloads.
KRaft removes the ZooKeeper dependency, but it does not remove control-plane engineering. Isolate controller responsibilities appropriately, protect controller listeners, monitor quorum health, and rehearse controller and broker replacement using the documentation for the deployed version.
Turn disaster recovery into a contract
- RPO: maximum acceptable event loss, expressed in time or records and measured at the destination.
- RTO: time from declaration to verified producer, consumer, and downstream recovery.
- Authority: who declares failover, changes routing, and prevents split-brain writes.
- Dependencies: schemas, secrets, DNS, connectors, ACLs, and downstream capacity exist in the recovery region.
- Return path: failback and reconciliation are rehearsed, not deferred until after an incident.
Managed Kafka is shared responsibility
A managed or “serverless” offering may automate broker placement, upgrades, and scaling, but the application team still owns keys, schemas, retention intent, consumer behavior, access design, cost controls, and end-to-end correctness. Confirm quotas, networking, private connectivity, observability export, version policy, and exit/replay options before adoption.
Primary references and next reading
Use version-matched documentation for configuration defaults. The links below point to maintained upstream documentation rather than copied defaults.
Historical comments from Datanizant
No public comments on this article
No approved public comments were included in the WordPress export for this article.