Stay up to date with AI tools, model releases, and developer workflows that matter.
Weekly. Free. One click to leave.
Industry observers note that 7 Day Free Trial. Cancel Anytime.
Recent findings suggest that most Kafka deployments in the industry plan to survive the loss of a single Kafka broker or an availability zone. But that's not enough for critical Tier-0 infrastructure where downtime or data loss of even a few seconds could have a massive business impact.
Recent findings suggest that in such scenarios, the failure model should consider not just the loss of a broker or a zone, but the loss of an entire region. The business is expected to operate without interruption even when a whole region goes down, and it should be possible to also roll the cluster safely while operating in that state, to apply any break-glass changes.
Sources indicate that but what actually happens inside Kafka when an entire region goes down? What happens to the replicas, producer writes, and KRaft quorum? And what happens when that region comes back?
Let's walk through these scenarios using a Tier-0 multi-region Kafka architecture.
For a Tier-0 multi-region Kafka cluster, we need three different regions with isolated failure domains and a minimum of two brokers in each region.
Brokers in each region should be configured with rack awareness to indicate their failure domains as follows:.
For Tier-0 workloads, each topic's partitions should be configured to have a replication factor (RF) of 6. Kafka's rack-aware replica assignment will spread the replicas across the three regions. If the automatic replica assignment doesn't allocate two replicas per region, we have to explicitly reassign replicas immediately after topic creation to meet our availability guarantees.
This way, if one region goes down entirely, we still have four replicas surviving and the cluster would still continue to operate.
Recent findings suggest that but just running brokers in multiple regions is not the solution here. In fact, that's the easy part. The hard question is, what guarantee remains when one of those regions disappears, and does it solve the business needs?
Let's say Region A goes down entirely.
The two brokers in that region become unavailable, leaving four surviving replicas across Regions B and C.
For any partitions whose leaders were in Region A, Kafka needs to elect new leaders from the surviving in-sync replicas.
But having four surviving replicas alone does not determine whether Kafka will continue accepting writes.
The next step is to define what is considered a successful write in this Kafka setup. For this, we need two configs:.
min.insync.replicas determines the minimum number of replicas that should be present in the in-sync replica set (ISR) of a partition for Kafka to accept writes when producers use acks=all.
acks=all is a producer-side config that ensures that the producer will get an acknowledgment only when the write has been successfully replicated to the required in-sync replicas.
These two settings work together to satisfy the durability guarantees we want.
Sources indicate that when all three regions are healthy, ISR would include all six replicas. Let's say one region goes down, then the ISR would drop down to four.
Since minISR is three, writes will still continue to work.
According to recent reports, there may be temporary interruptions while Kafka detects the failure, elects latest leaders, and clients reconnect to them. But once the surviving leaders are available, the cluster can continue accepting writes as long as the ISR requirements are met.
The goal isn't availability at any cost, but rather the system should be available as long as the durability guarantees that we designed for can be maintained.
Now let's make the scenario more interesting.
Region A is still down. We have four replicas surviving across Regions B and C.
What if we need to roll the cluster one broker at a time, to apply any break-glass changes?
According to recent reports, during that time, the ISR can go down from four to three. But since minISR is three, writes will still continue to work.
Let's look at how the cluster behaves under different failure scenarios.
Kafka will stop accepting writes only when the surviving ISR drops below three.
This is an notable distinction for Tier-0 workloads.
It's not sufficient to design a cluster that survives a region failure. We also need to consider whether the cluster can continue operating safely while already in that degraded state.
So far, we focused on the data plane. Now switching gears, the KRaft controllers have a completely different availability model.
They form the metadata quorum, and a majority is needed to continue operation.
For our setup, five controllers could be distributed as 2 + 2 + 1 across the three regions, requiring a majority of three to maintain quorum.
If either of the regions containing two controllers goes down, three controllers remain. If the region containing one controller goes down, four remain.
So, the controller quorum survives the loss of any one region.
But, unlike brokers, we cannot necessarily roll the controllers at that point because taking one controller down could cause the quorum to drop below three.
Sources indicate that with a static quorum, one way to solve this is to use more controllers, and a 3 + 3 + 3 distribution would provide this. Losing one region would leave six controllers with room to still roll the controllers, as we only need five to maintain quorum.
According to recent reports, on the flip side, an increase in quorum size also increases the coordination overhead. The official Kafka documentation typically recommends three or five controllers, which tolerate one or two concurrent failures respectively.
Alternatively, with dynamic KRaft quorum (KIP-853), introduced in Kafka 3.9, we can add or remove controllers from the voting quorum without restarting the cluster.
For example, with our 2 + 2 + 1 setup, if a region containing two controllers goes down, the remaining three controllers can still maintain quorum. We can then remove the failed controllers from the voting membership, one at a time, reducing the quorum size to three. This allows us to roll one of the surviving controllers while still maintaining the new majority of two.
Industry observers note that however, dynamic quorum reconfiguration is not automatic. The surviving controllers must be able to maintain quorum and successfully commit the membership changes before we can proceed with the controller restart.
You need to consider whether you want the same availability guarantee for the controllers as that of the data plane, and whether dynamic quorum reconfiguration during a regional outage is acceptable for your Tier-0 workloads.
So far, we assumed that a region either works or goes down completely.
But the WAN itself becomes a failure mode.
In this setup, a partly working WAN between two regions can be more dangerous than a link that's completely down.
Increased latency, packet loss, or intermittent connectivity issues between regions can cause replication to lag, ultimately resulting in replicas falling out of ISR.
The three regions might be independently healthy, but if the connectivity between them is not, then the availability of the cluster might be impacted.
Going back to the CAP theorem, to tolerate a network partition, we must choose between consistency and availability.
So, if a partition's ISR falls below the configured min.insync.replicas of three due to a network partition, Kafka stops accepting writes with acks=all rather than weakening those guarantees.
Similarly, if the KRaft controller quorum is lost, then no metadata changes, like leader elections, can happen in the cluster, even though existing data-plane traffic may continue for some time.
The availability guarantees we discussed also come with their own costs.
The first cost is client latency. With cross-region replication and acks=all, the network latency between these regions directly affects producer latency. Consumers can also pay the WAN latency cost when consuming from partition leaders located in a different region.
In a recent development, the second cost is replication overhead. With RF=6, there is additional storage, network traffic, and replication-related coordination cost in the brokers compared to a smaller number of replicas.
So, in an architecture like this, monitoring the health of the network paths between these regions is as critical as monitoring the Kafka brokers and controllers.
Metrics like round-trip latency (RTT) between regions, packet loss, TCP retransmissions, connection errors/timeouts, and network throughput/bandwidth utilization are a good start.
When one region goes down, all the partition leaders in that region would move to the two other surviving regions.
This implies that those two regions would have to handle more producer and consumer traffic, network throughput, and disk I/O.
Even if the cluster is designed to satisfy its ISR and controller quorum requirements, if it's not sized to handle these failures when one region goes down, the surviving regions would be overloaded, resulting in higher latencies, timeouts, and operational instability.
So, for Tier-0 infrastructure, capacity planning should always be done for failure states and not stop at normal operations.
Now let's say Region A comes back online.
The brokers in the recovered region might have to catch up on a significant amount of data before they become fully in sync again.
This can result in a sudden burst of cross-region network traffic and disk I/O across all regions.
Region recovery needs to be controlled so that it doesn't overwhelm the healthy part of the cluster.
This can be achieved by throttling the replica catch-up, with the trade-off of longer recovery time to minimize the impact on live production traffic.
Replication throttling needs a deeper discussion of its own, so we won't go into details here.
According to recent reports, the crucial point is that bringing a failed region back online is not simply a matter of restarting its brokers. We need to make sure that the recovery process itself doesn't impact the business.
Before putting a Tier-0 multi-region Kafka cluster into production, validate these scenarios:.
In a recent development, there is no shortage of ways to deploy Kafka across multiple regions. The challenge here is to pick the right architecture that addresses all our durability guarantees.
For Tier-0 workloads, this means designing the architecture around failures first.
Sources indicate that what happens when a region goes down? Can we still roll the cluster? What continues to work, and at what point do we deliberately stop accepting writes?
Lastly, can we safely bring back the region that was down without impacting the business?
That is ultimately the difference between simply running Kafka across multiple regions versus actually supporting a mission-critical Tier-0 workload.
Sources indicate that 7 Day Free Trial. Cancel Anytime.
