Dual Control Planes for Geo-Partitioned Event Sourcing: Surviving Region Blackouts
Audience: platform, SRE, and staff engineers running multi-region Kafka/postgres event stores who need to survive full regional loss without breaking audit guarantees.
Table of Contents
Introduction
Event-sourced systems love audit trails, reliable replays, and append-only histories. They hate ambiguous failovers, partial replicas, and control planes that vanish mid-incident. In a single region, the control plane that owns schemas, topic policies, consumer offsets, and projection rollouts can be centralized. In a geo-partitioned deployment, centralization becomes a liability: if the region hosting the control plane disappears, operators are blind and the replay choreography that keeps projections coherent freezes.
This piece distills the pattern into concrete commands, configs, and a step-by-step recovery flow you can lift into a runbook.
Story: When the East Coast Disappears
It is 08:12 UTC on a Tuesday. An upstream cloud networking incident silently isolates us-east-1. Your Kafka control plane, Schema Registry primary, and GitOps controllers all live there. Producers in eu-west-1 keep writing locally. Consumers in ap-southeast-1 stall because their offset commits target a control plane that is now unreachable. Within 15 minutes, incident commanders say, "Fail east traffic to EU." The runbook says nothing about projection rebuild order or who owns schema evolution. Without a second control plane, you are left with hand-edited configs and hope.
Why Dual Control Planes in Event-Sourced Stacks
Dual control planes are not about hot spares; they are about autonomy. Each region needs a control plane that can:
- Approve or reject schema changes and topic ACLs without waiting for a remote admin API.
- Coordinate projection rebuilds and idempotent replay guards locally.
- Own consumer offset commits and DLQ routing within the region.
- Synchronize policy, not command, across peers so that one region's outage does not block others from acting.
Instead of a single orchestration brain, you operate two peer brains that exchange state through a low-frequency, signed policy mirror. During an outage, each region keeps processing with its last validated policy and queues diffs for later reconciliation.
Architecture Blueprint
The pattern builds four planes per region: data (Kafka + event store), control (GitOps + registries + orchestration), compute (services + projections), and observability (metrics/logs/traces). Duplicate the control plane in at least two regions and mirror policy as code through signed Git remotes.
Event Store Topology
For geo-partitioned event sourcing, the typical topology is:
- Kafka clusters per region with MirrorMaker 2 (or Cluster Linking) replicating topics across peer regions.
- Region-pinned partitions for latency-sensitive aggregates, plus geo-replicated audit topics for global reads.
- Write-local, read-anywhere semantics: producers write to their local cluster; global consumers read from replicas with well-defined lag budgets.
Control Plane Mechanics and Failover
Dual control planes operate in active-active mode with a treaty:
- Policy is mirrored Git: schemas, ACLs, consumer group ownership, projection rollout manifests.
- Runtime state is local: offsets, DLQ topics, replay cursors.
- Leaders are regional: each control plane is authoritative for its region's compute plane.
During normal operations, policy PRs merge in either region, signed, and mirrored. During a blackout, the surviving control plane freezes cross-region policy merges but keeps applying region-local manifests, then reconciles deterministically when the failed region returns.
Step-by-Step Recovery Flow
The runbook below assumes us-east-1 went dark while eu-west-1 survived:
- Freeze cross-region changes: Pause MirrorMaker/Cluster Linking for topics sourced from the failed region.
- Promote standby schemas: In EU, set Schema Registry compatibility to
BACKWARDand apply the last signed schema bundle. - Re-point producers: Toggle DNS or service mesh to route US traffic to EU ingress.
- Replay critical projections: Kick off scoped replays from a checkpointed offset snapshot, canceling on first failure.
- Switch read models: Update API read endpoints to point to EU projections.
- Drain DLQs locally: EU control plane owns DLQ processing; halt cross-region DLQ shipping until east recovers.
- Observe lag budgets: Alert if replication lag exceeds your RPO (e.g., 5 minutes).
- Restore east: Keep it read-only, reconcile schemas/ACLs, then re-enable mirrors from EU to US.
- Offset reconciliation and unfreeze: Export EU offsets, import to US once caught up, then reopen cross-region automation.
Failure Modes to Drill
- Schema divergence: One region merges a backward-incompatible schema before the mirror pauses. Drill rollbacks using the signed bundle mechanism.
- Mirror partition skew: After reconnection, some partitions are ahead in EU; rehearse selective catch-up with
kafka-reassign-partitions. - Projection poisoning: A bad event batch replays twice. Validate idempotency keys and ensure replay jobs stop on first duplicate detection.
Trade-offs and Costs
Dual control planes add expense: duplicate GitOps controllers, schema registries, and CI runners per region, plus more storage for mirrored topics. The payoff is autonomy: regional outages become localized incidents, and you keep audit-grade guarantees because replay order and schema governance never rely on a single region.
Mistakes to Avoid
- Letting offset edits bypass signed policy bundles.
- Failing to pause mirrors before rerouting producers, causing backfill storms.
- Replaying projections without idempotency keys or dedupe fences.
- Running a single Schema Registry for all regions.
- Treating DLQs as trash cans instead of structured queues with ownership.
Key Takeaways
- Control planes must be regional citizens with their own autonomy, not distant overlords.
- Policy mirrors and signed bundles keep schemas, ACLs, and ownership aligned across blackouts.
- Replay discipline, lag budgets, and DLQ ownership matter more than raw replication speed.
- Runbooks should be executable scripts with clear cancellation semantics, not prose.
Conclusion
Surviving a regional blackout in an event-sourced world is less about "flip DNS" and more about choreography: pausing mirrors, rerouting producers, replaying projections in the right order, and reconciling offsets with proof. Dual control planes give you that choreography even when one side of the world goes dark.