Meta’s engineering team has deployed ZGateway, a stateless proxy layer positioned between client applications and ZippyDB, the company’s primary key-value store. This system now manages more than 1 billion operations per second and processes approximately 40% of ZippyDB traffic, a figure expected to rise to 60%.
The connection sprawl problem
Before the proxy, every ZippyDB client established a direct connection to every database host it required. A single application could touch tens of thousands of shards across hundreds of thousands of servers. Each idle connection consumed memory, CPU, and a file descriptor on both ends. As client cohorts grew, inbound connection counts spiked. Reconnection storms frequently caused crashes from file descriptor exhaustion and out-of-memory errors. One incident involving a routing bug forced every client to open a connection per shard, driving the entire fleet into a reboot loop. Fixing this on the client side was impractical because hundreds of teams own the client fleet.
How ZGateway functions
ZGateway operates as a regional tier discovered through ServiceRouter, Meta’s service mesh. It exists in two modes: a pure proxy and a read-through cache. The engine is Meta’s C++ ZippyDB client, meaning ZGateway effectively runs a managed ZippyDB client service.
A client sends requests over a sticky connection to a regional ZGateway host. The host terminates TLS, authorises requests against the use case’s ACLs, and applies per-tenant admission control and shaping. It then resolves the shard, checks the local cache on caching tiers, batches the request with other in-flight work for that shard, and forwards it to the correct replicas. Responses are demultiplexed back with per-use-case metrics, traces, and quota usage recorded. TLS remains in the Thrift/ServiceRouter stack while replica selection stays in the embedded client.
The fan-in and fan-out math
Meta models the fleet as balls thrown into bins. With B shards and H hosts, a host is hit with probability E(H,B) = H(1 – e^-B/h). Using mock figures of 20 regions, 500,000 database hosts, 30,000 proxy hosts, 1,000,000 clients, and 50,000 shards per client, per-host connection counts collapse by roughly 97 to 98% and total persistent connections drop about 19x. The deeper win is scaling: direct-access fan-in grows linearly with clients, while ZGateway fan-in reduces to roughly regions times shard density per host, independent of both fleets.
Added capabilities
- Safe migration: Configuration flags scoped per service and shard prefix provide a percentage ramp, a region filter, and a global kill switch.
- Discriminant Load Shedding (DLS): Requests map to per-tenant buckets split by priority that drain round-robin, so a flooding tenant only fills its own bucket. In a controlled overload above 90% CPU across roughly 1,350 tenant buckets, only 6 noisy neighbours shed load, the rest executed 99.9% of requests with zero rejections, goodput held near 97 to 98%, and the machinery cost about 8% of CPU.
- Read caching: Cache tiers serve hot reads in-process, take a per-key fill lock on misses, and stay fresh via change-data-capture events under a bounded-staleness contract.
- Load balancing: Tiers mix roughly 26-core to 126-core hosts, so a control-plane balancer nudges each host’s ServiceRouter weight opposite to its recent CPU load.
- Cross-region resilience: Global routing, mega-regions, and rings let a saturated regional tier fail over to healthy capacity nearby.
- Transactions: Client-side bookkeeping moved into the gateway, consolidated in nine phases to 100% of transaction traffic with no reliability regression.
What it means
For the teams maintaining Meta’s infrastructure, the shift moves connection management from the application layer to a centralised proxy. This eliminates the risk of fleet-wide crashes caused by connection storms and allows for granular control over traffic admission. The ability to batch requests across clients reduces latency for hot keys, while the new load shedding mechanism ensures that a few misbehaving tenants cannot degrade service for the rest of the system. The solution is not available outside Meta, but the architectural patterns offer a blueprint for managing similar scale.
