Netflix engineers have reduced read latency for wide Apache Cassandra partitions from seconds to low double-digit milliseconds by splitting partitions per ID. The work targets the TimeSeries Abstraction, the platform handling temporal event data across the streaming service.
Dynamic partitioning splits wide Cassandra partitions per TimeSeries ID, asynchronously and transparently, with no application changes.
- Detection runs on the read path via byte counting and a Kafka event; splitting targets immutable partitions first.
- Bloom filters (single-digit microseconds) plus a cached
wide_rowmetadata lookup route reads to the smaller child partitions. - Checksums, retained original partitions, Data Bridge Spark checks, and shadow comparison guard correctness.
- Reads improved from seconds to low double-digit milliseconds; tail latency fell to ~200 ms; 500MB+ partitions stayed available.
What is Dynamic Repartitioning?
Netflix’s TimeSeries Abstraction ingests and queries petabytes of temporal event data with millisecond latency. It uses Apache Cassandra 4.x as the underlying storage. The team chose Cassandra for throughput, latency, cost, and operational maturity.
Time-series data is organized into partitions that group events by identifier and time range. These partitions can grow ‘wide’ as events accumulate over time. Dynamic repartitioning splits an oversized partition into smaller child partitions asynchronously. Applications keep querying the same logical partition while the storage layout evolves transparently.
Why Wide Partitions Hurt Reads
For most datasets, average read latency stays in single-digit milliseconds. When partitions grow too wide, tail read latencies rise into seconds. These slow reads can produce read timeouts. In extreme cases, clusters see Garbage Collection pauses, high CPU utilization, and thread queueing.
TimeSeries servers also handle very high read throughput, which compounds the problem. Scaling up the Cassandra cluster is always an option. But the Netflix team wanted smarter alternatives than just throwing more money at the problem.
The Partitioning Strategy Behind TimeSeries
TimeSeries breaks datasets into discrete time chunks: Time Slices, time buckets, and event buckets. This lets the Netflix team query and drop data by time without creating tombstones. When a namespace (dataset) is created, users specify anticipated workload characteristics. A provisioning pipeline runs Monte Carlo simulations to pick infrastructure and partition configuration.
This up-front approach falls short in three situations. Workload can be unknown or inaccurately estimated early in a project. Workload can also evolve as traffic and product requirements change. Finally, data outliers exist, where a few IDs receive far more events.
Discrete Time Slices provide a natural escape hatch for the first two cases. Each new Time Slice can use a different partitioning strategy. But manually tuning thousands of datasets is not sustainable, so automation is required.
Solution 1: Time Slice Re-Partitioning
Cassandra exposes introspection APIs such as nodetool tablehistograms for partition-size percentiles. A background worker monitors these histograms and exposes them through a Cassandra virtual table. It computes an adjustment factor when partition sizes miss a configured density. That density is often set between 2 MiB and 10 MiB, depending on workload.
For example, provisioning once selected 60-second time buckets, producing partitions under 10 KB. That over-partitioning caused high read amplification and thread queueing. The worker updates future Time Slices with a corrected strategy:
DynamicTimeSliceConfigWorker:
namespace: my_dataset_1
Observed: TimeSlices have p99 partitions below configured target of 10MB.
Proposed: time_bucket interval: 60s -> 604800sThis reduced read latencies and timeouts caused by thread queueing. But it only helps when most of the table warrants re-partitioning. It does not help when only a percentage of IDs are wide. For those partial cases, Netflix team offers three options:
- Do Nothing: the right approach when top-level metrics show no impact.
- Partial Returns: aborts an in-flight request breaching a latency SLO, returning data collected so far.
- Block IDs: an extreme step for test or spam IDs that destabilize the system.
Block IDs is applied through configuration, listing the offending TimeSeries IDs:
dgwts.config.<dataset>.block.Ids: "<tsid-1>, <tsid-2>, <tsid-3>"These options do not help when valid, important IDs grow wide. Those callers still need every event, which is where Solution 2 applies.
Solution 2: Dynamic Partitioning per ID
Dynamic partitioning is an asynchronous pipeline that splits wide partitions per TimeSeries ID. It operates at the ID level, not the table level. It has three stages: Detection, Planning & Splitting, and Serving Reads.
Detection happens on the read path. Every read tracks bytes read for a partition. When bytes exceed a configured threshold, the server emits an event to Kafka:
{
"time_slice": "data_20260328",
"time_series_id": "profileId:123",
"time_bucket": 7,
"event_bucket": 2,
"immutable": true,
"version": "0"
}Here, immutable marks a partition no longer receiving writes. The version field is reserved for future use, such as invalidation. The Netflix team detects on reads, not writes, because most data never needs splitting. Some reads on wide partitions stay slow for seconds until the pipeline catches up. The initial implementation targets immutable partitions to reduce complexity.
Planning reads the entire partition once to compute an accurate split plan. Checkpointing lets failed planning reads resume from the last saved point. The wide_row metadata table stores split states, checkpoints, and routing information.
Splitting delegates to a strategy such as EventBucketPartitionSplitStrategy. It assigns more event buckets to the same time bucket. For ultra-wide partitions, it caps event buckets to control read amplification. Spreading across partitions still distributes reads over Cassandra replicas.
Validating compares a pre-split checksum against a post-split checksum. A split is marked COMPLETED only when both checksums match. The Netflix team also tracks pre- and post-split partition sizes to confirm splits are sized well.
The Read Path: Bloom Filters and Metadata Routing
TimeSeries servers periodically load completed split partition-keys into in-memory Bloom filters. Every read checks the Bloom filter, which responds in single-digit microseconds. That check is small enough to be practically invisible to callers. On a hit, the server reads wide_row metadata to route the query:
{
"pre_split_data": {
"time_slice": "data_20260328",
"time_series_id": "6313825",
"time_bucket": 0,
"event_bucket": 2
},
"post_split_data": {
"time_slice": "wide_data_20260328_0",
"event_bucket_partition_strategy": {
"target_event_buckets": 2,
"start_event_bucket": 32
}
}Source Read original →Related articles
