Databricks adds on-demand Spark state repartitioning

Databricks has introduced a public preview of on-demand state repartitioning for Apache Spark Structured Streaming. Previously, developers running stateful queries were locked into the partition count established at checkpoint creation, often defaulting to 200 shuffle partitions. Changing this number required abandoning the checkpoint and losing accumulated state, which is highly disruptive for long-running pipelines like fraud detection models or sessionization jobs. The new capability allows users to resize partitions dynamically during a restart without losing any historical state.

To use the feature, developers must run Databricks Runtime 18 or above with the RocksDB state store provider, which has been the default since version 17.3. Instead of modifying the standard shuffle parameter, users set the new spark.sql.streaming.stateStore.partitions configuration and restart the query. Upon restarting, the system completes any pending microbatch and performs a physical redistribution of the state data across the new partition count. Practitioners can monitor this process using the controlBatch.REPARTITION metric, measured in milliseconds, within the standard query progress events.

This update changes how practitioners manage fluctuating workloads, backfills, and post-launch optimization. For instance, a pipeline can be scaled down from 200 to 100 partitions to reduce coordination overhead, or scaled up to handle sudden traffic spikes. Early adopters are already seeing financial benefits. Alexis Chicoine, a developer at Coveo, noted that the feature helped the company "cut our related Amazon S3 API costs by 40%" by eliminating the need to overprovision or rebuild checkpoints.