Optimizing ML Inference in Databricks: 4 Data Strategies

Machine learning pipelines often face performance bottlenecks when scaling to large datasets. In this article, we explore how to optimize ML inference in Databricks by analyzing four data partitioning strategies. By balancing cluster utilization and inference speed, teams can avoid costly delays and maximize resource efficiency.

The Challenge: Balancing Speed and Resources

When processing 550 million rows across four product lines, inefficient data partitioning can lead to clusters sitting idle. Our initial 420-core cluster spent 10 hours processing just 18 partitions due to uneven data distribution. The solution lies in strategic data organization to leverage parallelism effectively.

Key Optimization Strategies

Here are four approaches we tested to improve ML inference performance:

1. Partitioned Table (Non-Salted)

Basic partitioning by RunDate and ProductLine without additional balancing. While simple, this approach often leads to uneven workloads when data distribution is skewed.

2. Salted Partitioned Table

Adding dynamic salting based on product volume distribution. Larger products like Product D (79.7% of data) receive more partitions, while smaller products get fewer. This creates predictable workloads across the cluster.

3. Liquid-Clustered Table (Non-Salted)

Using Databricks’ liquid clustering for automatic data organization. This method maintains sorted data but lacks explicit partitioning control, which can limit parallelism.

4. Salted Liquid-Clustered Table

Combining liquid clustering with dynamic salting. This approach balances the benefits of sorted data with the workload distribution advantages of salting.

Implementation Details

The optimization process involves:

• Calculating data proportions by ProductLine and AttrB
• Generating salt buckets proportional to data volume
• Enforcing 1 million row limits per partition
• Using repartition() with salt, AttrB, and ProductLine

Performance Results

Our tests revealed:

• 1M row limit consistently outperformed larger partitions
• Salted approaches reduced runtime by 40-60% compared to non-salted
• Liquid clustering improved I/O efficiency but required careful tuning

Why AQE Isn’t Enough

While Spark’s Adaptive Query Execution (AQE) optimizes queries, it doesn’t address ML inference workloads. Our salted partitioning strategy provided better control over parallelism and resource allocation for model inference tasks.

Conclusion: Choosing the Right Strategy

For teams facing similar challenges, we recommend:

1. Start with salted partitioning for skewed datasets
2. Enforce row limits to prevent large partitions
3. Test liquid clustering for sorted data access patterns

By implementing these strategies, we achieved 3x faster inference times while maintaining 95% cluster utilization. The full implementation details and benchmark results are available in our GitHub repository.