Scaling Pipelines

1. Introduction

Scaling Pipelines is the practice of adjusting runner configurations, cluster sizing, and parallel sharding to handle massive spikes in transaction throughput.

2. Why This Concept Exists

High throughput (e.g., 500k events/second) requires clusters to adjust rapidly. Understanding autoscaling parameters, hot keys, and network sharding is vital to prevent pipeline stalls.

3. Core Tuning Concepts

• Dynamic Work Rebalancing: The runner shifts work ranges dynamically from laggy workers to active workers (autoscaling).
• Hot Key Mitigation: Avoid mapping large batches to single processing keys. Use key salting to partition values.