Autoscaling

1. Introduction

Autoscaling is a core capability of Google Cloud Dataflow that automatically provisions and de-provisions Compute Engine worker VMs during pipeline execution. Based on real-time execution telemetry, Dataflow increases resources to maintain processing performance during workload spikes and reduces resources when idle to prevent waste.

2. Why This Concept Exists

In traditional server-bound compute clusters, developers face two major capacity management problems:

• Under-provisioning: A cluster is too small for the incoming dataset. Data queues grow, latency spikes, and the pipeline falls behind real-time demands.
• Over-provisioning: To avoid latency spikes, a cluster is sized for peak loads. However, during low-volume periods (such as overnight), expensive servers sit idle, wasting budget.

Dataflow's serverless autoscaling addresses this by adjusting the compute pool size dynamically, aligning infrastructure cost directly with data volume.

3. Key Terminology

• Horizontal Autoscaling: Scaling the system by adding or removing VM instances (workers) from the worker pool.
• Vertical Autoscaling: Scaling the system by increasing or decreasing the hardware specs (CPU, memory) of individual workers (available in Dataflow Prime).
• Liquid Sharding: Dataflow's proprietary mechanism for dynamically re-partitioning remaining work chunks across available workers during batch jobs.
• System Lag: The time difference between the current wall-clock time and the event time of the oldest unprocessed record in the pipeline.

4. How It Works

Dataflow determines scale using different heuristics for batch and streaming pipelines:

• Batch Autoscaling:
  1. Dataflow divides the input data source into dynamic partitions (shards).
  2. As execution proceeds, the control plane calculates the estimated time remaining based on worker throughput.
  3. If adding workers would accelerate completion (up to --max_num_workers), it provisions new VMs and uses Liquid Sharding to dynamically split and redistribute active work files from slower workers to the new ones.
• Streaming Autoscaling:
  1. Dataflow constantly monitors resource indicators: average CPU utilization, data throughput, and System Lag (backlog size).
  2. If system lag rises and CPU limits are saturated, Dataflow provisions additional workers.
  3. If the backlog is cleared and CPU usage drops, idle workers are gracefully terminated.

5. Visual Diagram

6. Code Example

The following python script configures autoscaling options for a pipeline, establishing initial worker pools and maximum resource boundaries:

import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions, StandardOptions, WorkerOptions

def run_autoscaled_pipeline():
    options = PipelineOptions()
    
    # 1. Access Worker and Standard Options
    worker_options = options.view_as(WorkerOptions)
    std_options = options.view_as(StandardOptions)
    
    # 2. Configure execution runner
    std_options.runner = "DataflowRunner"
    
    # 3. Enable throughput-based autoscaling (default)
    worker_options.autoscaling_algorithm = "THROUGHPUT_BASED"
    
    # 4. Configure worker boundaries
    worker_options.num_workers = 2       # Initial number of VMs to spin up at launch
    worker_options.max_num_workers = 15   # Upper limit of VMs to prevent runaway budget costs
    
    # 5. Define pipeline logic
    with beam.Pipeline(options=options) as p:
        (p
         | "ReadLogStream" >> beam.io.ReadFromText("gs://my-bucket/inputs/*.log")
         | "ParseLogs" >> beam.Map(lambda log: log.split(","))
         | "FilterErrors" >> beam.Filter(lambda fields: fields[0] == "ERROR")
         | "WriteErrors" >> beam.io.WriteToText("gs://my-bucket/outputs/errors")
        )

if __name__ == "__main__":
    run_autoscaled_pipeline()

7. Code Explanation

• autoscaling_algorithm = "THROUGHPUT_BASED" tells Dataflow to monitor incoming data rates and CPU loads to dynamically scale the worker VM count.
• num_workers = 2 ensures the job initializes with 2 workers immediately, avoiding start latency.
• max_num_workers = 15 defines the safety boundary. Dataflow will never scale the cluster beyond 15 nodes, shielding the user from unexpected GCP charges.

8. Real Production Example

A delivery logistics company tracks live driver GPS coordinates using a streaming Dataflow pipeline.

• Normal hours: GPS events average 500/sec; Dataflow runs stably on 2 worker VMs.
• Rush hour spike: At 5:00 PM, GPS events spike to 20,000/sec. Backlog lag increases.
• Autoscale response: Dataflow senses the backlog growth and CPU load, scaling the cluster up to 10 worker VMs in 5 minutes, keeping processing latency under 3 seconds.
• Night cool-down: At midnight, event rates drop. Dataflow scales the cluster back down to 2 worker VMs.

9. Common Mistakes

• No Maximum Cap: Leaving max_num_workers unset. If the pipeline encounters a logic bug (e.g. an infinite retry loop or massive data loop), Dataflow might attempt to autoscale to its maximum organizational limit (e.g. 100+ instances), resulting in a sudden billing spike.
• Non-Splittable Inputs in Batch: Reading a single, compressed file (like a massive .tar.gz file) that cannot be parallelized. Dataflow cannot shard a single non-splittable file, so the job runs on a single worker VM, rendering autoscaling useless.

10. Interview Perspective

• Question: Why does batch autoscaling use "Liquid Sharding" while streaming does not?
• Answer: Batch data size is finite and known at launch. Dataflow can split files dynamically and move shards to balance execution. Streaming data is infinite and continuous; it cannot be partitioned beforehand. Streaming instead scales based on active key partition distribution and queue backlog.
• Question: If my streaming pipeline lag is rising but CPU usage is low (e.g. 20%), why isn't Dataflow autoscaling to resolve the lag?
• Answer: Autoscaling is throttled if the bottleneck is not CPU-bound. If your code is blocked waiting for external database writes (I/O block), adding more workers won't improve throughput. Dataflow avoids scaling up under these conditions.

11. Best Practices

• Always set max_num_workers to establish a budget ceiling.
• Ensure input files are stored in splittable formats (like Snappy-compressed Avro, Parquet, or uncompressed Text) to leverage liquid sharding.
• Use Streaming Engine for streaming jobs; it speeds up scaling times by removing local state-shuffling delays.

12. Summary

• Autoscaling adjusts worker VM counts dynamically.
• Batch scaling leverages Liquid Sharding to balance partitions.
• Streaming scaling uses CPU, throughput, and system lag metrics.
• max_num_workers caps execution resources to control costs.

13. Interactive Challenges

14. Related Content

• Worker Configuration
• Performance Optimization
• Introduction to Dataflow