BeamPlayArena Curriculum Roadmap

Follow our structured pathway to learn Apache Beam and Google Cloud Dataflow from scratch. Choose a module from the curriculum below or select a lesson in the sidebar.

Interactive Hands-On Practice Bypasses Setup!

Put your knowledge to the test inside the browser

Don't just read about data pipelines—build them. Write Beam scripts in our Playground or challenge yourself on 40 sandbox exercises evaluated via WebAssembly.

1. Fundamentals

14 structured lessons in this unit.

Introduction to Apache Beam Why Apache Beam?Batch vs Streaming Beam Programming Model Apache Beam Architecture Installation & Environment Setup Creating Your First Beam Pipeline Pipeline Basics Project Structure Pipeline Options Pipeline Runners Direct Runner Dataflow Runner (Introduction)Running Your First Pipeline

2. Core Concepts

12 structured lessons in this unit.

Pipeline PCollection PTransform Create Transform Read Transform Write Transform Type Hints Coders Serialization Side Inputs Side Outputs Tagged Outputs

3. Transformations

18 structured lessons in this unit.

Map FlatMap Filter Distinct Flatten Partition Count Mean Top Sample GroupByKey CombinePerKey CombineGlobally Custom CombineFn DoFn ParDo CoGroupByKey Aggregations

4. Windowing

16 structured lessons in this unit.

Why Windowing?Windowing Overview Event Time Processing Time Timestamp Assignment Global Window Fixed Windows Sliding Windows Session Windows Custom Windows Watermarks Allowed Lateness Late Data Triggers Trigger Types Accumulation Modes

5. Streaming

13 structured lessons in this unit.

Streaming Basics Bounded vs Unbounded Pub/Sub Streaming Kafka Streaming Handling Out-of-Order Events Deduplication Streaming Aggregations Stateful Processing State API Timers Fault Tolerance Exactly Once Processing Checkpointing Concepts

6. IO Connectors

20 structured lessons in this unit.

File IO TextIO CSV Files JSON Files Avro Parquet Cloud Storage Google Cloud Storage Amazon S3 Messaging Pub/Sub IO Kafka IO Databases BigQuery IO JDBC IO Bigtable IO Elasticsearch IO MongoDB IO Others Custom IO

7. Google Cloud Dataflow

11 structured lessons in this unit.

Introduction to Dataflow Dataflow Architecture Direct Runner vs Dataflow Runner Pipeline Templates Flex Templates Runtime Parameters Monitoring Jobs Logs Autoscaling Worker Configuration Best Practices

8. Beam Patterns & Design

8 structured lessons in this unit.

Composite Transforms Reusable Transforms Error Handling Patterns Dead Letter Queue (DLQ)Side Input Patterns Join Patterns Streaming Design Patterns Pipeline Design Patterns

9. Production & Best Practices

13 structured lessons in this unit.

Pipeline Performance Tuning Cost Optimization Debugging Pipelines Logging & Monitoring Metrics Testing Beam Pipelines Unit Testing Integration Testing Deployment Strategies CI/CD for Beam Common Production Issues Scaling Pipelines Security Best Practices