Custom DoFn & Advanced⏱️ ~18 mins
19. Multi-Output Tagged Branch Routing
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Real-time routing pipelines split high-priority alerts from standard informational events into isolated destination queues in a single compute pass.
### Problem Statement
Implement a `DoFn` subclass `SplitOddsFn` that routes **even integers** to the main output, and **odd integers** to a tagged side output `"odds"` using `beam.pvalue.TaggedOutput`.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[1, 2, 3, 4]
Expected Output:
Evens: [2, 4], Odds: [1, 3]
Sample Example 2
Input Stream:
[10, 20]
Expected Output:
Evens: [10, 20], Odds: []
Topics:#ParDo#Tagged Outputs
solution.pyPython 3.11 (Apache Beam)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Input PCollection4 elements
| # | Element / Payload |
|---|---|
| 1 | 1 |
| 2 | 2 |
| 3 | 3 |
| 4 | 4 |
Expected Output PCollection
Evens: [2, 4], Odds: [1, 3]
Custom DoFn & Advanced⏱️ ~18 mins
19. Multi-Output Tagged Branch Routing
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Real-time routing pipelines split high-priority alerts from standard informational events into isolated destination queues in a single compute pass.
### Problem Statement
Implement a `DoFn` subclass `SplitOddsFn` that routes **even integers** to the main output, and **odd integers** to a tagged side output `"odds"` using `beam.pvalue.TaggedOutput`.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[1, 2, 3, 4]
Expected Output:
Evens: [2, 4], Odds: [1, 3]
Sample Example 2
Input Stream:
[10, 20]
Expected Output:
Evens: [10, 20], Odds: []
Topics:#ParDo#Tagged Outputs