Custom DoFn & Advanced⏱️ ~18 mins
20. Global Broadcast Side Input Share
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Market basket analytics calculates the percentage revenue contribution of each item SKU relative to the total store revenue passed as a broadcast Side Input.
### Problem Statement
Write a function `calc_share(input_pcoll, total_side_input)` that receives an item count PCollection and a singleton side input containing the total sum, calculating the percentage share `(item / total) * 100.0`.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
Items: [20, 80], Total: 100
Expected Output:
[20.0, 80.0]
Sample Example 2
Input Stream:
Items: [50], Total: 200
Expected Output:
[25.0]
Topics:#Side Inputs#Broadcast
solution.pyPython 3.11 (Apache Beam)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Input PCollection
Items: [20, 80], Total: 100
Expected Output PCollection2 elements
| # | Output Element |
|---|---|
| 1 | 20 |
| 2 | 80 |
Custom DoFn & Advanced⏱️ ~18 mins
20. Global Broadcast Side Input Share
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Market basket analytics calculates the percentage revenue contribution of each item SKU relative to the total store revenue passed as a broadcast Side Input.
### Problem Statement
Write a function `calc_share(input_pcoll, total_side_input)` that receives an item count PCollection and a singleton side input containing the total sum, calculating the percentage share `(item / total) * 100.0`.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
Items: [20, 80], Total: 100
Expected Output:
[20.0, 80.0]
Sample Example 2
Input Stream:
Items: [50], Total: 200
Expected Output:
[25.0]
Topics:#Side Inputs#Broadcast