Aggregations & Grouping⏱️ ~25 mins
29. Keyed Min & Max Aggregator
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Weather sensor grids compute the daily temperature range by aggregating the minimum and maximum observed temperatures for each device station ID.
### Problem Statement
Implement a custom `beam.CombineFn` or function to compute `(min_temp, max_temp)` for each key using `beam.CombinePerKey()`, returning `(device_id, (min_temp, max_temp))` tuples.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[('sensor1', 10), ('sensor1', 25), ('sensor1', 5)]
Expected Output:
[('sensor1', (5, 25))]
Sample Example 2
Input Stream:
[('sensor2', 50)]
Expected Output:
[('sensor2', (50, 50))]
Topics:#CombinePerKey#CombineFn
solution.pyPython 3.11 (Apache Beam)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Input PCollection
[('sensor1', 10), ('sensor1', 25), ('sensor1', 5)]Expected Output PCollection
[('sensor1', (5, 25))]Aggregations & Grouping⏱️ ~25 mins
29. Keyed Min & Max Aggregator
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Weather sensor grids compute the daily temperature range by aggregating the minimum and maximum observed temperatures for each device station ID.
### Problem Statement
Implement a custom `beam.CombineFn` or function to compute `(min_temp, max_temp)` for each key using `beam.CombinePerKey()`, returning `(device_id, (min_temp, max_temp))` tuples.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[('sensor1', 10), ('sensor1', 25), ('sensor1', 5)]
Expected Output:
[('sensor1', (5, 25))]
Sample Example 2
Input Stream:
[('sensor2', 50)]
Expected Output:
[('sensor2', (50, 50))]
Topics:#CombinePerKey#CombineFn