Aggregations & Grouping⏱️ ~15 mins
16. Grouped CSV String Aggregator
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
E-commerce order summarizers aggregate item IDs into a comma-delimited string grouped by customer ID.
### Problem Statement
Write a function `concat_strings(input_pcoll)` that groups string values by key and concatenates them with a comma delimiter (e.g. `('A', 'x')` and `('A', 'y')` -> `('A', 'x,y')`).
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[('A', 'x'), ('A', 'y')]
Expected Output:
[('A', 'x,y')]
Sample Example 2
Input Stream:
[('B', '1'), ('B', '2')]
Expected Output:
[('B', '1,2')]
Topics:#CombinePerKey#KV Pairs
solution.pyPython 3.11 (Apache Beam)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Input PCollection
[('A', 'x'), ('A', 'y')]Expected Output PCollection
[('A', 'x,y')]Aggregations & Grouping⏱️ ~15 mins
16. Grouped CSV String Aggregator
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
E-commerce order summarizers aggregate item IDs into a comma-delimited string grouped by customer ID.
### Problem Statement
Write a function `concat_strings(input_pcoll)` that groups string values by key and concatenates them with a comma delimiter (e.g. `('A', 'x')` and `('A', 'y')` -> `('A', 'x,y')`).
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[('A', 'x'), ('A', 'y')]
Expected Output:
[('A', 'x,y')]
Sample Example 2
Input Stream:
[('B', '1'), ('B', '2')]
Expected Output:
[('B', '1,2')]
Topics:#CombinePerKey#KV Pairs