Core Transformations⏱️ ~18 mins
12. Multi-Source User Activity Join
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Data warehouses frequently join disparate stream topics (such as user profile metadata and user engagement scores) on a shared user identifier key.
### Problem Statement
Write a function `cogroup_data(names_pcoll, scores_pcoll)` that accepts two KV collections (`names` and `scores`) and joins them by key using `beam.CoGroupByKey()`.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[('k1', 'Alice')] & [('k1', 99)]
Expected Output:
[('k1', {'names': ['Alice'], 'scores': [99]})]
Sample Example 2
Input Stream:
[('k2', 'Bob')] & [('k2', 80)]
Expected Output:
[('k2', {'names': ['Bob'], 'scores': [80]})]
Topics:#CoGroupByKey#KV Pairs
solution.pyPython 3.11 (Apache Beam)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Input PCollection
[('k1', 'Alice')] & [('k1', 99)]Expected Output PCollection
[('k1', {'names': ['Alice'], 'scores': [99]})]Core Transformations⏱️ ~18 mins
12. Multi-Source User Activity Join
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Data warehouses frequently join disparate stream topics (such as user profile metadata and user engagement scores) on a shared user identifier key.
### Problem Statement
Write a function `cogroup_data(names_pcoll, scores_pcoll)` that accepts two KV collections (`names` and `scores`) and joins them by key using `beam.CoGroupByKey()`.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[('k1', 'Alice')] & [('k1', 99)]
Expected Output:
[('k1', {'names': ['Alice'], 'scores': [99]})]
Sample Example 2
Input Stream:
[('k2', 'Bob')] & [('k2', 80)]
Expected Output:
[('k2', {'names': ['Bob'], 'scores': [80]})]
Topics:#CoGroupByKey#KV Pairs