Core Transformations⏱️ ~12 mins
13. Flatten Nested Record Batches
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Batch micro-services often emit nested arrays of sensor payloads that must be flattened into an individual element stream for event-driven downstream consumers.
### Problem Statement
Write a function `flatten_lists(input_pcoll)` that takes a `PCollection` of string lists (e.g. `[['A', 'B'], ['C']]`) and flattens them into a single continuous stream of strings.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[['A', 'B'], ['C']]
Expected Output:
['A', 'B', 'C']
Sample Example 2
Input Stream:
[['X'], ['Y']]
Expected Output:
['X', 'Y']
Topics:#FlatMap#Fundamentals
solution.pyPython 3.11 (Apache Beam)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Input PCollection2 elements
| # | Element / Payload |
|---|---|
| 1 | ["A","B"] |
| 2 | ["C"] |
Expected Output PCollection3 elements
| # | Output Element |
|---|---|
| 1 | "A" |
| 2 | "B" |
| 3 | "C" |
Core Transformations⏱️ ~12 mins
13. Flatten Nested Record Batches
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Batch micro-services often emit nested arrays of sensor payloads that must be flattened into an individual element stream for event-driven downstream consumers.
### Problem Statement
Write a function `flatten_lists(input_pcoll)` that takes a `PCollection` of string lists (e.g. `[['A', 'B'], ['C']]`) and flattens them into a single continuous stream of strings.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[['A', 'B'], ['C']]
Expected Output:
['A', 'B', 'C']
Sample Example 2
Input Stream:
[['X'], ['Y']]
Expected Output:
['X', 'Y']
Topics:#FlatMap#Fundamentals