Core Transformations⏱️ ~10 mins
1. Filtering Even Numbers
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
In IoT sensor data pipelines, telemetry readings with odd error codes represent diagnostic pings, while even-numbered readings represent valid measurement frames.
### Problem Statement
Write a function `filter_evens(input_pcoll)` that filters an incoming `PCollection` of integers, returning only **even numbers** (`num % 2 == 0`).
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[1, 2, 3, 4, 5, 6]
Expected Output:
[2, 4, 6]
Sample Example 2
Input Stream:
[11, 22, 33, 44]
Expected Output:
[22, 44]
Topics:#Filter#Fundamentals
solution.pyPython 3.11 (Apache Beam)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Input PCollection6 elements
| # | Element / Payload |
|---|---|
| 1 | 1 |
| 2 | 2 |
| 3 | 3 |
| 4 | 4 |
| 5 | 5 |
| 6 | 6 |
Expected Output PCollection3 elements
| # | Output Element |
|---|---|
| 1 | 2 |
| 2 | 4 |
| 3 | 6 |
Core Transformations⏱️ ~10 mins
1. Filtering Even Numbers
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
In IoT sensor data pipelines, telemetry readings with odd error codes represent diagnostic pings, while even-numbered readings represent valid measurement frames.
### Problem Statement
Write a function `filter_evens(input_pcoll)` that filters an incoming `PCollection` of integers, returning only **even numbers** (`num % 2 == 0`).
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[1, 2, 3, 4, 5, 6]
Expected Output:
[2, 4, 6]
Sample Example 2
Input Stream:
[11, 22, 33, 44]
Expected Output:
[22, 44]
Topics:#Filter#Fundamentals