Core Transformations⏱️ ~10 mins
5. High-Value Payment Filter
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Payment processing gateways segregate micro-transactions from standard payment flows by filtering for transactions meeting a minimum threshold.
### Problem Statement
Write a function `filter_threshold(input_pcoll)` that filters transaction amounts, keeping only amounts that are greater than or equal to **100** (`amount >= 100`).
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[50, 100, 150, 20]
Expected Output:
[100, 150]
Sample Example 2
Input Stream:
[101, 99, 200]
Expected Output:
[101, 200]
Topics:#Filter#Finance
solution.pyPython 3.11 (Apache Beam)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Input PCollection4 elements
| # | Element / Payload |
|---|---|
| 1 | 50 |
| 2 | 100 |
| 3 | 150 |
| 4 | 20 |
Expected Output PCollection2 elements
| # | Output Element |
|---|---|
| 1 | 100 |
| 2 | 150 |
Core Transformations⏱️ ~10 mins
5. High-Value Payment Filter
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Payment processing gateways segregate micro-transactions from standard payment flows by filtering for transactions meeting a minimum threshold.
### Problem Statement
Write a function `filter_threshold(input_pcoll)` that filters transaction amounts, keeping only amounts that are greater than or equal to **100** (`amount >= 100`).
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[50, 100, 150, 20]
Expected Output:
[100, 150]
Sample Example 2
Input Stream:
[101, 99, 200]
Expected Output:
[101, 200]
Topics:#Filter#Finance