Core Transformations⏱️ ~15 mins
11. Fraudulent Transaction Classifier
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Real-time fraud prevention systems continuously monitor stream payloads to flag transactions exceeding safety limits from unverified origins.
### Problem Statement
Write a function `filter_fraud(input_pcoll)` that filters transaction dictionaries, retaining records where:
`amount > 5000` AND `status == 'unverified'`.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[{id: 1, amount: 6000, unverified}, {id: 2, amount: 2000}]
Expected Output:
[1]
Sample Example 2
Input Stream:
[{id: 3, amount: 4000, verified}, {id: 4, amount: 7000, unverified}]
Expected Output:
[4]
Topics:#Filter#Conditionals
solution.pyPython 3.11 (Apache Beam)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Input PCollection
[{id: 1, amount: 6000, unverified}, {id: 2, amount: 2000}]Expected Output PCollection1 elements
| # | Output Element |
|---|---|
| 1 | 1 |
Core Transformations⏱️ ~15 mins
11. Fraudulent Transaction Classifier
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Real-time fraud prevention systems continuously monitor stream payloads to flag transactions exceeding safety limits from unverified origins.
### Problem Statement
Write a function `filter_fraud(input_pcoll)` that filters transaction dictionaries, retaining records where:
`amount > 5000` AND `status == 'unverified'`.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[{id: 1, amount: 6000, unverified}, {id: 2, amount: 2000}]
Expected Output:
[1]
Sample Example 2
Input Stream:
[{id: 3, amount: 4000, verified}, {id: 4, amount: 7000, unverified}]
Expected Output:
[4]
Topics:#Filter#Conditionals