Core Transformations⏱️ ~12 mins
9. Deterministic Polarity Partitioning
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Risk telemetry systems route positive credit balances to asset accounts and negative debit values to liability workflows for independent processing.
### Problem Statement
Write a function `partition_numbers(input_pcoll)` that splits a stream of integers into two partition channels:
- **Partition 0**: non-negative values (`val >= 0`)
- **Partition 1**: negative values (`val < 0`)
Return a list of two PCollections: `[pos_pcoll, neg_pcoll]`.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[10, -5, 20, -1]
Expected Output:
P0: [10, 20], P1: [-5, -1]
Sample Example 2
Input Stream:
[-10, -20]
Expected Output:
P0: [], P1: [-20, -10]
Topics:#Partition#Routing
solution.pyPython 3.11 (Apache Beam)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Input PCollection4 elements
| # | Element / Payload |
|---|---|
| 1 | 10 |
| 2 | -5 |
| 3 | 20 |
| 4 | -1 |
Expected Output PCollection
P0: [10, 20], P1: [-5, -1]
Core Transformations⏱️ ~12 mins
9. Deterministic Polarity Partitioning
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Risk telemetry systems route positive credit balances to asset accounts and negative debit values to liability workflows for independent processing.
### Problem Statement
Write a function `partition_numbers(input_pcoll)` that splits a stream of integers into two partition channels:
- **Partition 0**: non-negative values (`val >= 0`)
- **Partition 1**: negative values (`val < 0`)
Return a list of two PCollections: `[pos_pcoll, neg_pcoll]`.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[10, -5, 20, -1]
Expected Output:
P0: [10, 20], P1: [-5, -1]
Sample Example 2
Input Stream:
[-10, -20]
Expected Output:
P0: [], P1: [-20, -10]
Topics:#Partition#Routing