Custom DoFn & Advanced⏱️ ~25 mins
21. Stateful Keyed Running Counter
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Financial ledger streaming engines maintain persistent debit/credit balances across millions of user accounts using keyed state.
### Problem Statement
Implement a stateful `DoFn` subclass `RunningSumDoFn` using `ReadModifyWriteStateSpec("running_sum", VarIntCoder())` from `apache_beam.transforms.userstate` to maintain and emit the cumulative running sum for each incoming key.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[('u1', 5), ('u1', 10), ('u2', 100)]
Expected Output:
[('u1', 5), ('u1', 15), ('u2', 100)]
Sample Example 2
Input Stream:
[('a', 1), ('a', 2), ('a', 3)]
Expected Output:
[('a', 1), ('a', 3), ('a', 6)]
Topics:#Stateful#State API
solution.pyPython 3.11 (Apache Beam)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Input PCollection
[('u1', 5), ('u1', 10), ('u2', 100)]Expected Output PCollection
[('u1', 5), ('u1', 15), ('u2', 100)]Custom DoFn & Advanced⏱️ ~25 mins
21. Stateful Keyed Running Counter
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Financial ledger streaming engines maintain persistent debit/credit balances across millions of user accounts using keyed state.
### Problem Statement
Implement a stateful `DoFn` subclass `RunningSumDoFn` using `ReadModifyWriteStateSpec("running_sum", VarIntCoder())` from `apache_beam.transforms.userstate` to maintain and emit the cumulative running sum for each incoming key.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[('u1', 5), ('u1', 10), ('u2', 100)]
Expected Output:
[('u1', 5), ('u1', 15), ('u2', 100)]
Sample Example 2
Input Stream:
[('a', 1), ('a', 2), ('a', 3)]
Expected Output:
[('a', 1), ('a', 3), ('a', 6)]
Topics:#Stateful#State API