Custom DoFn & Advanced⏱️ ~25 mins
22. Stateful User Session History
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Real-time recommendation engines maintain the sequence of recent page interactions per user session to serve personalized product suggestions.
### Problem Statement
Implement a stateful `DoFn` subclass `ListAccDoFn` using `ReadModifyWriteStateSpec("seen_items", FastPrimitivesCoder())` from `apache_beam.transforms.userstate` to accumulate all seen values for a given key into a persistent list.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[('k1', 'A'), ('k1', 'B')]
Expected Output:
[('k1', ['A']), ('k1', ['A', 'B'])]
Sample Example 2
Input Stream:
[('k2', 'X')]
Expected Output:
[('k2', ['X'])]
Topics:#Stateful#State API
solution.pyPython 3.11 (Apache Beam)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Input PCollection
[('k1', 'A'), ('k1', 'B')]Expected Output PCollection
[('k1', ['A']), ('k1', ['A', 'B'])]Custom DoFn & Advanced⏱️ ~25 mins
22. Stateful User Session History
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Real-time recommendation engines maintain the sequence of recent page interactions per user session to serve personalized product suggestions.
### Problem Statement
Implement a stateful `DoFn` subclass `ListAccDoFn` using `ReadModifyWriteStateSpec("seen_items", FastPrimitivesCoder())` from `apache_beam.transforms.userstate` to accumulate all seen values for a given key into a persistent list.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[('k1', 'A'), ('k1', 'B')]
Expected Output:
[('k1', ['A']), ('k1', ['A', 'B'])]
Sample Example 2
Input Stream:
[('k2', 'X')]
Expected Output:
[('k2', ['X'])]
Topics:#Stateful#State API