Custom DoFn & Advanced⏱️ ~25 mins
28. Dead Letter Queue Error Isolation
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Robust production pipelines must isolate malformed records (e.g. invalid integer strings) into a Dead Letter Queue (DLQ) without crashing the entire streaming job.
### Problem Statement
Implement a `DoFn` subclass `SafeParseIntDoFn` that attempts to cast strings to integers. Valid integers are yielded to the main output, while invalid strings that throw `ValueError` are yielded to tagged side output `"dlq"` with `TaggedOutput("dlq", element)`.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
['10', 'bad_record', '20']
Expected Output:
Valid: [10, 20], DLQ: ['bad_record']
Sample Example 2
Input Stream:
['100']
Expected Output:
Valid: [100], DLQ: []
Topics:#ParDo#Error Handling
solution.pyPython 3.11 (Apache Beam)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Input PCollection3 elements
| # | Element / Payload |
|---|---|
| 1 | "10" |
| 2 | "bad_record" |
| 3 | "20" |
Expected Output PCollection
Valid: [10, 20], DLQ: ['bad_record']
Custom DoFn & Advanced⏱️ ~25 mins
28. Dead Letter Queue Error Isolation
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Robust production pipelines must isolate malformed records (e.g. invalid integer strings) into a Dead Letter Queue (DLQ) without crashing the entire streaming job.
### Problem Statement
Implement a `DoFn` subclass `SafeParseIntDoFn` that attempts to cast strings to integers. Valid integers are yielded to the main output, while invalid strings that throw `ValueError` are yielded to tagged side output `"dlq"` with `TaggedOutput("dlq", element)`.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
['10', 'bad_record', '20']
Expected Output:
Valid: [10, 20], DLQ: ['bad_record']
Sample Example 2
Input Stream:
['100']
Expected Output:
Valid: [100], DLQ: []
Topics:#ParDo#Error Handling