Windowing & Streaming⏱️ ~25 mins
30. Watermark & Event Timestamp DoFn
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Streaming audit pipelines inspect the watermark-aligned event timestamp metadata on incoming telemetry events to calculate latency lag.
### Problem Statement
Implement a `DoFn` subclass `ExtractTimestampDoFn` that uses `beam.DoFn.TimestampParam` argument injection in `process()` to emit each element's unix event timestamp value as an integer.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
TimestampedValue('A', 12345)
Expected Output:
[12345]
Sample Example 2
Input Stream:
TimestampedValue('B', 999)
Expected Output:
[999]
Topics:#DoFn#Timestamps
solution.pyPython 3.11 (Apache Beam)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Input PCollection
TimestampedValue('A', 12345)Expected Output PCollection1 elements
| # | Output Element |
|---|---|
| 1 | 12345 |
Windowing & Streaming⏱️ ~25 mins
30. Watermark & Event Timestamp DoFn
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Streaming audit pipelines inspect the watermark-aligned event timestamp metadata on incoming telemetry events to calculate latency lag.
### Problem Statement
Implement a `DoFn` subclass `ExtractTimestampDoFn` that uses `beam.DoFn.TimestampParam` argument injection in `process()` to emit each element's unix event timestamp value as an integer.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
TimestampedValue('A', 12345)
Expected Output:
[12345]
Sample Example 2
Input Stream:
TimestampedValue('B', 999)
Expected Output:
[999]
Topics:#DoFn#Timestamps