Core Transformations⏱️ ~10 mins
18. ISO Date Year Extraction
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Partitioning big data tables into annual historical partitions requires extracting the integer year component from incoming ISO `"YYYY-MM-DD"` string timestamps.
### Problem Statement
Write a function `parse_years(input_pcoll)` that parses date strings of format `"YYYY-MM-DD"` and returns the Year as a Python integer.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
['2026-07-02', '2024-01-15']
Expected Output:
[2024, 2026]
Sample Example 2
Input Stream:
['1999-12-31']
Expected Output:
[1999]
Topics:#Map#Dates
solution.pyPython 3.11 (Apache Beam)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Input PCollection2 elements
| # | Element / Payload |
|---|---|
| 1 | "2026-07-02" |
| 2 | "2024-01-15" |
Expected Output PCollection2 elements
| # | Output Element |
|---|---|
| 1 | 2024 |
| 2 | 2026 |
Core Transformations⏱️ ~10 mins
18. ISO Date Year Extraction
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Partitioning big data tables into annual historical partitions requires extracting the integer year component from incoming ISO `"YYYY-MM-DD"` string timestamps.
### Problem Statement
Write a function `parse_years(input_pcoll)` that parses date strings of format `"YYYY-MM-DD"` and returns the Year as a Python integer.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
['2026-07-02', '2024-01-15']
Expected Output:
[2024, 2026]
Sample Example 2
Input Stream:
['1999-12-31']
Expected Output:
[1999]
Topics:#Map#Dates