Core Transformations⏱️ ~10 mins
4. User Identity Enrichment
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
During CRM ingestion, usernames must be formatted and tagged with a verification label before publishing to customer-facing analytical dashboards.
### Problem Statement
Write a function `enrich_users(input_pcoll)` that takes a `PCollection` of username strings (e.g. `"Alice"`) and transforms each into the format: `"Name: <name> [Verified]"`.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
['Alice', 'Bob']
Expected Output:
['Name: Alice [Verified]', 'Name: Bob [Verified]']
Sample Example 2
Input Stream:
['Charlie']
Expected Output:
['Name: Charlie [Verified]']
Topics:#Map#Formatting
solution.pyPython 3.11 (Apache Beam)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Input PCollection2 elements
| # | Element / Payload |
|---|---|
| 1 | "Alice" |
| 2 | "Bob" |
Expected Output PCollection2 elements
| # | Output Element |
|---|---|
| 1 | "Name: Alice [Verified]" |
| 2 | "Name: Bob [Verified]" |
Core Transformations⏱️ ~10 mins
4. User Identity Enrichment
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
During CRM ingestion, usernames must be formatted and tagged with a verification label before publishing to customer-facing analytical dashboards.
### Problem Statement
Write a function `enrich_users(input_pcoll)` that takes a `PCollection` of username strings (e.g. `"Alice"`) and transforms each into the format: `"Name: <name> [Verified]"`.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
['Alice', 'Bob']
Expected Output:
['Name: Alice [Verified]', 'Name: Bob [Verified]']
Sample Example 2
Input Stream:
['Charlie']
Expected Output:
['Name: Charlie [Verified]']
Topics:#Map#Formatting