Core Transformations⏱️ ~10 mins
6. Inverted Index Key Swapper
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Building distributed inverted indexes (e.g. for Elasticsearch or Bigtable) requires transforming `(document_id, term)` records into `(term, document_id)` tuples.
### Problem Statement
Write a function `swap_kv(input_pcoll)` that receives a `PCollection` of 2-tuples `(Key, Value)` and inverts their positions to produce `(Value, Key)`.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[('A', 1), ('B', 2)]
Expected Output:
[(1, 'A'), (2, 'B')]
Sample Example 2
Input Stream:
[('x', 'y')]
Expected Output:
[('y', 'x')]
Topics:#Map#KV Pairs
solution.pyPython 3.11 (Apache Beam)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Input PCollection
[('A', 1), ('B', 2)]Expected Output PCollection
[(1, 'A'), (2, 'B')]
Core Transformations⏱️ ~10 mins
6. Inverted Index Key Swapper
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Building distributed inverted indexes (e.g. for Elasticsearch or Bigtable) requires transforming `(document_id, term)` records into `(term, document_id)` tuples.
### Problem Statement
Write a function `swap_kv(input_pcoll)` that receives a `PCollection` of 2-tuples `(Key, Value)` and inverts their positions to produce `(Value, Key)`.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[('A', 1), ('B', 2)]
Expected Output:
[(1, 'A'), (2, 'B')]
Sample Example 2
Input Stream:
[('x', 'y')]
Expected Output:
[('y', 'x')]
Topics:#Map#KV Pairs