Aggregations & Grouping⏱️ ~15 mins
15. Top-3 Leaderboard Extractor
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Gaming analytics platforms track top leaderboard scores without executing heavy distributed sorting over terabytes of raw gameplay logs.
### Problem Statement
Write a function `top_three(input_pcoll)` that extracts the **top 3 largest integers** from a PCollection using `beam.combiners.Top.Largest(3)`.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[10, 50, 20, 40, 30]
Expected Output:
[50, 40, 30]
Sample Example 2
Input Stream:
[5, 1, 9]
Expected Output:
[9, 5, 1]
Topics:#Top#Combiners
solution.pyPython 3.11 (Apache Beam)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Input PCollection5 elements
| # | Element / Payload |
|---|---|
| 1 | 10 |
| 2 | 50 |
| 3 | 20 |
| 4 | 40 |
| 5 | 30 |
Expected Output PCollection3 elements
| # | Output Element |
|---|---|
| 1 | 50 |
| 2 | 40 |
| 3 | 30 |
Aggregations & Grouping⏱️ ~15 mins
15. Top-3 Leaderboard Extractor
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Gaming analytics platforms track top leaderboard scores without executing heavy distributed sorting over terabytes of raw gameplay logs.
### Problem Statement
Write a function `top_three(input_pcoll)` that extracts the **top 3 largest integers** from a PCollection using `beam.combiners.Top.Largest(3)`.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[10, 50, 20, 40, 30]
Expected Output:
[50, 40, 30]
Sample Example 2
Input Stream:
[5, 1, 9]
Expected Output:
[9, 5, 1]
Topics:#Top#Combiners