Aggregations & Grouping⏱️ ~12 mins
7. Word Frequency Aggregator
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Real-time search trend monitoring calculates the volume of incoming search query tokens across distributed workers.
### Problem Statement
Write a function `count_words(input_pcoll)` that takes a `PCollection` of word strings and counts the total frequency of each word, outputting `(word, count)` tuples using `beam.CombinePerKey(sum)`.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
['apple', 'banana', 'apple']
Expected Output:
[('apple', 2), ('banana', 1)]
Sample Example 2
Input Stream:
['beam', 'beam', 'beam']
Expected Output:
[('beam', 3)]
Topics:#CombinePerKey#KV Pairs
solution.pyPython 3.11 (Apache Beam)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Input PCollection3 elements
| # | Element / Payload |
|---|---|
| 1 | "apple" |
| 2 | "banana" |
| 3 | "apple" |
Expected Output PCollection
[('apple', 2), ('banana', 1)]Aggregations & Grouping⏱️ ~12 mins
7. Word Frequency Aggregator
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Real-time search trend monitoring calculates the volume of incoming search query tokens across distributed workers.
### Problem Statement
Write a function `count_words(input_pcoll)` that takes a `PCollection` of word strings and counts the total frequency of each word, outputting `(word, count)` tuples using `beam.CombinePerKey(sum)`.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
['apple', 'banana', 'apple']
Expected Output:
[('apple', 2), ('banana', 1)]
Sample Example 2
Input Stream:
['beam', 'beam', 'beam']
Expected Output:
[('beam', 3)]
Topics:#CombinePerKey#KV Pairs