Core Transformations⏱️ ~15 mins
27. JSON Payload Serialization
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Streaming data export pipelines serialize internal Python dictionary objects into sorted JSON formatted strings before publishing to Pub/Sub or Kafka topics.
### Problem Statement
Write a function `dict_to_json(input_pcoll)` that converts each dictionary in a PCollection to a deterministic JSON string representation using `json.dumps(record, sort_keys=True)`.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[{'a': 1}]
Expected Output:
['{"a": 1}']
Sample Example 2
Input Stream:
[{'b': 2, 'c': 3}]
Expected Output:
['{"b": 2, "c": 3}']
Topics:#Map#JSON
solution.pyPython 3.11 (Apache Beam)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Input PCollection1 elements
| # | Element / Payload |
|---|---|
| 1 | {"a":1} |
Expected Output PCollection
['{"a": 1}']Core Transformations⏱️ ~15 mins
27. JSON Payload Serialization
Enterprise Architecture Context
In production stream-processing architectures (Google Cloud Dataflow / Flink), pipeline stages must handle parallel transformations without data loss, managing schema mutations and aggregations across distributed worker workers.
Problem Statement
### Business Context
Streaming data export pipelines serialize internal Python dictionary objects into sorted JSON formatted strings before publishing to Pub/Sub or Kafka topics.
### Problem Statement
Write a function `dict_to_json(input_pcoll)` that converts each dictionary in a PCollection to a deterministic JSON string representation using `json.dumps(record, sort_keys=True)`.
Key Learning Objectives
- Understand distributed Apache Beam execution DAG stages and pipeline lifecycle.
- Apply idiomatic functional Python transforms using the pipe operator
|. - Ensure data consistency and idempotency across distributed stream workers.
Sample Data Fixtures
Sample Example 1
Input Stream:
[{'a': 1}]
Expected Output:
['{"a": 1}']
Sample Example 2
Input Stream:
[{'b': 2, 'c': 3}]
Expected Output:
['{"b": 2, "c": 3}']
Topics:#Map#JSON