Map Transform

1. Introduction

2. Why This Concept Exists

Raw data is rarely in the format you need. You often need to perform simple conversions on every single record—such as parsing strings, converting dates, or capitalizing text. The Map transform does this in parallel, dividing the work across all available worker machines.

3. Key Terminology

• One-to-One Mapping: If you feed 10 elements into a Map transform, you will get exactly 10 elements out.
• Predicate / Mapper Function: The function that contains your conversion logic.
• Lambda Function: A small, quick one-line function written in Python (e.g. lambda x: x * 2).

4. How It Works

• The mapper function is loaded onto worker nodes.
• Each element in the PCollection is fed into the function.
• The function runs and outputs the new element.
• The results are packaged into a new PCollection.

5. Visual Diagram

Here is how elements map individually inside the pipeline:

6. Code Example

Converting temperatures from Celsius to Fahrenheit:

pythonCopy
import apache_beam as beam

def c_to_f(celsius):
    return (celsius * 9/5) + 32

with beam.Pipeline() as p:
    temps_c = p | "CreateTemps" >> beam.Create([0, 10, 20])
    temps_f = temps_c | "ConvertToF" >> beam.Map(c_to_f)

7. Code Explanation

• temps_c contains three Celsius readings: 0, 10, and 20.
• We pass the named function c_to_f to beam.Map.
• Beam runs this function on each number, returning the Fahrenheit values: [32.0, 50.0, 68.0].

8. Real Production Example

pythonCopy
rounded_amounts = raw_amounts | "RoundPrices" >> beam.Map(lambda price: round(price, 2))

9. Common Mistakes

• Returning lists: If your mapping function returns a list, Map outputs nested lists (e.g. [[1, 2], [3]]). If you want to expand lists into individual records, use FlatMap.
• Updating shared counters: Never try to increment a local variable inside a Map transform. The code runs on distributed worker nodes with no shared memory state.

10. Best Practices

• Keep mapping functions pure (no side effects like writing to databases or querying external web APIs).
• Add error logging blocks to intercept dirty records safely.

11. Summary

• Applies a 1-to-1 conversion function.
• Runs concurrently across all elements.
• Outputs exactly one record per input record.

12. Interactive Challenges

13. Related Content

• FlatMap Transform
• Filter Elements