PCollection Basics

1. Introduction

2. Why This Concept Exists

3. Key Terminology

• Bounded Dataset: A dataset of a fixed, unchanging size. Example: reading a static log file from storage.
• Unbounded Dataset: An infinite dataset that grows continuously. Example: reading messages from a live streaming queue.
• Immutability: Once a PCollection is created, you cannot add, remove, or modify items inside it. You can only apply a transform to create a new PCollection.

4. How It Works

• Create: Ingest data from files or messaging queues to create an initial PCollection.
• Transform: Apply operations. A PCollection is passed to a transform, which returns a new PCollection.
• No Random Access: You cannot access elements by index (e.g. pcollection[2] is invalid). You must perform operations on all elements as a collection.

5. Visual Diagram

Here is how transforms consume and output PCollections:

6. Code Example

Here is a script demonstrating how PCollections are declared and manipulated:

pythonCopy
import apache_beam as beam

with beam.Pipeline() as p:
    # 1. Ingesting list elements creates a PCollection
    numbers = p | "CreateNumbers" >> beam.Create([10, 20, 30])
    
    # 2. Transforming the PCollection outputs a new PCollection
    doubled = numbers | "DoubleValues" >> beam.Map(lambda x: x * 2)

7. Code Explanation

• numbers is a bounded PCollection containing three integers: 10, 20, and 30.
• numbers | ... feeds the collection into the mapping transform.
• The original numbers collection is untouched (immutability).
• A new collection doubled is returned containing [20, 40, 60].

8. Real Production Example

In a streaming analytics pipeline, a source read operation produces an unbounded PCollection:

pythonCopy
messages = p | "ReadFromQueue" >> beam.io.ReadFromPubSub(subscription="projects/my-gcp-project/subscriptions/my-sub")

This PCollection does not have a fixed size; it updates continuously as new messages are published to the queue.

9. Common Mistakes

• Trying to write loops: You cannot write for item in pcollection: because the dataset is distributed across workers and not stored locally in memory. Use transforms like beam.Map instead.
• Assuming element ordering: Elements in a PCollection are processed in parallel across worker nodes. Their order is not guaranteed.

10. Best Practices

• Always associate schema definitions or class types with PCollection elements.
• Keep records relatively small to avoid overloading worker memory during groupings.

11. Summary

• PCollections represent distributed datasets.
• They are immutable to guarantee parallel execution safety.
• Represented in two shapes: Bounded (batch) and Unbounded (streaming).

12. Interactive Challenges

13. Related Content

• PTransform Basics
• DoFn Deep Dive