Pipeline Architectural Topologies

1. Introduction

2. Why This Concept Exists

Real-world enterprise pipelines rarely process data in a single straight line from source to sink. For example:

• A single stream of customer transactions must be sent to an analytics database, checked for fraud, and logged to cold storage simultaneously (Fan-Out).
• Data from multiple regional databases (e.g. US sales and EU sales) must be combined into a single unified dashboard stream (Fan-In).

Understanding pipeline topologies allows engineers to design scalable, non-blocking data workflows.

3. Key Terminology

• Linear Topology: A simple sequential pipeline step sequence ($A \rightarrow B \rightarrow C$).
• Fan-Out (Branching): Applying multiple independent PTransforms to the exact same input PCollection.
• Fan-In (Merging): Combining multiple PCollections of identical data types into a single PCollection using beam.Flatten.

4. How It Works

1. Linear Chain: Output of transform $A$ is piped directly into transform $B$.
2. Fan-Out Execution: A single PCollection variable is referenced as the input for multiple independent transform expressions (pcoll | TransformA and pcoll | TransformB). Beam duplicates metadata references without duplicating underlying data elements in memory.
3. Fan-In Execution: Pass a tuple or list of PCollections to beam.Flatten(): (pcoll1, pcoll2) | beam.Flatten().

5. Visual Diagram

6. Code Example

Below is an example showing both Fan-Out and Fan-In topologies in Python:

pythonCopy
import apache_beam as beam

with beam.Pipeline() as p:
    # Source data
    raw_numbers = p | "CreateNumbers" >> beam.Create([1, 2, 3, 4, 5])

    # 1. FAN-OUT: Branch 1 (Squares)
    squares = raw_numbers | "CalculateSquares" >> beam.Map(lambda x: ("Square", x * x))

    # 2. FAN-OUT: Branch 2 (Cubes)
    cubes = raw_numbers | "CalculateCubes" >> beam.Map(lambda x: ("Cube", x * x * x))

    # 3. FAN-IN: Merge both PCollections back into a single stream
    merged = (squares, cubes) | "MergeStreams" >> beam.Flatten()

    # Log merged output
    merged | "LogResult" >> beam.Map(print)

7. Code Explanation

• squares and cubes both consume raw_numbers. This creates a Fan-Out topology where both transforms execute in parallel across worker nodes.
• (squares, cubes) | beam.Flatten() merges both resulting PCollections into a single unified merged stream without altering individual element data types.

8. Real Production Example

In a real-time banking architecture:

• Incoming transaction events fan out: Branch A validates transactions and writes to BigQuery; Branch B streams high-value transactions to Pub/Sub for immediate push notification alerts.
• Regional transactions from US, EU, and APAC branches fan in via beam.Flatten() to calculate global daily revenue.

9. Common Mistakes

• Attempting to Flatten mismatched data types: Flattening a PCollection of strings with a PCollection of integers causes type errors in downstream transforms.
• Creating circular dependencies: A PCollection cannot depend on its own downstream outputs. DAG graphs must remain strictly acyclic.

10. Best Practices

• Use Fan-Out for independent analytical tasks on the same dataset to maximize parallel cluster utilization.
• Ensure all PCollections passed to beam.Flatten() share an identical schema.

11. Summary

• Fan-Out branches a single PCollection into multiple parallel transform paths.
• Fan-In (beam.Flatten) merges multiple matching PCollections into one.
• Pipeline graphs must always remain Directed Acyclic Graphs (DAGs).

12. Interactive Challenges

13. Related Content

• Streaming Design Patterns
• Composite Transforms