Streaming Architecture Design Patterns

1. Introduction

2. Why This Concept Exists

Unlike batch data where dataset boundaries are known ahead of time, streaming data arrives continuously 24/7.

Key streaming challenges solved by design patterns:

• Infinite Datasets: Computations cannot wait for "all data" to arrive before calculating totals.
• Out-of-Order Events: Mobile app events arrive seconds or minutes late due to poor cellular network coverage.
• Moving Statistics: Applications require rolling updates (e.g. 5-minute moving average calculated every 10 seconds).

3. Key Terminology

• Sliding Window Pattern: Calculating rolling window metrics (e.g. size 10 minutes, sliding every 1 minute) where events belong to multiple overlapping windows.
• Session Window Pattern: Grouping user actions into dynamic windows based on periods of inactivity (e.g. user activity sessions with a 15-minute gap threshold).
• Late Data Routing: Diverting late-arriving events that arrive after a window's watermark has passed to a dedicated side output.

4. How It Works

1. Timestamp Assignment: Parse event timestamps from raw payload metadata.
2. Window Into: Apply SlidingWindows or Sessions windowing functions.
3. Watermark Tracking: The runner tracks time progress using watermarks. When the watermark passes a window's end time, the window triggers.
4. Emit Panes: The triggered pane emits window totals downstream.

5. Visual Diagram

6. Code Example

Below is a complete Python pipeline calculating a rolling moving average over sliding event-time windows:

pythonCopy
import apache_beam as beam
from apache_beam.transforms.window import SlidingWindows
import time

# Create timestamped streaming elements
now = time.time()
events = [
    beam.window.TimestampedValue(("sensor-1", 72.5), now),
    beam.window.TimestampedValue(("sensor-1", 75.0), now + 10),
    beam.window.TimestampedValue(("sensor-1", 78.2), now + 30)
]

with beam.Pipeline() as p:
    (p
     | "CreateStream" >> beam.Create(events)
     # 10-minute window size, evaluated every 1 minute
     | "SlidingWindow" >> beam.WindowInto(SlidingWindows(size=600, period=60))
     | "CalculateAvg" >> beam.CombinePerKey(lambda values: sum(values) / len(values))
     | "PrintMovingAvg" >> beam.Map(print))

7. Code Explanation

• SlidingWindows(size=600, period=60) creates a 10-minute window ($600$s) that slides every 1 minute ($60$s).
• Each temperature reading is assigned to multiple overlapping 10-minute windows, recalculating moving averages dynamically.

8. Real Production Example

In a fraud-detection streaming application for credit card transactions:

• Transactions are evaluated using a 5-minute sliding window sliding every 5 seconds.
• If a card generates more than 5 transactions across different physical locations within the 5-minute sliding window, the pipeline triggers an immediate security lock alert.

9. Common Mistakes

• Setting tiny slide periods with massive window sizes: Creating a 24-hour window sliding every 1 second creates 86,400 overlapping windows per element, exhausting runner memory.
• Confusing Event Time with Processing Time: Calculating moving averages using system clock time (Processing Time) causes incorrect calculations when processing late data.

10. Best Practices

• Use sliding windows for continuous dashboard metrics and trend monitoring.
• Use without_defaults() when combining empty global windows to avoid emitting null values.

11. Summary

• Sliding windows calculate rolling metrics over overlapping time frames.
• Session windows group user events based on periods of inactivity.
• Event-time processing ensures correct calculations even when network delays cause late data arrival.

12. Interactive Challenges

13. Related Content

• Sliding Windows
• Session Windows