Streaming Design Patterns

1. Introduction

Streaming Design Patterns are specialized structures used to process infinite, out-of-order data streams while ensuring correctness and bounded resource utilization.

2. Why This Concept Exists

Infinite data streams cannot be fully aggregated in memory. Design patterns like event-time windowing, watermarking, and garbage collection policies are required to define clear operational boundaries for computations.

3. Code Example

Aggregating stream entries using sliding event-time windows:

import apache_beam as beam
from apache_beam.transforms.window import SlidingWindows
import time

now = time.time()
events = [
    beam.window.TimestampedValue("click", now),
    beam.window.TimestampedValue("click", now + 1),
    beam.window.TimestampedValue("click", now + 5)
]

with beam.Pipeline() as p:
    (p
     | beam.Create(events)
     | "SlidingWindows" >> beam.WindowInto(SlidingWindows(size=4, period=2))
     | "Count" >> beam.CombineGlobally(len).without_defaults()
     | beam.Map(print))

4. Key Takeaways

• Sliding windows calculate running statistics (e.g. moving average over the last 10 minutes, evaluated every minute).
• Watermarks represent processing progress, tracking when a source is unlikely to yield older events.