Lab: Real-Time Dashboard Pipeline

1. Business Scenario

Calculate continuous dashboard aggregations with dynamic triggers.

2. Input Dataset (\`dataset.csv\`)

Save the following raw rows locally as \`dataset.csv\` to test your pipeline:

timestamp,metric_name,value
1719830400,cpu_util,45.2
1719830405,cpu_util,48.0
1719830412,cpu_util,55.1
1719830420,cpu_util,60.5

3. Starter Code Skeleton

Create a local file named \`starter.py\` and copy the following skeleton. Complete the missing transformations:

# starter.py - Real-Time Dashboard Pipeline
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions

def run_pipeline():
    options = PipelineOptions()
    with beam.Pipeline(options=options) as p:
        # TODO: Set sliding windows (600s length, 60s slide)
        # TODO: Setup custom trigger logic
        pass

if __name__ == "__main__":
    run_pipeline()

4. Lab Requirements

• Assign event-time timestamps to telemetry logs.
• Apply 10-minute sliding windows (sliding every 1 minute).
• Configure triggers to fire early updates before watermarks pass when values exceed 50.0.

5. Step-by-Step Guide & Solution