Side Inputs
Revision GuideActive Topic
Cheatsheet: Side Inputs
Recommended reading: 3 mins
Core Description
Pass supplementary lookup tables and configuration data into transforms.
beam.pvalue.AsDict()
returns: SideInputPurpose
Loads a PCollection as a read-only dictionary lookup side input within a ParDo.
Syntax Signature
beam.pvalue.AsDict(pcollection)Usage Example
import apache_beam as beam
with beam.Pipeline() as p:
meta = p | "Meta" >> beam.Create([("id1", "Active"), ("id2", "Pending")])
meta_dict = beam.pvalue.AsDict(meta)
events = p | "Events" >> beam.Create(["id1", "id2"])
# Pass metadata dictionary as side input parameter
events | "Lookup" >> beam.Map(
lambda ev, m: (ev, m.get(ev, "Unknown")), m=meta_dict
) | beam.Map(print)Expected Stdout / Output
('id1', 'Active')
('id2', 'Pending')Used In
Joining main data streams with smaller metadata lookup tables.
Common Pitfall
Passing side inputs exceeding memory boundaries (>100MB), causing memory exhaustion errors.
Remember:
Beam matches window bounds automatically; the side input window must align with the main element window.
More Free Data Engineering Cheatsheets (DataPlayArena)Interactive syntax references