Pub/Sub IO
Revision GuideActive Topic
Cheatsheet: Pub/Sub IO
Recommended reading: 3 mins
Core Description
Integrate with serverless Google Cloud Pub/Sub for messaging.
ReadFromPubSub()
returns: PCollectionPurpose
Reads serverless message publisher queues in real-time.
Syntax Signature
beam.io.ReadFromPubSub(topic=None, subscription=None)Usage Example
import apache_beam as beam
# Read from a Google Cloud Pub/Sub subscription
messages = p | "ReadPubSub" >> beam.io.ReadFromPubSub(
subscription="projects/my-proj/subscriptions/my-sub"
)Used In
Starting point for real-time streaming analytics pipelines.
Common Pitfall
Reading directly from a topic instead of a subscription, which can result in data loss during scaling.
Remember:
Pub/Sub automatically assigns element event-time timestamps based on message publication date.
More Free Data Engineering Cheatsheets (DataPlayArena)Interactive syntax references