DoFn
Revision GuideActive Topic
Cheatsheet: DoFn
Recommended reading: 4 mins
Core Description
Define custom element processing logic using the standard DoFn lifecycle.
DoFn Lifecycle Hooks
returns: Generator / YieldPurpose
Manages class lifecycles (setup, start_bundle, process, finish_bundle, teardown) on worker nodes.
Syntax Signature
class MyDoFn(beam.DoFn):Usage Example
import apache_beam as beam
class DBConnectionFn(beam.DoFn):
def setup(self):
# Runs once per class instance initiation (worker start)
self.db_client = "Connected"
def start_bundle(self):
# Runs once per bundle processing window
self.buffer = []
def process(self, element):
# Runs for every element in the collection
yield f"{element}_processed_by_db"
def teardown(self):
# Runs once when worker container shuts down
self.db_client = "Closed"Used In
Optimizing database client pools, metrics tracking, and batch updates.
Common Pitfall
Instantiating heavy API clients or database pools inside process() instead of setup().
Remember:
Group elements into batches inside start_bundle / finish_bundle to minimize network request overhead.
More Free Data Engineering Cheatsheets (DataPlayArena)Interactive syntax references