Unit Testing

1. Introduction

Unit Testing in Apache Beam focuses on isolating and testing individual processing functions (DoFn classes or lambda maps) rather than testing the entire pipeline graph.

2. Why This Concept Exists

If a bug occurs in a custom processing step, running the entire pipeline to verify the fix is slow. Isolating DoFns allows developers to verify business logic, state APIs, and edge cases in milliseconds using standard test runners.

3. Code Example

Testing an isolated DoFn class:

import unittest
import apache_beam as beam
from apache_beam.testing.test_pipeline import TestPipeline
from apache_beam.testing.util import assert_that, equal_to

class DoubleDoFn(beam.DoFn):
    def process(self, element):
        yield element * 2

class TestUnitDoFn(unittest.TestCase):
    def test_do_fn(self):
        with TestPipeline() as p:
            out = (p 
                   | beam.Create([5, 10])
                   | beam.ParDo(DoubleDoFn()))
            assert_that(out, equal_to([10, 20]))

4. Key Takeaways

• Unit tests should mock external data calls entirely.
• Keep test datasets small and fast to execute.