Flex Templates

1. Introduction

Dataflow Flex Templates represent the modern standard for packaging, sharing, and running reusable Apache Beam pipelines on Google Cloud. By containerizing your pipeline code and its execution environment using Docker, Flex Templates separate pipeline development from pipeline deployment.

2. Why This Concept Exists

Before Flex Templates, Dataflow relied on Classic Templates. While useful, Classic Templates had major limitations:

• Static Graph Compilation: The execution graph (DAG) was constructed when the developer created the template. This meant you could not use dynamic branching, dynamic sources, or customize step structures based on runtime parameters.
• Dependency Management Issues: Classic templates packaged dependencies in local archives. Version mismatches between the submission environment and worker VMs often caused execution errors.

Flex Templates solve these problems by packaging the pipeline's execution logic, dependencies, and environment inside a Docker image. Instead of staging a pre-compiled graph, the Dataflow service runs the container at launch time to construct the execution graph dynamically, enabling flexible, parameter-driven pipelines.

3. Key Terminology

• Flex Template Spec File: A small JSON file stored in Google Cloud Storage that links to the template's Docker image and defines its runtime parameters.
• SDK Container Image: The Docker image hosting your Python/Java code, runtime environment, custom packages, and the Apache Beam SDK.
• Launch API: The Dataflow API endpoint used to instantiate a job from a template, allowing orchestration tools (like Airflow or Cloud Scheduler) to trigger runs.
• Graph Compilation: The step where the pipeline code runs and generates the physical execution DAG. For Flex Templates, this happens on a managed launcher VM in GCP during launch.

4. How It Works

The lifecycle of building and running a Flex Template involves:

1. Develop: Write your Apache Beam pipeline code, exposing runtime options as custom command-line arguments.
2. Containerize: Write a Dockerfile that copies your code and installs required packages on top of a standard Apache Beam SDK base image.
3. Build and Register: Compile the Docker image and push it to Google Artifact Registry.
4. Create Spec: Generate a JSON metadata specification file listing the Docker image URI and input parameters, and upload it to Google Cloud Storage.
5. Run: Launch the template via gcloud, an API request, or an orchestrator (e.g. Cloud Composer), passing values for the parameters.

5. Visual Diagram

6. Code Example

Here is a complete setup for a Python-based Flex Template, including the pipeline script (main.py), Dockerfile, and the template specification JSON structure.

1. The Pipeline Code (main.py)

import logging
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions

class CustomOptions(PipelineOptions):
    @classmethod
    def _add_argparse_args(cls, parser):
        parser.add_value_provider_argument("--input_file", type=str, help="Input GCS path")
        parser.add_value_provider_argument("--output_file", type=str, help="Output GCS path")

def run():
    options = PipelineOptions()
    custom_options = options.view_as(CustomOptions)
    
    with beam.Pipeline(options=options) as p:
        (p
         | "Read" >> beam.io.ReadFromText(custom_options.input_file)
         | "Upper" >> beam.Map(lambda line: line.upper())
         | "Write" >> beam.io.WriteToText(custom_options.output_file)
        )

if __name__ == "__main__":
    logging.getLogger().setLevel(logging.INFO)
    run()

2. The Container Configuration (Dockerfile)

FROM gcr.io/dataflow-templates-base/python311-template-launcher-base:latest

ENV FLEX_TEMPLATE_PYTHON_MAIN_FILE="/app/main.py"

# Copy code and install requirements
WORKDIR /app
COPY main.py /app/main.py
COPY requirements.txt /app/requirements.txt

RUN pip install --no-cache-dir -U -r /app/requirements.txt

# Set entrypoint to run the template launcher
ENTRYPOINT ["/opt/google/dataflow/python_template_launcher"]

3. The Specification File (metadata.json)

{
  "image": "us-central1-docker.pkg.dev/my-project/my-repo/my-template:latest",
  "metadata": {
    "name": "File Uppercase Transform",
    "description": "Reads a file from GCS, converts contents to uppercase, and writes to GCS.",
    "parameters": [
      {
        "name": "input_file",
        "label": "Input GCS File Path",
        "helpText": "GCS path where input text file is located.",
        "isOptional": false
      },
      {
        "name": "output_file",
        "label": "Output GCS File Path",
        "helpText": "GCS path to write the uppercase file.",
        "isOptional": false
      }
    ]
  }
}

7. Code Explanation

• _add_argparse_args registers custom variables. Using add_value_provider_argument allows the runner to inject values at runtime, preventing compile-time hardcoding.
• The Dockerfile inherits from python311-template-launcher-base provided by Google. This base image includes the necessary runtime hooks.
• ENV FLEX_TEMPLATE_PYTHON_MAIN_FILE points the container to the entry script.
• ENTRYPOINT ["/opt/google/dataflow/python_template_launcher"] sets the default execution entrypoint. This binary handles bootstrapping the Python SDK harness when launching.

8. Real Production Example

A retail business extracts transactional sales metrics. They build a Flex Template that accepts dynamic arguments like --start_date and --end_date. A Cloud Composer (Airflow) DAG runs every night, calling the Dataflow Launch API, passing yesterday's date dynamically. The launcher VM compiles a custom graph to load and aggregate partitions from BigQuery specifically for that date window.

9. Common Mistakes

• Static Arguments in Code: Using standard parser.add_argument() instead of parser.add_value_provider_argument(). Standard arguments are processed during compilation, meaning templates launched with new parameters won't recognize them correctly.
• Image Region Mismatches: Storing the Docker image in a repository in one region (e.g. europe-west3) and launching the Dataflow worker VMs in another (e.g. us-east1), resulting in slow startup times or deployment permission errors.

10. Interview Perspective

• Question: Why do Flex Templates require a "Launcher VM" during job startup?
• Answer: Flex Templates do not submit pre-compiled graphs. Instead, they submit a container image. Dataflow boots a temporary "Launcher VM", downloads the container, runs the script inside the container to build the execution graph (DAG), passes the runtime options, submits the graph to the Dataflow service, and then shuts down.
• Question: How do you pass dependency credentials to workers in a Flex Template?
• Answer: Since the runtime environment is fully containerized, all necessary python packages and dependencies are pre-installed in the Docker image during the build process, eliminating the need to pass --requirements_file at job launch.

11. Best Practices

• Use lightweight base images and clear layers in your Dockerfile to keep container startup latency to a minimum.
• Validate parameter types in the metadata.json spec file to catch invalid user inputs before the launcher VM boots.
• Always tag template container versions explicitly in production (e.g., :v1.2.0) instead of relying on the :latest tag.

12. Summary

• Flex Templates containerize pipeline definitions inside Docker.
• They compile the pipeline graph at runtime, allowing dynamic execution trees.
• A Flex Template requires a Docker image, GCS metadata spec, and launcher VM.
• Dependencies are packaged in the image, reducing worker boot issues.

13. Interactive Challenges

14. Related Content

• Runtime Parameters
• Worker Configuration
• Introduction to Dataflow