Installation

1. Introduction

To develop and run Apache Beam pipelines locally, you must set up your programming environment by installing the Apache Beam SDK.

2. Why This Concept Exists

Before you can run pipelines on local Direct Runners or publish them to Cloud Dataflow, you need the necessary libraries, CLI tools, and runtime dependencies installed on your workstation.

3. Key Terminology

• Virtual Environment (venv): An isolated directory that keeps Python packages for different projects separate.
• Beam SDK Extras: Additional package selections that install dependency extensions (e.g. [gcp], [aws], [interactive]).

4. How It Works

1. Select runtime: Verify Python version (supported: 3.8 to 3.12).
2. Isolate environment: Create and activate a virtual environment.
3. Install core: Run pip install apache-beam.
4. Install extras: Run pip install apache-beam[gcp] if utilizing GCP resources (Dataflow/BigQuery).
5. Verify: Run a quick python import statement.

5. Visual Diagram

6. Code Example

Setting up your environment using terminal commands:

bashCopy
# 1. Create a virtual environment
python -m venv beam-env

# 2a. Activate on macOS/Linux
source beam-env/bin/activate

# 2b. Activate on Windows
beam-env\Scripts\activate

# 3. Upgrade pip & Install Beam with GCP extras
pip install --upgrade pip
pip install "apache-beam[gcp]"

7. Code Explanation

• python -m venv generates an isolated environment folder.
• source beam-env/bin/activate or beam-env\Scripts\activate forces the terminal session to use this local env.
• apache-beam[gcp] installs the core SDK along with Google Cloud APIs (Pub/Sub, BigQuery, Storage).

8. Real Production Example

When setting up automated CI/CD runners (like GitHub Actions), you write workflow steps that install the pinned SDK version from requirements files before executing pipeline checks:

bashCopy
pip install apache-beam==2.53.0

9. Common Mistakes

• Installing without virtual environments: This causes system-level dependency conflicts with other python libraries.
• Mismatched Python versions: Running on legacy Python versions (e.g. Python 2.7 or unsupported releases) will cause installation failures.

10. Best Practices

• Always pin your Apache Beam SDK version in requirements.txt to prevent sudden build breaks when new versions release.
• Activate virtual environments before running pip.

11. Summary

• Install using pip install apache-beam.
• Use [gcp] extras for Cloud Dataflow integration.
• Verify using python import statements.

12. Interactive Challenges

13. Related Content

• Pipeline Basics
• Pipeline Options