Cheatsheet: BigQuery IO

Topic Overview

Read and write data high-throughput at scale to Google BigQuery.

Syntax Snapshot

import apache_beam as beam
from apache_beam.io.gcp.bigquery import WriteToBigQuery

# Write records in storage api mode
records | "WriteBQ" >> WriteToBigQuery(
    "my-project:my_dataset.my_table",
    write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND,
    create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
    method=WriteToBigQuery.Method.STORAGE_WRITE_API
)

Key Points

• Read: Native SQL queries or direct table exports.
• Write: Streaming Inserts (immediate, expensive) vs Storage Write API (high throughput, recommended).
• Create disposition: CREATE_IF_NEEDED vs CREATE_NEVER.
• Write disposition: WRITE_APPEND, WRITE_TRUNCATE (batch only), WRITE_EMPTY.

Production Recommendations