Parquet

1. Introduction

2. Why This Concept Exists

For analytical queries (OLAP), row-oriented databases must read full rows even if only querying 2 columns. Parquet stores values column-by-column, allowing readers to skip irrelevant columns (projection) and read ranges of data efficiently.

3. Key Terminology

• Columnar Layout: Organizing values of the same column contiguously on disk.
• Projection Pushdown: Only loading specific columns from disk into memory.
• PyArrow: The underlying Apache Arrow Python library used to interact with Parquet.

4. How It Works

• Reading: ReadFromParquet leverages pyarrow to parse metadata, extract specified column arrays, and construct dictionaries.
• Writing: WriteToParquet receives elements, formats them to PyArrow tables based on a PyArrow schema, and writes columnar files.

5. Visual Diagram

6. Code Example

Reading specific columns from Parquet files:

pythonCopy
import apache_beam as beam
from apache_beam.io.parquetio import ReadFromParquet

with beam.Pipeline() as p:
    (p
     | "ReadParquet" >> ReadFromParquet("data/*.parquet", columns=["user_id", "amount"])
     | "LogAmounts" >> beam.Map(print)
    )

7. Code Explanation

• ReadFromParquet opens Parquet files.
• The columns argument specifies columns to load, avoiding the overhead of reading the entire row schema.

8. Real Production Example

Writing a PCollection to Parquet with a PyArrow schema:

pythonCopy
import pyarrow as pa
import apache_beam as beam
from apache_beam.io.parquetio import WriteToParquet

# Define pyarrow schema
pyarrow_schema = pa.schema([
    ("user_id", pa.int64()),
    ("amount", pa.float64())
])

with beam.Pipeline() as p:
    data = (p | beam.Create([
        {"user_id": 101, "amount": 45.50},
        {"user_id": 102, "amount": 90.00}
    ]))
    
    data | "WriteParquet" >> WriteToParquet(
        "gs://my-bucket/payments", 
        schema=pyarrow_schema, 
        file_name_suffix=".parquet"
    )

9. Common Mistakes

• Skipping PyArrow schema: Writing Parquet files requires a defined pyarrow schema. Skipping this will trigger runtime errors.
• Reading unused columns: Not passing a projection list when reading columns loads everything into memory, degrading network performance.

10. Best Practices

• Always define columns to read via projection.
• Use standard PyArrow types to model schemas.

11. Summary

• Parquet is a columnar binary file format.
• Uses Projection Pushdown to save CPU and network bandwidth.
• Requires PyArrow schema setup for writing.

12. Interactive Challenges

13. Related Content

• Avro
• File IO