As data scale grows, maintaining on-premise Hadoop clusters or running complex batch/streaming pipelines on legacy frameworks becomes a significant operational bottleneck.
In this case study, we examine how a major e-commerce retail platform migrated their batch and streaming pipelines to Apache Beam executed on Google Cloud Dataflow, analyzing the architectural changes, performance improvements, and cost savings.
1. The Legacy Architecture & Pain Points
Historically, the platform ran two separate systems to process user transaction logs:
- Batch Pipeline: A daily Hadoop MapReduce job that parsed yesterday's raw CSV events and calculated transaction reports. It took 6 hours to execute, and failures required re-running the entire job.
- Streaming Pipeline: An Apache Storm pipeline designed to detect immediate security alerts. Maintaining it required manual scaling of virtual machines (VMs) and writing different code from the batch system.
Key Pain Points
- High overhead costs for managing physical Hadoop server nodes.
- Dual codebase maintenance (Java for Storm, Scala/Python for MapReduce).
- Lack of exactly-once delivery guarantees in streaming, leading to duplicate event calculations.
2. The Unified Beam Architecture
The company chose Apache Beam due to its unified programming model and portability. They rewrote the processing logic into a single Python pipeline that processes both batch and streaming inputs.
[Raw Event Logs] ---> [Pub/Sub / GCS] ---> [Apache Beam Pipeline] ---> [Google BigQuery]
(Runs on Dataflow)
Key Architectural Enhancements
- Unified Codebase: The exact same enrichment, filtering, and mapping logic applies to daily GCS text files and real-time Pub/Sub queues.
- Streaming Engine: By offloading state tracking to Cloud Dataflow's Streaming Engine layer, worker VMs scaled rapidly during traffic surges (such as Black Friday) without running out of disk space.
- Exactly-Once Processing: Dataflow's automated message deduplication resolved duplicate transaction counts, ensuring financial reporting accuracy.
3. Results and Metrics
The migration yielded impressive performance improvements:
- Ingestion Latency: Dropped from 6 hours (batch processing delay) to under 5 seconds of continuous, real-time data availability in BigQuery.
- Developer Efficiency: Code size was reduced by 40%, and developers could test and debug pipelines locally using the Direct Runner before deploying to production.
- Infrastructure Cost Savings: Switching to a serverless model (where Dataflow VM instances autoscale down to zero during off-peak hours) reduced monthly compute costs by 35%.
4. Key Takeaways
- [ ] Decouple Ingestion from Compute: Moving to a serverless runner like Dataflow eliminates server maintenance overhead and allows teams to focus entirely on write pipelines.
- [ ] Design for Deduplication: Always assign unique transaction IDs at the source to leverage Beam's exactly-once processing capabilities.
- [ ] Start with Small Iterations: Validate new pipeline logic on subset samples before launching high-throughput streaming migrations.