Distributed Order & Inventory Data Platform
A scalable enterprise Lakehouse platform that centralises order and inventory data from more than 100 distribution centers into a single governed source of truth.
14h → 45m
Daily reporting latency
$250K
Annual cost savings
10M+
Records processed daily
100+
Distribution centers
70%
Faster Power BI refresh
Problem
Order and inventory data arrived daily from 100+ distribution centers as flat files on on-premises network shares. Reporting depended on long-running overnight jobs, so business teams worked with data that was up to 14 hours old and had no reliable way to reconcile conflicting records between regions.
Architecture
Stage 01 · On-Premises Network Shares
Daily CSV deliveries from more than 100 distribution centers.
Simplified flow · No confidential implementation details
Solution
- Designed a Medallion architecture on Databricks (Bronze, Silver, Gold) fed by parameterised Azure Data Factory pipelines that ingest each distribution center's CSV drops automatically.
- Standardised ingestion metadata, validation, deduplication and quarantine paths so malformed files fail visibly instead of silently corrupting downstream reports.
- Modelled curated Gold Delta tables for order and inventory facts, tuned with Z-Ordering and partitioning for the reporting access patterns Power BI actually uses.
- Governed every layer with Unity Catalog, giving analysts discoverable, permissioned access to the same trusted tables.
Key engineering contributions
- Automated daily ingestion of CSV files from multiple distribution centers.
- Processed 10M+ transactional records daily across the lakehouse.
- Implemented Bronze, Silver, and Gold Medallion Architecture.
- Used Unity Catalog for data governance and access control.
- Implemented validation, deduplication, and quarantine handling.
- Optimized Gold Delta tables using Z-Ordering.
- Improved Power BI refresh performance by 70%.
Technologies
- Azure Data Factory
- Azure Blob Storage
- Databricks
- PySpark
- Unity Catalog
- Delta Lake
- Power BI
Want the full story behind this platform?
Happy to walk through the engineering decisions and trade-offs.
Mukesh Varma Gangaraju · Senior Data Engineer at Stryker