Back to portfolio
Data EngineeringAnalytics

Distributed Order & Inventory Data Platform

A scalable enterprise Lakehouse platform that centralises order and inventory data from more than 100 distribution centers into a single governed source of truth.

14h → 45m

Daily reporting latency

$250K

Annual cost savings

10M+

Records processed daily

100+

Distribution centers

70%

Faster Power BI refresh

Problem

Order and inventory data arrived daily from 100+ distribution centers as flat files on on-premises network shares. Reporting depended on long-running overnight jobs, so business teams worked with data that was up to 14 hours old and had no reliable way to reconcile conflicting records between regions.

Architecture

Stage 01 · On-Premises Network Shares

Daily CSV deliveries from more than 100 distribution centers.

Simplified flow · No confidential implementation details

Solution

  • Designed a Medallion architecture on Databricks (Bronze, Silver, Gold) fed by parameterised Azure Data Factory pipelines that ingest each distribution center's CSV drops automatically.
  • Standardised ingestion metadata, validation, deduplication and quarantine paths so malformed files fail visibly instead of silently corrupting downstream reports.
  • Modelled curated Gold Delta tables for order and inventory facts, tuned with Z-Ordering and partitioning for the reporting access patterns Power BI actually uses.
  • Governed every layer with Unity Catalog, giving analysts discoverable, permissioned access to the same trusted tables.

Key engineering contributions

  • Automated daily ingestion of CSV files from multiple distribution centers.
  • Processed 10M+ transactional records daily across the lakehouse.
  • Implemented Bronze, Silver, and Gold Medallion Architecture.
  • Used Unity Catalog for data governance and access control.
  • Implemented validation, deduplication, and quarantine handling.
  • Optimized Gold Delta tables using Z-Ordering.
  • Improved Power BI refresh performance by 70%.

Technologies

  • Azure Data Factory
  • Azure Blob Storage
  • Databricks
  • PySpark
  • Unity Catalog
  • Delta Lake
  • Power BI

Want the full story behind this platform?

Happy to walk through the engineering decisions and trade-offs.

Mukesh Varma Gangaraju · Senior Data Engineer at Stryker