incremental-loading

Designed a production-grade Azure Data Engineering project centered on Azure Data Factory. Built dynamic, metadata-driven pipelines to ingest data from on-prem systems, REST APIs, and Azure SQL into ADLS Gen2 using Medallion Architecture, incremental loading, and enterprise-scale orchestration patterns.

End-to-end data warehouse for multi-country market-share analytics (FMCG / nutrition) across two markets — Spain (ES) and Portugal (PT) — built on a medallion architecture in Snowflake, feeding Power BI dashboards.

End-to-end lakehouse pipeline for retail data: Postgres to Databricks (query-based incremental ingestion), dbt (incremental models, SCD2, metadata-driven OBT), Airflow orchestration, S3 secondary ingestion, CI-gated on dbt tests.

End-to-end analytics platform on Microsoft Fabric — metadata-driven pipelines, incremental loading, SQL Warehouse, semantic model, Power BI

End-to-End Retail ETL Pipeline using Apache Airflow, PostgreSQL, Docker, and Python with Incremental Loading and Data Quality Validation.

Enterprise SQL data platform demonstrating metadata-driven ETL, staging, dimensional warehousing, incremental loading, auditability, and Docker/LocalDB deployment using SQL Server and SSDT.

End-to-end ETL pipeline: incremental API ingestion, dbt transforms, Airflow orchestration, and CI

Building my first ETL pipelines using Python to learn more about data engineering. Learning by doing.

Production-inspired ETL pipeline built with Python and PostgreSQL featuring data validation, transformation, incremental loading, audit logging, reporting, archiving, CLI support, retry mechanism, and automated testing.

This project pulls historical and forecast weather data for multiple cities, cleans and transforms it, performs quality checks, and stores the results in tidy daily and monthly summary datasets.

A Databricks data engineering project simulating a full medallion architecture (Bronze → Silver → Gold) for e-commerce sales data, featuring incremental loading, a Kimball-style star schema, and SCD Type 1 & Type 2 implementations using PySpark and Delta Lake MERGE INTO.

My first data warehousing project in Databricks SQL a layered pipeline (staging → transformation → core) modeled into a star schema, with incremental loading and SCD Type 1 using MERGE INTO.

End-to-end Retail Sales ETL Pipeline using Python, PostgreSQL, Apache Airflow, Docker, Star Schema Data Warehouse Modeling, KPI Generation, and Incremental Data Loading.

End-to-End Data Engineering Pipeline using Snowflake, dbt, AWS, and Medallion Architecture (Bronze, Silver, Gold) with Incremental Models, Snapshots, Macros, and Data Quality Testing.

Production-ready ETL Pipeline that extracts GitHub repositories, performs incremental loading into PostgreSQL, sends email notifications, and runs automatically using APScheduler.

Discover how Syncfusion® .NET MAUI List View outperforms Collection View with features like fast virtualization, sticky group headers, swipe, drag‑and‑drop, and incremental loading.

Incremental ETL Pipeline with Star Schema Data Warehouse using Python and PostgreSQL

Complete Data Vault 2.0 (DV2) data warehouse implementation with Medallion Architecture on SQL Server. 67 objects, 60 SSIS packages, watermark-based incremental ETL.

Incremental data ingestion pipeline using Azure Data Factory, SQL Server and ADLS Gen2 with watermark-based loading and Parquet storage

End-to-end retail data engineering platform with incremental ETL, SCD Type 2, data quality, audit logging, MinIO, PostgreSQL, Apache Hop, Superset, and Docker.