Generate synthetic Spotify music stream dataset to create dashboards. Spotify API generates fake event data emitted to Kafka. Spark consumes and processes Kafka data, saving it to the Datalake. Airflow orchestrates the pipeline. dbt moves data to Snowflake, transforms it, and creates dashboards.

No description available

Code to be contributed to the Apache Airflow (incubating) project for ETL workflow management for integrating with the Snowflake Data Warehouse.

This is end to end data engineering project using dbt , snowflake and apache airflow

Sample Airflow DAGs to load data from the CovidTracking API to Snowflake via an AWS S3 intermediary.

Explore building an advanced infrastructure for enhancing QuantConnect with Snowflake, Databricks, Airflow & AWS. Learn the basics of quant trading workflows, from selecting US cash equities datasets to efficient trade execution. Dive into computing indicators and ML-based signals across thousands of symbols using a distributed framework.

A reference architecture for building an ETL/ELT pipeline with Apache Airflow® and Snowflake.

End-to-end ELT pipeline for 160K+ Skytrax airline reviews: Airflow orchestration, BeautifulSoup scraping, S3 staging, Snowflake warehouse, dbt star schema transformation, Terraform IaC, GitHub Actions CI/CD, and Next.js dashboard with LangChain RAG chatbot.

Built a Data Pipeline for a Retail store using AWS services that collects data from its transactional database (OLTP) in Snowflake and transforms the raw data (ETL process) using Apache spark to meet business requirements and also enables Data Analyst create Data Visualization using Superset. Airflow is used to orchestrate the pipeline

No description available

airflow-dbt-snowflake-poc

Spotify API, Airflow, Docker, AWS S3, Snowflake, dbt, localstack, Looker Studio

An ETL Data Pipelines Project that uses AirFlow DAGs to extract employees' data from PostgreSQL Schemas, load it in AWS Data Lake, Transform it with Python script, and Finally load it into SnowFlake Data warehouse using SCD type 2.

End-to-end ELT batch pipeline | Medallion Architecture (Bronze → Silver → Gold) | dbt + Snowflake + AWS S3 + Apache Airflow + Docker | 3.5M+ records

Analytics engineering capstone from DataExpert. I used python, Snowpark, Snowflake, dbt, airflow, machine learning (LightGBM), and Power BI. Ingested 700k rows of raw flight arrival data and paired with a weather api to make delay predictions with ML.

Data Engineering with Apache Airflow, Snowflake & dbt

Pipeline using Snowflake, dbt core and Airflow

Create data pipeline using Lambda architecture with Spark, Kafka, Airflow and Snowflake

Build a complete ELT pipeline in 1 hour using industry-standard tools like dbt, Snowflake, and Airflow. This project demonstrates step-by-step setup, basic data modeling techniques (fact tables, data marts), Snowflake RBAC concepts, and orchestration of a dbt project with Airflow. Perfect for hands-on learning in Data Engineering!

End-to-end Banking Data Pipeline using Kafka, Debezium, S3, Snowflake, dbt, and Airflow.

Python, SQL, Snowflake, AWS services, Google Cloud Platform, Hadoop, HDFS, Mapreduce, Hive, Pig, MongoDB, HBase, Apache kafka, Apache Airflow, Docker, Tableau

No description available

A production‑grade, modular data pipeline that automates the ingestion, transformation, and loading of retailer data into Snowflake using Python, Airflow, Terraform, and Docker. Ideal for data engineers, DevOps engineers, and DataOps workflows.

SnowflakeAirflowDbtCosmo project, a demonstration of integrating Airflow, DBT, and Snowflake with Snowpark for advanced data analysis. This project, generated with astro dev init using the Astronomer CLI, showcases how to run Apache Airflow locally, building both simple and advanced data pipelines involving Snowflake.

This project showcases an end-to-end ELT (Extract, Load, Transform) pipeline leveraging the TPCH orders table from Snowflake's sample database. The primary goal is to demonstrate modern data engineering practices using Snowflake, dbt (Data Build Tool), and Apache Airflow.

No description available

This project is designed to provide practical experience with Airflow for workflow management, Soda for data quality checks, and Snowflake for secure and efficient data storage.

Ticketmaster Analytics Pipeline - Music Event Demand Timing Analysis using Airflow, dbt, and Snowflake

No description available

No description available