Docker Apache Airflow

A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!

A docker image and kubernetes config files to run Airflow on Kubernetes

Source code of the Apache Airflow Tutorial for Beginners on YouTube Channel Coder2j (https://www.youtube.com/c/coder2j)

An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.

Docker with Airflow and Spark standalone cluster

🐳📊🤓Cookiecutter template to launch an awesome dockerized Data Science toolstack (incl. Jupyster, Superset, Postgres, Minio, AirFlow & API Star)

Apache Airflow in Docker Compose (for both versions 1.10.* and 2.*)

An Airflow docker image preconfigured to work well with Spark and Hadoop/EMR

Data Forge — a modern data stack playground to practice flows and best practices, not just tools. Spark, Trino, Kafka, Iceberg, ClickHouse, Airflow, MinIO, Superset — all wired together locally with Docker Compose.

A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for cloud data warehouse and Metabase to serve the needs of data visualizations such as analytical dashboards.

Execution of DBT models using Apache Airflow through Docker Compose

The goal of this project is to track the expenses of Uber Rides and Uber Eats through data Engineering processes using technologies such as Apache Airflow, AWS Redshift and Power BI.

End to end data engineering project with kafka, airflow, spark, postgres and docker.

scaffold of Apache Airflow executing Docker containers

The goal of this project is to build a docker cluster that gives access to Hadoop, HDFS, Hive, PySpark, Sqoop, Airflow, Kafka, Flume, Postgres, Cassandra, Hue, Zeppelin, Kadmin, Kafka Control Center and pgAdmin. This cluster is solely intended for usage in a development environment. Do not use it to run any production workloads.

Sync DAG changes from Git to Airflow

Docker Airflow - Contains a docker compose file for Airflow 2.0

Dockerized analytics pipeline where Airflow moves mock-API data into Postgres, runs dbt transformations, and serves clean tables for Power BI consumption.

How to use the DockerOperator in Airflow within Docker Compose?

A portable Datamart and Business Intelligence suite built with Docker, Airflow, dbt, duckdb and Superset

No description available

Tutorial like code for how to deploy airflow using docker and how to use the DockerOperator.

Airflow Pipeline for Machine Learning

No description available

🎵 LyricWave – AI Music Composer (Proof of Concept) 🎶 A personal project exploring automatic generation of unique MP4 songs. LyricWave blends lyrics with AI-generated melodies and synthetic vocals to experiment with new forms of musical expression. A creative testbed to push your ideas into sound. 🚀🎧

A workspace to experiment with Apache Spark, Livy, and Airflow in a Docker environment.

Writes the CSV file to Postgres, read table and modify it. Write more tables to Postgres with Airflow.

Quickly get a kubernetes executor airflow environment provisioned on GKE. Azure Kubernetes Service instructions included also as are instructions for docker-for-mac.

Orchestration of data science and earth observation models in Apache Airflow, scale-up with Celery Executor, experiment with jupyter notebook using a docker containers composition