This project creates a robust data pipeline for efficient ingestion, processing, and storage. Using Apache Airflow for orchestration, it integrates Python, Kafka, Zookeeper, and Spark for real-time data processing, with Cassandra for storage. Docker containerization ensures smooth deployment and scalability of all components.