A plugin to Apache Airflow to allow you to run Spark Submit Commands as an Operator

Airflow Livy Spark Operator using Batch concept

Collected about 2 million tweets from Twitter APIs, explored, cleaned, and copied the data to S3 using Spark and Pandas Python libraries, orchestrated the data in Redshift using Airflow DAGs and Plugin Operators and performed necessary data quality checks thereafter to give to end-users/data scientists for sentiment analysis.