This project demonstrates real-time data streaming and processing architecture using Kafka, Spark Streaming, and Debezium for capturing CDC (Change Data Capture) events. The pipeline collects transaction data, processes it in real time, and updates a dashboard to display real-time analytics for smartphone data.

This project demonstrates the implementation of a Change Data Capture (CDC) system using Debezium, Kafka, Postgres, and Docker.

Power your Down Stream Elastic Search Stack From Apache Hudi Transaction Datalake with CDC

Architecture Powering Down Stream System with CDC from HUDI Transactional Datalake

Data pipeline --> transactional DB, CDC, streaming, real time analytics ||| cloud infrastructure, data lake, distributed processing, transformations, data warehouse ||| orchestration, containerization

A production-inspired real-time data pipeline that streams banking transactions from a transactional RDBMS using CDC and Kafka into a scalable data lake, enabling low-latency analytics, reliable recovery from failures, and analytics-ready datasets.

This project demonstrates the implementation of a Change Data Capture (CDC) system using Debezium, Kafka, Postgres, and Docker.

No description available

No description available

Streaming analytics pipeline for UPI transactions using change data capture to process real‑time payments data for fraud detection and behavioral insights.

CDC ( Change Data Capture ) streaming pipeline for UPI ( Unified Payment Interface ) transaction using Databricks. Delta Lake and Apache Spark Structured Streaming . The Pipeline processes transaction data in real-time with CDC support,performs mechant aggregations , and provides comprehensive monitoring capabilities for financial transaction.

Real-time CDC streaming ETL pipeline using Apache Flink and Kafka to keep data lakes updated with transactional DB changes.

Event-driven system leveraging transactional-outbox design pattern, CockroachDB’s Change Data Capture (CDC), and event streaming with Apache Kafka (KRaft mode) for reliability, scalability, and consistency

Real-time UPI transaction processing using Databricks, Spark Streaming & Delta Lake. CDC feeds from source table enable streaming aggregations for merchant settlement calculations. Target Delta table stores latest settlement data, queryable via SQL Data Warehouse for instant BI insights.

End-to-end data pipeline that captures changes from a transactional database using CDC, streams them through a lakehouse architecture, and serves analytics via a Metabase dashboard

Real-Time Banking Data Engineering Platform - A production-ready data lakehouse implementation demonstrating CDC-based streaming ingestion, batch processing, and automated transformations for banking transaction data. Built with Kafka, Debezium, Airflow, dbt, and Snowflake.

Banking Streaming Data Platform is an end-to-end real-time analytics system built with Kafka, Debezium CDC, Airflow, DBT, MinIO, and Snowflake. It simulates banking operations and streams transactional data to build a modern, scalable data platform for analytics and insights.

This project demonstrates Change Data Capture (CDC) using Debezium, Kafka, PostgreSQL, and Docker. It includes a Python script that generates financial transaction data with the Faker library and inserts it into a PostgreSQL database. The setup provides a test environment for experimenting with CDC pipelines and streaming data in real time.

A resilient Change Data Capture (CDC) pipeline implementing the Transactional Outbox pattern. Streams PostgreSQL WAL changes to Kafka using pgoutput and TypeScript to ensure at-least-once delivery and strict data consistency in distributed systems.

Designed and implemented a real-time Change Data Capture (CDC) pipeline to capture and stream transactional updates from PostgreSQL into a data warehouse. Leveraged Debezium with Kafka Connect and Confluent Schema Registry to ensure schema evolution and backward compatibility.

An end-to-end, production-style real-time fraud detection platform for financial transactions. The system ingests transactional data from MSSQL using CDC (Debezium + Kafka), processes events in real time with Flink Structured Streaming, applies rule-based and ML-driven fraud detection, and persists results into analytical data models using dbt.