Change Data Capture (CDC) from PostgreSQL to ClickHouse

This project sets up a real-time data pipeline utilizing Change Data Capture (CDC) to stream changes from a PostgreSQL database to a ClickHouse database through Apache Kafka. Using Debezium to capture and stream database changes, and Kafka Connect to sink the data into ClickHouse, this pipeline allows for efficient data synchronization.

Easy-to-run demo of a CDC pipeline using Debezium (Kafka Connect), PostgreSQL, Redpanda, and ClickHouse.

This project sets up a Change Data Capture (CDC) pipeline using Debezium, Kafka, PostgreSQL, and ClickHouse in Kubernetes.

[Data] Flink CDC demo - PostgreSQL to ClickHouse change data capture streaming pipeline proof of concept. | Apache Flink

End-to-end Data Engineering pipeline using PostgreSQL CDC (Debezium), Kafka, and Spark to implement a Medallion Architecture on a MinIO Data Lakehouse, with analytics serving via ClickHouse.

An End-to-End Real-time Data Pipeline using Debezium (CDC) to stream changes from PostgreSQL to Kafka, processed by Apache Spark (Structured Streaming), and sunk into ClickHouse for analytics. Orchestrated by Airflow and fully containerized with Docker Compose.

No description available

No description available

Kafka Debezium PostgreSQL to ClickHouse CDC Pipeline

rsync-like PostgreSQL to ClickHouse CDC replicator

Pipeline CDC real-time: sinkronisasi data dari PostgreSQL ke ClickHouse menggunakan PeerDB

No description available

Real-time Change Data Capture (CDC) pipeline for auto sales ecommerce using Debezium, Kafka, PostgreSQL, and ClickHouse

A resilient, global insurance data platform prototype featuring Partitioned PostgreSQL, Row-Level Security (RLS), Transactional Outbox, Debezium CDC, Redpanda, and ClickHouse.

Streams PostgreSQL changes to ClickHouse using Debezium for CDC and Kafka for real-time data flow. Enables low-latency replication and analytics.

A Change Data Capture (CDC) pipeline using PostgreSQL, MinIO, ClickHouse, and Airflow. It syncs data from PostgreSQL to ClickHouse via MinIO, supports batch processing, incremental updates, and SCD Type II for historical tracking, with Airflow handling automation and orchestration.

This repository provides a complete real-time data pipeline using Change Data Capture (CDC) to stream changes from a PostgreSQL database into Kafka, and then ingest the data into ClickHouse using its built-in Kafka engine.

CDC(capture data changed) коннектор, который отслеживает изменения в базе данных и предаёт сигнал об этих изменениях через Apache Kafka другим сервисам. Вданный момент подключение реализовано для PostgreSQL, SQLite, Clickhouse. В скором времени появятся подключения к MongoDB, MySQL и ms sql server

This project is a real-time CDC pipeline that streams banking transactions from PostgreSQL to Kafka using Debezium. A Java Kafka Streams app performs live fraud scoring and handles data quality via a Dead Letter Queue. Finally, data is synced to ClickHouse for high-speed OLAP analytics with near-zero source impact.