Big Data.
05 posts — Posts about Big Data
Dinky
Dinky is an open-source platform for developing and managing Flink jobs, with online Flink SQL editing and debugging, whole-database synchronization, and data lineage analysis. Here are my notes on what it does and where it fits.
The Canal Component
Canal is Alibaba's open-source component for incremental subscription and consumption based on MySQL Binlog. This post walks through how it works, how it compares to other common CDC tools, and what to watch out for in a highly available cluster deployment.
Flink CDC Components
An introduction to Flink CDC and its built-in components: starting from how change-log capture works, then covering the Source, Debezium Connector, Sink, and other core modules, plus common caveats in practice.
Data Lakes and Iceberg
Sorting out the differences between data lakes and data warehouses, and introducing Apache Iceberg, an open-source table format: versioning, metadata management, partitioning, and snapshots, plus some notes for technology selection.
Working with Kettle
Notes on using Kettle for multi-source data sync, migration, and transformation: the core concepts, what the common components are for, and the scenarios and limits of query-based CDC.