Skip to main content

The Canal Component

· 7 min read
tanzhuo

Introduction​

When building business systems, you frequently run into this requirement: a row changes in the database, and downstream caches, search engines, and data warehouses all need to change with it. Dual writes in application code are easy to miss; periodic full-table comparisons are far too heavy. There is a more elegant answer to this class of problems — subscribing directly to the database's change log — and that's why I put together these notes on Canal.

Canal is an open-source Alibaba component for real-time data subscription and consumption based on MySQL Binlog, commonly used to implement Change Data Capture (CDC).

In real-world business scenarios, database changes often need to be propagated beyond ordinary reads and writes, for example:

  • Real-time data synchronization (database → data warehouse)
  • Pushing data changes to an MQ (Kafka / RocketMQ)
  • Building real-time analytics or monitoring systems
  • Syncing data to a search engine (such as Elasticsearch)

What these scenarios share is that downstream systems don't care about "what the database looks like right now" — they care about "what changed." If application code has to manually notify downstream systems after every write, it's not only invasive but also very hard to keep consistent with the database transaction: if the write succeeds but the notification fails, the data quietly drifts out of sync. The Binlog, on the other hand, is the log MySQL itself guarantees to write, so using it as the data source sidesteps the dual-write problem entirely.

To make this work, Canal subscribes to the Binlog by impersonating a MySQL slave, parses out the insert/update/delete operations, and pushes these change events to downstream consumers in real time, achieving near-real-time data synchronization.

How It Works​

Canal borrows the mechanics of MySQL master-slave replication. In normal replication, the slave sends a dump request to the master, the master continuously streams Binlog events to the slave, and the slave replays those events to stay in sync.

What Canal does is "disguise" itself as a slave:

  1. It sends a dump protocol request to MySQL, using exactly the same interaction as a real replica;
  2. MySQL streams Binlog events to Canal, which parses the binary log and reconstructs the changes for every row;
  3. The parsed, structured events are handed to downstream consumers — either pulled by clients over TCP, or delivered directly to a message queue such as Kafka or RocketMQ.

From MySQL's point of view, Canal is just an ordinary replica. No plugin needs to be installed on the database side, which is why this approach is so low-touch.

For Canal to subscribe successfully, MySQL needs to meet a few prerequisites:

# Key settings in my.cnf
log-bin=mysql-bin        # Enable Binlog
binlog-format=ROW        # Must be ROW mode to capture the full before/after image of every row
server_id=1              # Unique ID for each node in the replication topology; must not clash with Canal's slaveId

The Binlog format must be ROW. STATEMENT mode records the SQL statements themselves, from which the concrete per-row changes cannot be reconstructed; only ROW mode records row-level before-and-after images, which is what makes CDC meaningful. In addition, the account Canal uses to connect to MySQL needs the REPLICATION SLAVE and REPLICATION CLIENT privileges — the same requirements as a real replica.

Comparing Common CDC Components​

The CDC components you'll commonly see today are:

  • Canal
  • Debezium
  • Flink CDC

Canal currently supports only MySQL, versions 5.x and 8.x.

GitHub - alibaba/canal: Alibaba's MySQL binlog incremental subscription & consumption component

The core mechanism of all these tools is essentially the same: parse the database's Binlog to obtain the changed data along with the operation type (INSERT / UPDATE / DELETE), and turn those changes into structured events for downstream systems to consume.

Where they differ is mainly in ecosystem positioning. Debezium is built on Kafka Connect, supports MySQL, PostgreSQL, and several other databases, and suits teams that already run Kafka. Flink CDC embeds change capture directly into Flink stream processing, so capture and transformation happen in the same job. Canal is more lightweight and focuses exclusively on MySQL.

Note that Canal currently supports mainly MySQL (versions 5.x and 8.x), which is why it is so widely used to build real-time synchronization and data distribution systems within the MySQL ecosystem. If your stack is pure MySQL and you don't want to bring in a heavy dependency like Kafka Connect or Flink, Canal is a pragmatic choice.

High Availability​

For high availability, Canal offers a cluster deployment mode. In a cluster, each Server node manages multiple synchronization task instances (Instances), and a task-distribution mechanism spreads the load.

Cluster mode relies on ZooKeeper for coordination: multiple Server nodes compete for the right to run a given Instance, only one node actually runs it at any moment, and the rest stand by. When the running node fails, a standby node detects this via ZooKeeper and takes over the task. The consumption position (i.e., the Binlog consumption progress) is also stored in ZooKeeper, so the node taking over can resume from the last recorded position, avoiding data loss or large-scale duplication.

In practice, though, network hiccups, database connection failures, or resource limits can occasionally cause synchronization instances to terminate. In production, Canal is therefore usually paired with an automatic restart mechanism or an ops monitoring system so that tasks recover automatically after failures.

Pitfalls and Caveats​

Building on the mechanics above, a few points are worth thinking through before rollout:

  1. Binlog format and retention. The upstream database must run in ROW mode, and Binlog retention needs to be long enough. Otherwise, if Canal is down for a while and the logs at its saved position have already been purged, the task cannot resume and must be re-initialized from scratch.

  2. The delivery semantics are at-least-once. After a failover or restart, a small window of change events may be delivered again. Downstream consumers must deduplicate by primary key and never assume each change arrives exactly once.

  3. Ordering depends on the partitioning strategy. When delivering to an MQ, if changes to multiple tables or primary keys get scattered across different partitions, the consumption order may diverge from the Binlog order. For order-sensitive scenarios, route partitions by table name or primary key.

  4. Zombie Instances. Task instances sometimes don't exit cleanly — they stall at a position and stop advancing. Monitoring can't just check process liveness; it must also verify that the position keeps moving forward and track the lag against the database's current Binlog position.

tip

Before going live, run a "link-break drill": manually kill the running Server node and watch whether the standby takes over as expected and whether the position resumes correctly. It's far more comfortable than validating this for the first time during a real incident.

Wrapping Up​

Canal's idea is not complicated: impersonate a MySQL replica, subscribe to the Binlog over the standard replication protocol, and reconstruct row-level changes into structured events for downstream consumption. It's zero-intrusion on the database and lightweight to deploy, making it a great fit for real-time sync and data distribution in a pure-MySQL stack. What deserves engineering attention are the ROW format and Binlog retention policy, idempotent consumption downstream, and monitoring based on position advancement — get those solid, and the pipeline's stability follows.

COMMENTS