Skip to main content

Understanding Flink

· 4 min read
tanzhuo

Throughout the evolution of big data systems, one core problem has never gone away: data keeps growing, but processing keeps getting slower. This article starts from batch processing and works through why Flink emerged and what problems it actually solves.

From batch processing to stream processing​

Traditional data processing is mostly offline: run a job every night to aggregate user behavior, compute ad metrics, generate reports, and analyze business KPIs. This approach is called batch processing, with typical tools such as:

  • Hadoop MapReduce
  • Hive
  • Spark

These systems are great at crunching massive volumes of historical data. But as the internet grew, more and more businesses started needing real-time data processing, for example:

  • Real-time risk control
  • Real-time recommendations
  • Real-time monitoring
  • Real-time ad bidding
  • Real-time log analysis

These scenarios share one trait: data must be processed as it is produced — you can't wait until tomorrow. This gave rise to a new computation model: stream processing.

Early real-time systems mostly relied on Storm and Spark Streaming, but both had issues.

Storm is a genuine stream processor, but development is complex, fault tolerance is mediocre, and state management is hard. Spark Streaming is stable, but it is fundamentally micro-batch processing — in other words, not true streaming. For example:

Process one batch of data every second

For many scenarios, this still isn't real-time enough. So Flink emerged, with a very clear goal: build a true stream processing system.

1) Real-time data processing​

Flink delivers true millisecond-level real-time computation. Take a user clicking on a product:

Click → Kafka → Flink → real-time recommendation

The whole pipeline can complete in tens of milliseconds.

2) State management​

One crucial problem in stream processing is state. Take counting user clicks:

User A clicks: 1 time
User A clicks: 2 times
User A clicks: 3 times

The system has to remember what it saw before. Flink has built-in state management (Keyed State, Operator State) and supports both local state and RocksDB-backed state storage, which allows it to handle state at enormous scale.

3) Fault tolerance​

In distributed systems, node failures are the norm. Flink protects data through its checkpoint mechanism: the system periodically saves computation state, and if a node goes down, computation resumes from the checkpoint. This mechanism guarantees exactly-once processing — each record is processed exactly one time.

4) Event time​

Stream processing has another notoriously hard problem: event time. Log data may arrive late, and if you compute purely by system time, your statistics come out wrong.

Flink introduced the watermark mechanism. Watermarks help the system decide when data has "mostly all arrived", which makes it possible to correctly handle late data, window computations, and time-based aggregations.

1) Stream First​

Flink's core philosophy is that everything is a stream — batch processing is just a bounded stream. As a result, Flink runs batch and streaming on the same engine.

2) State Driven​

Flink is a state-driven computation system: all computation revolves around state. User behavior analytics, real-time aggregation, real-time monitoring — all of them depend on state management.

3) Exactly Once​

Flink puts heavy emphasis on correctness of data processing. Through checkpoints, distributed snapshots, and two-phase commit, Flink achieves exactly-once semantics — a capability that many real-time systems consider essential.

Flink is used extensively across internet companies, in scenarios such as real-time recommendation systems, user behavior analytics, ad click counting, real-time risk control, and real-time log analysis. A common data pipeline looks like:

User behavior → Kafka → Flink → ClickHouse

Flink sits in the middle, handling data cleaning, real-time computation, and real-time aggregation.

Wrapping up​

Flink exists, at its core, to do "true stream processing". It unifies batch and streaming on a single engine, supports large-scale stateful computation with built-in state management, buys exactly-once correctness with its checkpoint mechanism, and handles out-of-order and late data with watermarks. Once you understand these points, you understand the fundamental difference between Flink and earlier real-time systems — and the specific APIs and cluster deployment will make much more sense afterwards.

COMMENTS