Data Lakes and Iceberg

Notes on Iceberg, currently one of the most popular open-source data lake projects.

While reviewing big-data storage options recently, there was no getting around the term "data lake". Business data long ago stopped being just tables in a relational database: event logs, JSON payloads, images, audio, and video all need somewhere to land. Traditional data warehouses struggle more and more with these workloads, and a wave of open-source projects built around the data lake concept fills exactly that gap. This post sorts out the difference between data lakes and warehouses, and takes a look at Iceberg, one of the most representative projects in the space — my own study notes.

Data Lakes vs. Data Warehouses

A data lake is a centralized repository for storing both structured and unstructured data. It can hold data of every type: traditional relational data, semi-structured data, unstructured data, and more. Its defining traits are flexibility and scalability — data can be stored and processed with minimal friction.

A core idea behind data lakes is "store first, manage later" (schema-on-read): data is written in its raw format to cheap distributed storage (HDFS or object storage), and its structure is only parsed when it's actually analyzed. This is the exact opposite of the warehouse approach of "model first, then load" (schema-on-write), and it's why lakes can accommodate data in any shape.

Traditional data warehouses, by contrast, are built mainly for structured data and typically use relational databases for storage and management. They are highly normalized and structured, which guarantees data accuracy and consistency. The downside is that warehouses are inflexible: they can't store or process unstructured or semi-structured data, and they're hard to scale and upgrade.

The main advantages of a data lake over a data warehouse:

1. Unrestricted storage: a data lake can hold every kind of data — structured, semi-structured, unstructured. That makes it flexible enough for storage needs of any type and scale.
2. Faster processing: data lakes typically use distributed storage and compute, enabling high-speed processing and analysis. Traditional warehouses often need multiple rounds of data transformation and computation, which slows things down.
3. Lower cost: data lakes are usually built on open-source technology like Hadoop and Spark, keeping costs relatively low, whereas traditional warehouses require commercial database software.
4. More flexibility: a data lake makes it easy to store and process data however you need, while a traditional warehouse's repeated transformation and computation steps make the process complex and cumbersome.

Of course, flexibility has a price. A bare data lake is just a pile of file directories — no transactions, no schema enforcement. If writers and readers aren't careful, they trample each other, and over time the lake easily degrades into a "data swamp". The table format layer of abstraction exists to solve exactly this: it defines "table" semantics on top of the files. Iceberg is one of those projects.

Apache Iceberg

Apache Iceberg

Iceberg is an open-source table format and processing library designed to solve the problems of managing and processing tables in a data lake. It was developed at Netflix and has since become a top-level project of the Apache Software Foundation.

Iceberg addresses several pain points of traditional data lakes, such as table versioning, table metadata management, partition management, and table snapshot management. It works with a range of storage and processing systems, including Hadoop, Spark, Presto, and Flink.

Its working mechanism deserves a closer look. Iceberg itself doesn't store data — the data still lives in columnar files like Parquet and ORC; what Iceberg manages is the metadata layer on top. The metadata has roughly three levels:

1. Snapshots: the complete state of the table at a point in time. Every committed write produces a new snapshot, and old snapshots remain readable.
2. Manifest lists and manifest files: they record which data files a snapshot contains, along with each file's partition ranges, row counts, column-level statistics, and so on.
3. Data files: the actual columnar files. Metadata only references them, never modifies them.

On write, the new metadata is persisted first, and the commit completes with a single atomic swap of the table pointer — giving you database-like ACID semantics on top of a file system. On read, the engine consults the metadata first and uses its statistics to skip irrelevant files, instead of listing an entire directory the way Hive does — on object storage, that makes a huge difference to performance.

Iceberg's Key Features

Iceberg's main features:

1. Table versioning: Iceberg supports version control for tables. You can easily roll back to a historical version, or merge different versions of a table.
2. Table metadata management: Iceberg provides a metadata mechanism that makes tables easy to manage and query. Through the metadata you can inspect a table's structure, partitions, data statistics, and more.
3. Partition management: Iceberg supports multiple partitioning schemes, including hash partitioning, range partitioning, and hybrid hash-range partitioning. Pick whichever fits your needs.
4. Table snapshot management: Iceberg supports snapshots of tables, making backup and recovery straightforward.

One more detail on partitioning: Iceberg's partitions are hidden partitions. Partition values are derived automatically from column values through transform functions (say, truncating a timestamp to the day), so queries just filter on the original columns and the engine applies partition pruning automatically. Whoever writes the query doesn't need to know how the table is partitioned — much friendlier than Hive's manually maintained partition columns. The partition spec itself can also evolve, and changing it doesn't require rewriting historical data.

Schema evolution is another strong suit: adding, dropping, and renaming columns only touch metadata. Columns are tracked by unique IDs rather than by name or position, which avoids the classic old-table-format problem where renaming a column could make you read the wrong data.

Gotchas

1. The small-files problem still exists. High-frequency streaming writes produce lots of small data files and metadata files; you need to run compaction regularly, or query performance keeps degrading.
2. Snapshots aren't cleaned up automatically. Every commit leaves a snapshot behind, and files referenced by historical snapshots are never deleted. Skip snapshot expiration for long enough and storage just keeps piling up.
3. Think through your catalog choice first. Iceberg's table pointer needs a catalog to manage it (Hive Metastore, JDBC, and others all work). For different engines to access the same table they must share the same catalog, and migrating later is painful.
4. Delta Lake and Apache Hudi are peer projects — similar positioning, different ecosystem emphasis. Before choosing, compare them against the compute engine you actually use most; don't just follow the crowd.

Wrapping Up

Data lakes solve the "store anything" problem, but storage without management turns a lake into a swamp fast. Table formats like Iceberg fill in transactions, versioning, partitioning, and schema management on top of the files, giving the lake table semantics close to a warehouse's while keeping the advantages of open formats and low cost. For me, the key to understanding it is that three-level metadata structure — once you see how snapshots and manifest files are organized, rollback, time travel, and partition pruning all fall into place naturally. When I get the chance, I'll write up the hands-on experience of reading and writing Iceberg tables from Flink / Spark separately.