Skip to main content

An Introduction to HBase, Compared with MongoDB

· 11 min read
tanzhuo

HBase, short for Hadoop Database, is a highly reliable, high-performance, column-oriented, scalable distributed storage system that lets you build large-scale structured storage clusters on inexpensive PC servers.

HBase is designed to store and process massive amounts of data. Using nothing but standard hardware, it can handle enormous datasets with thousands of rows and columns, supporting PB-scale storage and queries at hundreds of millions of QPS.

It suits business scenarios that don't demand strict real-time behavior. HBase stores byte arrays and doesn't care about data types, which allows for a dynamic, flexible data model.

Architecture​

HBase consists of HMaster and HRegionServer nodes and follows a master-slave architecture. HBase splits a logical table into multiple data blocks called HRegions and stores them on HRegionServers.

The HMaster manages all HRegionServers. It stores no data itself — only the mapping from data to HRegionServers (the metadata).

All nodes in the cluster are coordinated by Zookeeper, which handles the various issues HBase may run into at runtime. The basic architecture looks like this:

Client: Communicates with the HMaster and HRegionServers via HBase's RPC mechanism, submitting requests and receiving results. Administrative operations go through RPC calls to the HMaster; data reads and writes go through RPC calls to the HRegionServers.

Zookeeper: Every node in the cluster registers its state with Zookeeper, so the HMaster can always know the health of each HRegionServer. It also removes the HMaster as a single point of failure.

HMaster: Manages all HRegionServers, tells each one which HRegions it should maintain, and monitors their health. When a new HRegionServer registers with the HMaster, the HMaster tells it to wait for data assignment; when an HRegion fails, the HMaster marks all the HRegions it was responsible for as unassigned and reassigns them to other HRegionServers. The HMaster has no single-point problem — HBase can start multiple HMasters, and Zookeeper's election mechanism ensures one is always running, improving cluster availability.

HRegion: When a table grows beyond a preset threshold, HBase automatically splits it into regions, each holding a subset of the table's rows. To the user, a table is one collection of data distinguished by primary key (RowKey); physically, a table is split into multiple blocks, each of which is an HRegion, identified by "table name + start/end key." An HRegion holds one contiguous range of a table's data, and the full table is stored across multiple HRegions.

HRegionServer: All HBase data normally lives in the underlying HDFS, and users access it through HRegionServers. Typically one node in the cluster runs one HRegionServer, and each HRegion is maintained by exactly one HRegionServer. The HRegionServer is responsible for serving user I/O requests and reading/writing data in HDFS — it is the core module of HBase. Internally, it manages a series of HRegion objects, each corresponding to a contiguous range of the logical table. An HRegion consists of multiple HStores, each of which stores one column family of the logical table. This means each column family is a concentrated storage unit, so for efficiency it's best to place columns with similar I/O characteristics in the same column family.

HStore: The heart of HBase storage, made up of a MemStore and StoreFiles. The MemStore is an in-memory buffer where incoming writes land first; when it fills up, it is flushed into a StoreFile (backed by HFile). When the number of StoreFiles crosses a threshold, a Compact operation merges multiple StoreFiles into one, performing version merging and data deletion along the way. In other words, HBase only ever appends data — all updates and deletes happen during later Compact operations — so a user write can return as soon as it lands in memory, which is what gives HBase its write performance. Continual compaction gradually produces larger and larger StoreFiles; once a single StoreFile crosses a size threshold, a Split operation divides the current HRegion into two. The parent HRegion goes offline, the HMaster assigns the two child HRegions to appropriate HRegionServers, and the load on the original HRegion is spread across the two new ones.

HLog: Every HRegionServer has an HLog object — the class that implements the write-ahead log. Each time a user writes to the MemStore, a copy of the data is also written to the HLog file. HLog files roll periodically and old files are deleted (their data having been persisted to StoreFiles). When the HMaster detects via Zookeeper that an HRegionServer has terminated unexpectedly, it first processes the leftover HLog files: it splits the log data by HRegion and places it into the corresponding HRegion directories, then reassigns the failed HRegions. The HRegionServers taking over these HRegions notice during loading that there are historical HLogs to process, replay the HLog data into the MemStore, then flush to StoreFiles, completing data recovery.

HBase is based on the BigTable model: a sparse, long-term (HDFS-backed), multidimensional, sorted map. The table is indexed by row key, column key, and timestamp. All data in HBase is strings, with no types.

HBase Read and Write Paths​

The diagram below shows the data storage relationships inside an HRegionServer. As described above, HBase uses the MemStore and StoreFiles to store table updates. When data is updated, it is first written to the HLog and the MemStore; data in the MemStore is kept sorted.

When the MemStore accumulates to a threshold, a new MemStore is created and the old one is added to a flush queue, where a dedicated thread flushes it to disk as a StoreFile. At the same time, the system records a CheckPoint in Zookeeper, indicating that all data changes before that point have been persisted. If the system fails unexpectedly, data in the MemStore may be lost — in that case, the HLog is used to recover the data after the CheckPoint.

StoreFiles are read-only; once created they can never be modified, so an HBase update is really an append. When the number of StoreFiles in an HStore reaches a threshold, a merge combines modifications to the same key into one large StoreFile; when a StoreFile's size reaches a threshold, it is split into two StoreFiles.

Write Path​

  • Step 1: The client, coordinated through Zookeeper, sends a write request to the HRegionServer, writing the data into an HRegion.
  • Step 2: The data goes into the HRegion's MemStore until the MemStore reaches its preset threshold.
  • Step 3: The data in the MemStore is flushed into a StoreFile.
  • Step 4: As the number of StoreFiles grows past a threshold, a Compact operation merges multiple StoreFiles into one, performing version merging and data deletion in the process.
  • Step 5: Through continual Compact operations, StoreFiles gradually grow larger and larger.
  • Step 6: When a single StoreFile exceeds the size threshold, a Split operation divides the current HRegion into two new HRegions. The parent goes offline, the HMaster assigns the two children to appropriate HRegionServers, and the load on the original HRegion is distributed away.

Read Path​

  • Step 1: The client contacts Zookeeper, finds the -ROOT- table, and from there obtains the .META. table information.
  • Step 2: From the .META. table, it looks up which HRegion holds the target data and finds the corresponding HRegionServer.
  • Step 3: It fetches the data through that HRegionServer.
  • Step 4: An HRegionServer's memory is divided into the MemStore and the BlockCache — the MemStore mainly serves writes, the BlockCache mainly serves reads. A read first checks the MemStore, then the BlockCache, and only then goes to the StoreFiles, placing the fetched result into the BlockCache.

When to Use HBase​

Semi-structured or unstructured data: for data whose fields can't be defined in advance or are messy, a fixed schema is hard to model, making it a good fit for HBase. As the business grows and needs more fields, an RDBMS requires downtime to alter the table structure, whereas HBase supports adding columns dynamically.

Very sparse records: an RDBMS table has a fixed set of columns, and empty columns simply waste storage; in HBase, empty columns take no storage at all, saving space and improving read performance.

Multi-versioned data: the value located by a RowKey and column qualifier can have any number of versions (with different timestamps), so HBase is very convenient for data whose change history needs to be kept.

Massive data volumes: as data keeps growing, an RDBMS gradually buckles. First you split reads from writes — one primary for writes, several replicas for reads, doubling server costs. As pressure keeps mounting and the primary can't keep up, you start sharding by database, splitting out nearly unrelated data, which breaks some join queries and forces you to introduce a middle layer. As volume grows further and single tables balloon, queries slow to a crawl, and you're forced to shard tables too (e.g., splitting by ID modulo) to keep per-table row counts down. Anyone who has lived through this knows how tedious it is.

HBase is far simpler: just add new nodes to the cluster and HBase scales out automatically, while seamless integration with Hadoop provides data reliability (HDFS) and high-performance analytics over massive data (MapReduce).

HBase and MapReduce​

The relationship between a Table and its Regions in HBase resembles that between a File and its Blocks in HDFS. Because HBase provides APIs for interacting with MapReduce — such as TableInputFormat and TableOutputFormat — HBase tables can serve directly as the input and output of Hadoop MapReduce jobs, simplifying MapReduce application development without requiring knowledge of HBase internals.

Data Structures​

HBase uses an LSM (Log-Structured Merge) tree, whereas the traditional database engine InnoDB uses a B+ tree.

Characteristics of the LSM tree:

  1. Optimized for writes
  2. Reads must consult multiple trees, requiring more I/O operations than a B+ tree, with higher variance
  3. Writes are all sequential I/O; random reads are random I/O, sequential reads are sequential I/O

Advantages:

  1. Dramatically better write performance
  2. Unaffected by SSD random-write amplification
  3. Unaffected by space amplification

Disadvantages:

  1. Read performance is sacrificed — more I/O operations than a B+ tree
  2. Requires periodic compaction, which amplifies overall network/disk I/O

Strengths of HBase​

  1. Data is stored by column, so queries touch only the columns involved, dramatically reducing system I/O; data in the same column shares a type and empty values aren't stored, enabling efficient compression.
  2. The key/value storage model means query performance barely degrades even as data grows to massive scale.
  3. As a columnar database (versus a traditional row-oriented one), when a table has many fields, different columns (in units of Regions) can be spread across different server instances to distribute load.

Weaknesses of HBase​

  1. No native secondary indexes — access is by primary key only; community-built secondary indexes lag behind data updates, causing painful consistency issues
  2. The wide-table model is awkward and hard to grasp, and it demands up-front modeling, limiting flexibility
  3. Few supported programming interfaces (Java, Thrift, RESTful API); no SQL — the API is the only option
  4. Complex cluster structure, with 8 different node types
  5. Primitive data types — only byte streams — which is unfriendly for development
  6. No consistent snapshot capability
  7. Requires periodic compaction, which severely affects continuous read/write workloads
  8. No table joins, so data analysis is a weak point — common operations like group by or order by can only be done by writing MapReduce jobs

Wrapping Up​

HBase excels at extremely high-volume write scenarios: ingest the data, then process and analyze it asynchronously. The matching business cases are those with loose real-time requirements but very heavy write volumes, such as monitoring and collecting data from hardware and vehicle systems, or user profiling. It primarily solves massive-scale storage; paired with downstream analytics components, it can power big-data analysis, user behavior tracking, real-time recommendations, risk control, and more.

This post hasn't said much about MongoDB. On equivalent cluster hardware, MongoDB's write throughput trails HBase's; but MongoDB builds indexes as data is written, so both its query speed and the dimensions it can query on exceed HBase's — HBase can only query along limited dimensions. In terms of query flexibility and efficiency under varied conditions, MongoDB has the edge and better suits scenarios needing real-time query and analysis. To raise MongoDB's write capacity, you add more high-performance cluster nodes. When actually choosing between them, analyze the fit against your own business scenario.

COMMENTS