A Real-Time Data Warehouse on Alibaba Cloud

Self-hosted big data clusters tend to be complex, hard to maintain, and expensive to deploy. Alibaba Cloud's product ecosystem lets you sidestep all of that. This post documents a real-time data warehouse architecture built entirely on Alibaba Cloud managed products—just enable the services you need to match your business requirements.

Pain Points

1. The underlying database can't handle massive data volumes. As a business grows, 10 TB, 100 TB, or even PB- and EB-scale data no longer fits, and fast query response, data analysis, and data mining become impossible to support.

2. Real-time compute performance falls short. You need a reliable compute engine for millisecond-level real-time processing, with data quality that is dependable, controllable, and observable.

3. Data model changes are too slow. Without the flexibility to restructure data models quickly, it's hard to keep up with reporting demands from the business.

Use Cases

• Real-time risk control with Flink and a rules engine
• Building a real-time anomaly detection system with Realtime Compute (Flink) and Gaussian models
• Building a simple real-time recommendation system with Realtime Compute (Flink)

The Real-Time Data Warehouse

Overall data development workflow:

The pipeline boils down to: data ingestion -> data buffering -> real-time compute -> sink to storage.

Component Selection

Flink

Alibaba Cloud Realtime Compute for Apache Flink

A fully managed Flink real-time compute engine from Alibaba Cloud, sparing you from building Yarn and Flink nodes yourself and from all the tedious configuration work. Its job is to process business data flowing through the DataHub message bus and route computed results downstream according to each data stream's business scenario. Being part of the Alibaba Cloud ecosystem also makes metadata management much more convenient.

DataHub

DataHub message bus

DataHub serves as the data buffering layer, providing high-performance storage and streaming transport for real-time data. Intermediate DWD and DWS layer data produced during Flink computation, as well as downstream data, all pass through it for task forwarding and storage—it acts as the message bus. A single topic can handle terabytes of reads and writes per day, with efficient and stable data flow. This component is also fully managed, so you never think about node counts.

Hologres

Hologres real-time data warehouse: unified interactive analytics and serving for big data

After Flink finishes computing, output for each layer sinks into the Hologres warehouse. Hologres supports row-oriented storage, column-oriented storage, and field-based partitioned storage, solving the massive-data storage problem and delivering sub-second analytics on PB-scale data. Under the hood, Hologres stores data on Pangu, Alibaba's in-house distributed file system (similar to HDFS). Architecturally it follows big data warehouse design principles with full Storage Disaggregation—compute nodes scale out horizontally to increase parallelism and keep queries over massive data fast.

Processing Pipeline

Warehouse Layers

In the big data world, ODS, DWD, DWS, and ADS are common data layering concepts used to organize and manage data, each playing a distinct role in how data is processed and consumed.

1. ODS (Operational Data Store): a storage layer for operational business data, typically holding raw data from source systems while preserving its completeness and detail. The ODS layer primarily supports real-time or near-real-time operational needs, such as online transaction processing and real-time queries. Data at this layer usually undergoes little heavy processing or computation—it serves operational workloads directly.
2. DWD (Data Warehouse Detail): the detail layer of the data warehouse, storing data that has been cleansed, integrated, and transformed—typically subject-oriented, queryable data. DWD data has been through ETL (extract, transform, load) and supports analytics, reporting, and decision support. It offers high quality and consistency, usually organized as fact tables and dimension tables.
3. DWS (Data Warehouse Summary): stores aggregated, summarized, and precomputed data, providing higher-level data summaries and better analytical performance. DWS data is typically organized as summary tables or precomputed metrics, supporting complex analytics, data mining, and business intelligence needs.
4. ADS (Application Data Service): an application-facing data storage layer holding data that has been further processed, computed, and optimized. ADS data is usually application-specific views or datasets, adapted and tuned to deliver faster data access and processing.

To sum up: the ODS layer serves operational workloads and preserves raw data intact; the DWD layer provides cleansed, integrated, transformed data ready to query; the DWS layer trades summarization and precomputation for better query performance; and the ADS layer serves specific applications with processed, optimized data.

This layered design brings several benefits:

1. Separation of data processing and management: splitting data into layers also splits the responsibilities. The DWD layer handles extraction, cleansing, and integration of raw data; the DWS layer handles aggregation and summarization; the ADS layer provides application-facing data services. Data processing becomes modular and more flexible.
2. Data quality and consistency: the DWD layer preserves the completeness and detail of raw data, useful for data quality analysis and lineage tracing; the DWS layer offers aggregated data with better query performance; the ADS layer guarantees consistency and accuracy across different application scenarios.
3. Query performance and development efficiency: through summarization and precomputation, the DWS layer speeds up analytics and decision-making; the ADS layer exposes application-oriented data services and interfaces, simplifying application development and boosting productivity.

Wrap-Up

The core idea of this architecture is to hand cluster operations over to the cloud vendor: DataHub buffers and routes data, Flink handles real-time compute, and Hologres takes care of storage and analytics—the whole pipeline is managed, with no nodes of your own to maintain. Combined with the ODS/DWD/DWS/ADS layered design, each layer has a clear responsibility, and both query performance and reporting productivity are well covered. For teams that don't want to sink headcount into a self-hosted big data cluster, this is a pragmatic path.