开源 数据仓库_使用这些开源工具进行数据仓库

开源 数据仓库

by Simon Späti

西蒙·斯派蒂(SimonSpäti)

使用这些开源工具进行数据仓库 (Use these open-source tools for Data Warehousing)

These days, everyone talks about open-source software. However, this is still not common in the Data Warehousing (DWH) field. Why is this?

如今，每个人都在谈论开源软件。 但是，这在数据仓库(DWH)字段中仍然不常见。 为什么是这样？

For this post, I chose some open-source technologies and used them together to build a full data architecture for a Data Warehouse system.

在这篇文章中，我选择了一些开源技术，并将它们一起用于构建数据仓库系统的完整数据体系结构。

I went with Apache Druid for data storage, Apache Superset for querying, and Apache Airflow as a task orchestrator.

我使用Apache Druid进行数据存储，使用Apache Superset进行查询，并使用Apache Airflow作为任务编排器。

德鲁伊—数据存储 (Druid — the data store)

Druid is an open-source, column-oriented, distributed data store written in Java. It’s designed to quickly ingest massive quantities of event data, and provide low-latency queries on top of the data.

Druid是一个用Java编写的开源，面向列的分布式数据存储。 它旨在快速提取大量事件数据，并在数据之上提供低延迟查询。

为什么要使用德鲁伊？ (Why use Druid?)

Druid has many key features, including sub-second OLAP queries, real-time streaming ingestion, scalability, and cost effectiveness.

Druid具有许多关键功能，包括亚秒级OLAP查询，实时流接收，可伸缩性和成本效益。

With the comparison of modern OLAP Technologies in mind, I chose Druid over ClickHouse, Pinot and Apache Kylin. Recently, Microsoft announced they will add Druid to their Azure HDInsight 4.0.

考虑到现代OLAP技术的比较 ，我选择了Druid而不是ClickHouse，Pinot和Apache Kylin。 最近， Microsoft宣布将把Druid添加到其Azure HDInsight 4.0中。

为什么不德鲁伊？ (Why not Druid?)

Carter Shanklin wrote a detailed post about Druid’s limitations at Horthonwork.com. The main issue is with its support for SQL joins, and advanced SQL capabilities.

Carter Shanklin在Horthonwork.com上写了一篇有关Druid局限性的详细文章 。 主要问题是它对SQL连接的支持以及高级SQL功能。

德鲁伊的体系结构 (The Architecture of Druid)