Hadoop相关技术

Apache的Hadoop是什么？

Apache的Hadoop项目™®开发出可靠的，可扩展的，分布式计算的开源软件。

Apache的Hadoop的软件库是一个框架，允许大型数据集通过计算机集群使用简单的编程模型，进行分布式处理。它的设计规模从单一服务器到数千台计算机，每个提供本地计算和存储。软件库是用来检测和处理应用层失败的，而不是依靠硬件提供高的有效度，因此在计算机集群上提供高度可用性服务，其中每个都有可能会有失败。

该项目包括这些模块：

•Hadoop Common：支持其他Hadoop模块的公共组件。

•Hadoop Distributed File System (HDFS™)：分布式文件系统提供了高吞吐量的访问应用程序数据。

•Hadoop YARN：作业调度和集群资源管理框架。

•Hadoop MapReduce：一种基于YARN的大型数据集的并行处理系统。

其他项目包括：

•Avro™（阿芙罗）：数据序列化系统。

•Cassandra™（卡桑德拉）：可扩展的没有单点故障的多路master数据库。

•Chukwa™：一个用于管理大型分布式系统的数据采集系统。

•HBase™：一个可扩展的，支持大型表的结构化数据存储的分布式数据库。

•Hive™：数据仓库的基础设施，提供数据汇总和即席查询。

•Mahout™：一个可扩展的机器学习和数据挖掘库。

•Pig™：一个高层次的数据流语言和执行框架的并行计算。

•ZooKeeper™：一个分布式应用程序的高性能的协调服务。

 

原文如下：

What Is Apache Hadoop?

The Apache™ Hadoop® project develops open-source software for reliable, scalable, distributed computing.

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-avaiability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-availabile service on top of a cluster of computers, each of which may be prone to failures.

The project includes these modules:

• Hadoop Common: The common utilities that support the other Hadoop modules.
• Hadoop Distributed File System (HDFS™): A distributed file system that provides high-throughput access to application data.
• Hadoop YARN: A framework for job scheduling and cluster resource management.
• Hadoop MapReduce: A YARN-based system for parallel processing of large data sets.

Other Hadoop-related projects at Apache include:

• Avro™: A data serialization system.
• Cassandra™: A scalable multi-master database with no single points of failure.
• Chukwa™: A data collection system for managing large distributed systems.
• HBase™: A scalable, distributed database that supports structured data storage for large tables.
• Hive™: A data warehouse infrastructure that provides data summarization and ad hoc querying.
• Mahout™: A Scalable machine learning and data mining library.
• Pig™: A high-level data-flow language and execution framework for parallel computation.
• ZooKeeper™: A high-performance coordination service for distributed applications.