【Big Data】笔记

YARN is Yet Another Resource Negotiator

YAML

Hadoop contains

- HDFS

- MapReduce

- YARN

- Commons

oozie: workflow scheduler system to manage Apache Hadoop jobs.

sqoop:

hbase: is a database

这节课讲总结Hadoop系统框架

后面会在AWS上用Hadoop, reason: easy to create clusters on AWS.

在一个clusters集群里，Namenode down了就都down了。

如果Namenode坏掉的话，就新建一个namenode，然后把metadata从secondaryNamenode里转过去。

一个Mapreduce有一个JobTracker， 一个Job被分成好几个task，每个task有一个taskTracker。

如果一个task fails了， 会报告给JobTracker然后从另外的datanode里读数据。

Node分成三种：Master Nodes，Namenode，SlaveNode

Slave Node：

Master Node：

Slave Node：

master 的daemon is namenode and jobtracker

slave's daemon is tasktracker and datanode

in real production clusters there is no server virtualization: reason: cause overwrite.

cat 5

cat 6 speed:

在顶层有一个router链接所有的机器，使这些机器能相互通信。

Typical Workflow in a hadoop system:

1.load data into the clusters (HDFS wrties)

2.analyze the data (Mapreduce)

3. store results in the cluster (HDFS writes)

4. read the results from the cluster (HDFS reads)

how to write files to HDFS

创建datanode一定要是different physical machine

Bandwidth and latency across racks