docker部署hadoop集群

镜像一： cdh6.3.2

比较占用内存+启动不需要的组件

Docker Hub

$ docker run --hostname=quickstart.cloudera --privileged=true -t -i -p 8888:8888 -p 8050:8050 -p 8051:8051 -p 25000:25000 -p 25010:25010 -p 25020:25020 -p 9870:9870 -p 8088:8088 -p 50070:50070 -p 14000:14000 -p 3306:3306  loganseo/cloudera-quickstart-docker:cdh6.3.2 /tmp/cdh_startup.sh

• Apache Impala Daemon http://localhost:25000/

• Apache Impala StateStore http://localhost:25010/

• Apache Impala Catalog http://localhost:25020/

• Hue http://localhost:8888/

• Namenode information http://localhost:9870/dfshealth.html#tab-overview

• YARN Resource Manager - All Applications http://localhost:8088/cluster

• Kudu Master http://localhost:8051/

• Kudu Tablet Server http://localhost:8050/

镜像二：Hadoop Custer

基本可用. 占用内存偏多.

 

相关： https://github.com/kiwenlau/hadoop-cluster-docker

1. pull docker image

sudo docker pull kiwenlau/hadoop:1.0

2. clone github repository

git clone https://github.com/kiwenlau/hadoop-cluster-docker

3. create hadoop network

sudo docker network create --driver=bridge hadoop

4. start container

cd hadoop-cluster-docker
vim ./start-container.sh
#修改 3节点->2节点 N=${1:-2}
#增加 -p 9000:9000 \
sudo ./start-container.sh

5.start hadoop

./start-hadoop.sh

6.run wordcount

./run-wordcount.sh

Hadoop网页管理地址:

• NameNode: http://dkmaster:50070/

• ResourceManager: http://dkmaster:8088/

镜像三： bde2020/hadoop

3.1 Hadoop3 版本套餐

GitHub - big-data-europe/docker-hadoop: Apache Hadoop docker image

vim docker-compose.yml
# 增加端口映射
docker-compose up

 

 

Hadoop3.x相较于之前的版本来说，在端口的设置上发生了一些变化，先就这些端口变化总结如下：

Namenode ports: 50470 --> 9871, 50070 --> 9870, 8020 --> 9820 Secondary NN ports: 50091 --> 9869, 50090 --> 9868 Datanode ports: 50020 --> 9867, 50010 --> 9866, 50475 --> 9865, 50075 --> 9864

3.2 满足基本需要 Hadoop2+Hive

https://github.com/big-data-europe/docker-hive

vim docker-compose.yml 
#namenode 增加 ports: - "8020:8020"
#hive-metastore-postgresql 增加 ports: - "5432:5432" 增加挂载 -v /mydata/postgresql:/var/lib/postgresql
docker-compose up

 

 

 

把不需要的 dockerhivemaster_presto-coordinator_1 停掉

修改副本数 -> 否则 flink checkpoint 失败

apt-get install vim
vim hdfs-site.xml
# 添加
<property><name>dfs.replication</name><value>1</value></property>

强制刷新该目录下创建的文件应为1副本(含以后创建)

hdfs dfs -setrep -R 1 /