镜像一: cdh6.3.2
比较占用内存+启动不需要的组件
$ docker run --hostname=quickstart.cloudera --privileged=true -t -i -p 8888:8888 -p 8050:8050 -p 8051:8051 -p 25000:25000 -p 25010:25010 -p 25020:25020 -p 9870:9870 -p 8088:8088 -p 50070:50070 -p 14000:14000 -p 3306:3306 loganseo/cloudera-quickstart-docker:cdh6.3.2 /tmp/cdh_startup.sh
-
Apache Impala Daemon http://localhost:25000/
-
Apache Impala StateStore http://localhost:25010/
-
Apache Impala Catalog http://localhost:25020/
-
Namenode information http://localhost:9870/dfshealth.html#tab-overview
-
YARN Resource Manager - All Applications http://localhost:8088/cluster
-
Kudu Master http://localhost:8051/
-
Kudu Tablet Server http://localhost:8050/
镜像二:Hadoop Custer
基本可用. 占用内存偏多.
相关: https://github.com/kiwenlau/hadoop-cluster-docker
1. pull docker image
sudo docker pull kiwenlau/hadoop:1.0
2. clone github repository
git clone https://github.com/kiwenlau/hadoop-cluster-docker
3. create hadoop network
sudo docker network create --driver=bridge hadoop
4. start container
cd hadoop-cluster-docker vim ./start-container.sh #修改 3节点->2节点 N=${1:-2} #增加 -p 9000:9000 \ sudo ./start-container.sh
5.start hadoop
./start-hadoop.sh
6.run wordcount
./run-wordcount.sh
Hadoop网页管理地址:
-
NameNode: http://dkmaster:50070/
-
ResourceManager: http://dkmaster:8088/
镜像三: bde2020/hadoop
3.1 Hadoop3 版本套餐
GitHub - big-data-europe/docker-hadoop: Apache Hadoop docker image
vim docker-compose.yml # 增加端口映射 docker-compose up
Hadoop3.x相较于之前的版本来说,在端口的设置上发生了一些变化,先就这些端口变化总结如下:
Namenode ports: 50470 --> 9871, 50070 --> 9870, 8020 --> 9820 Secondary NN ports: 50091 --> 9869, 50090 --> 9868 Datanode ports: 50020 --> 9867, 50010 --> 9866, 50475 --> 9865, 50075 --> 9864
3.2 满足基本需要 Hadoop2+Hive
https://github.com/big-data-europe/docker-hive
vim docker-compose.yml #namenode 增加 ports: - "8020:8020" #hive-metastore-postgresql 增加 ports: - "5432:5432" 增加挂载 -v /mydata/postgresql:/var/lib/postgresql docker-compose up
把不需要的 dockerhivemaster_presto-coordinator_1 停掉
修改副本数 -> 否则 flink checkpoint 失败
apt-get install vim vim hdfs-site.xml # 添加 <property><name>dfs.replication</name><value>1</value></property>
强制刷新该目录下创建的文件应为1副本(含以后创建)
hdfs dfs -setrep -R 1 /