docker镜像制作之dockercompose.yml文件---hadoop伪分布式

最新推荐文章于 2024-08-16 10:53:46 发布

xuguokun1986

最新推荐文章于 2024-08-16 10:53:46 发布

阅读量3.2k

点赞数

分类专栏： Docker hadoop

本文链接：https://blog.csdn.net/xuguokun1986/article/details/52962847

版权

本文介绍了如何在Docker环境中搭建Hadoop伪分布式集群，包括制作Hadoop基础镜像、构建各节点镜像，以及使用docker-compose编排集群的过程。

摘要由CSDN通过智能技术生成

一、其实对于hadoop集群不是太适合放在docker服务器里面来跑，因为docker提倡容器和服务是1:1的关系，但是

hadoop提倡datanode和nodemanager在一个节点上（容器），但是当docker使用swarm之后，还是可以考虑将hadoop

集群的各个服务扔进容器里面。

二、构建hadoop集群的基础镜像

1.构建hadoop集群的基础镜像需要如下文件：

├── build.sh
├── Dockerfile
├── entrypoint.sh
├── hadoop-2.7.3.tar.gz
└── jdk-8u92-linux-x64.tar.gz

2.说明：hadoop-2.7.3.tar.gz和jdk-8u92-linux-x64.tar.gz需要提前下载后放在该目录下

3.Dockerfile文件的内容如下

FROM debian:jessie-backports

RUN apt-get update \
    #&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends openjdk-8-jdk \
    && rm -rf /var/lib/apt/lists/*

ENV  JAVA_VERSION jdk1.8.0_92
COPY jdk-8u92-linux-x64.tar.gz /opt
RUN  tar -zxvf /opt/jdk-8u92-linux-x64.tar.gz -C /opt

ENV JAVA_HOME=/opt/$JAVA_VERSION/

RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends net-tools curl

RUN gpg --keyserver pool.sks-keyservers.net --recv-keys \
    07617D4968B34D8F13D56E20BE5AAA0BA210C095 \
    2CAC83124870D88586166115220F69801F27E622 \
    4B96409A098DBD511DF2BC18DBAF69BEA7239D59 \
    9DD955653083EFED6171256408458C39E964B5FF \
    B6B3F7EDA5BA7D1E827DE5180DFF492D8EE2F25C \
    6A67379BEFC1AE4D5595770A34005598B8F47547 \
    47660BC98BC433F01E5C90581209E7F13D0C92B9 \
    CE83449FDC6DACF9D24174DCD1F99F6EE3CD2163 \
    A11DF05DEA40DA19CE4B43C01214CF3F852ADB85 \
    686E5EDF04A4830554160910DF0F5BBC30CD0996 \
    5BAE7CB144D05AD1BB1C47C75C6CC6EFABE49180 \
    AF7610D2E378B33AB026D7574FB955854318F669 \
    6AE70A2A38F466A5D683F939255ADF56C36C5F0F \
    70F7AB3B62257ABFBD0618D79FDB12767CC7352A \
    842AAB2D0BC5415B4E19D429A342433A56D8D31A \
    1B5D384B734F368052862EB55E43CAB9AEC77EAF \
    785436A782586B71829C67A04169AA27ECB31663 \
    5E49DA09E2EC9950733A4FF48F1895E97869A2FB \
    A13B3869454536F1852C17D0477E02D33DD51430 \
    A6220FFCC86FE81CE5AAC880E3814B59E4E11856 \
    EFE2E7C571309FE00BEBA78D5E314EEF7340E1CB \
    EB34498A9261F343F09F60E0A9510905F0B000F0 \
    3442A6594268AC7B88F5C1D25104A731B021B57F \
    6E83C32562C909D289E6C3D98B25B9B71EFF7770 \
    E9216532BF11728C86A11E3132CF4BF4E72E74D3 \
 E8966520DA24E9642E119A5F13971DA39475BD5D \
    1D369094D4CFAC140E0EF05E992230B1EB8C6EFA \
    A312CE6A1FA98892CB2C44EBA79AB712DE5868E6 \
    0445B7BFC4515847C157ECD16BA72FF1C99785DE \
    B74F188889D159F3D7E64A7F348C6D7A0DCED714 \
    4A6AC5C675B6155682729C9E08D51A0A7501105C \
    8B44A05C308955D191956559A5CEE20A90348D47

ENV HADOOP_VERSION 2.7.3

COPY hadoop-$HADOOP_VERSION.tar.gz /opt
RUN  tar  -zxvf /opt/hadoop-$HADOOP_VERSION.tar.gz -C /opt
RUN  rm /opt/hadoop-$HADOOP_VERSION.tar.gz

RUN ln -s /opt/hadoop-$HADOOP_VERSION/etc/hadoop /etc/hadoop
RUN cp /etc/hadoop/mapred-site.xml.template /etc/hadoop/mapred-site.xml
RUN mkdir /opt/hadoop-$HADOOP_VERSION/logs

RUN mkdir /hadoop-data

ENV HADOOP_PREFIX=/opt/hadoop-$HADOOP_VERSION
ENV HADOOP_CONF_DIR=/etc/hadoop
ENV MULTIHOMED_NETWORK=1

ENV USER=root
ENV PATH $HADOOP_PREFIX/bin/:$PATH

ADD entrypoint.sh /entrypoint.sh
RUN chmod a+x /entrypoint.sh

ENTRYPOINT ["/entrypoint.sh"]

4. entrypoint.sh文件内容如下

#!/bin/bash

# Set some sensible defaults
export CORE_CONF_fs_defaultFS=${CORE_CONF_fs_defaultFS:-hdfs://`hostname -f`:9000}

function addProperty() {
  local path=$1
  local name=$2
  local value=$3

  local entry="<property><name>$name</name><value>${value}</value></property>"
  local escapedEntry=$(echo $entry | sed 's/\//\\\//g')
  sed -i "/<\/configuration>/ s/.*/${escapedE