docker部署大数据软件

• 1. hdfs

使用Docker部署单节点hadoop操作如下：

docker pull sequenceiq/hadoop-docker:2.7.1 

docker run -it sequenceiq/hadoop-docker:2.7.1 /etc/bootstrap.sh -bash

• 2. zookeeper

使用Docker部署单节点zookeeper操作如下：

docker pull zookeeper

docker run -d -p 2182:2181 --name zookeeper zookeeper

• 3. hbase

待补充

• 4. kafka

• 4.1 docker部署kafka

执行如下命令编写docker-compose.yml 文件

vim docker-compose.yml

添加如下内容，192.168.1.223为宿主机ip。执行命令部署docker：docker-compose up

version: '3'

services:

  broker1:

    image: wurstmeister/kafka

    container_name: broker1

    ports:

      - "19011:22"

    environment:

      - KAFKA_ZOOKEEPER_CONNECT=192.168.1.223:2181

      - KAFKA_ADVERTISED_HOST_NAME=broker1

      - KAFKA_ADVERTISED_PORT=9092

      - KAFKA_BROKER_ID=1

    restart: always

  broker2:

    image: wurstmeister/kafka

    container_name: broker2

    depends_on:

      - broker1

    environment:

      - KAFKA_ZOOKEEPER_CONNECT=192.168.1.223:2181

      - KAFKA_ADVERTISED_HOST_NAME=broker2

      - KAFKA_ADVERTISED_PORT=9092

      - KAFKA_BROKER_ID=2

    ports:

      - "19012:22"

    restart: always

  broker3:

    image: wurstmeister/kafka

    container_name: broker3

    depends_on:

      - broker2

    environment:

      - KAFKA_ZOOKEEPER_CONNECT=192.168.1.223:2181

      - KAFKA_ADVERTISED_HOST_NAME=broker3

      - KAFKA_ADVERTISED_PORT=9092

      - KAFKA_BROKER_ID=3

    ports:

      - "19013:22"

    restart: always

• 4.2 验证kafka

执行命令，进入kafka broker1节点

docker exec –it broker1 bash

分别执行如下命令，创建kafka的topic，查看topic，生产消息命令

kafka-topics.sh --create --zookeeper 192.168.1.223:2181 --replication-factor 2 --partitions 3 --topic test002

kafka-topics.sh --describe --topic test002 --zookeeper 192.168.1.223:2181

kafka-console-producer.sh --broker-list 172.23.0.2:9092,172.23.0.3:9092,172.23.0.4:9092 --topic test001

• 5. docker-compose

docker-compose离线安装

下载

https://github.com/docker/compose/releases下载docker-compose-Linux-x86_64

重命名

mv docker-compose-Linux-x86_64 /usr/local/bin/docker-compose

添加可执行权限

chmod +x /usr/local/bin/docker-compose

参考链接

https://www.cnblogs.com/52fhy/p/5991344.html

Docker之Compose服务编排