概要
这个文章主要讲已经理解docker的前提下,快速利用docker安装各种组件的全集,所有的命令都已经测试过,作为新如果公司的大数据工程师是非常快速有效的
docker 安装
安装docker
sudo yum install docker-ce docker-ce-cli containerd.io
安装docker compose
curl -L "https://github.com/docker/compose/releases/download/1.23.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
如果新机器,想要删除旧版本,切记,看清楚是否需要删除
sudo yum remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-engine
非root用户想要用root下安装的docker怎么办?
sudo groupadd docker #添加docker用户组
sudo gpasswd -a $USER docker #将登陆用户加入到docker用户组中
newgrp docker #更新用户组
docker ps #测试docker命令是否可以使用sudo正常使用
scp -r /Users/jiangwenfeng/docker/hummingbird-docker root@centos7:/data/docker
也可以用比较简单的安装docker
yum install docker
docker -v
替换阿里源
mkdir -p /etc/docker && touch /etc/docker/daemon.json
echo { "registry-mirrors": ["https://rvof9czj.mirror.aliyuncs.com"] } > /etc/docker/daemon.json
刷新
systemctl daemon-reload
重启
systemctl restart docker
查看是否生效
docker info|grep "Registry Mirrors" -A 1
输出
Registry Mirrors: https://rvof9czj.mirror.aliyuncs.com/
mysql 安装
版本可以自己选
docker search mysql
下载镜像
docker pull mysql:5.7
启动前构建本地目录用于存储mysql的日志和数据
mkdir -p /data/docker/data/mysql/log /data/docker/data/mysql/data
启动
docker run -d -p 3308:3306 -v /data/docker/data/mysql/log:/var/log/mysql \
-v /data/docker/data/mysql/data:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=123456 mysql:5.7 \
--character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci --这个参数可以去参考mysql的配置任意指定
Redis
下载
docker pull redis
启动
docker run --name redis -v /data/docker/redis/data:/data -p 6379:6379 -d --restart=always redis:latest redis-server --appendonly yes --requirepass "redis"
进入容器
docker exec -it xxxx redis-cli
配置初始密码
auth redis
config get requirepass
密码就是redis
elasticsearch
版本可以自己选,去dockerhub的官网上都可以看到
非持久化方案
docker pull docker.elastic.co/elasticsearch/elasticsearch:7.5.0
非持久化的方案
docker run -d --name es -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" \
docker.elastic.co/elasticsearch/elasticsearch:7.5.0
持久化方案
mkdir -p /data/elasticsearch/data /data/elasticsearch/logs
docker run --name es \
-p 9201:9200 -p 9301:9300 \
-v /data/elasticsearch/data:/var/lib/elasticsearch \
-v /data/elasticsearch/logs:/var/log/elasticsearch \
-d docker.elastic.co/elasticsearch/elasticsearch:7.5.0 \
-v /data/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
非必须
docker pull mobz/elasticsearch-head
docker run -d -p 9100:9100 --name elasticsearch-head elasticsearch-head
kafka && zookeeper
kafka是强依赖zookeeper的,所以这里一起介绍了
单机版
docker pull wurstmeister/zookeeper
docker pull wurstmeister/kafka
docker run -d --name zookeeper -p 2181:2181 wurstmeister/zookeeper
docker run -d --name kafka \
--env KAFKA_ADVERTISED_HOST_NAME=localhost \
--env KAFKA_ZOOKEEPER_CONNECT=localhost:2181 \
--env KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \
--env KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \
--env KAFKA_HEAP_OPTS="-Xmx256M -Xms128M" \
--net=host wurstmeister/kafka
单机版测试
单机测试:
docker exec -it kafka /bin/bash
cd /opt/kafka_2.12-2.4.0/
./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
./kafka-console-producer.sh --broker-list localhost:9092 --topic test
./kafka-topics.sh --list --zookeeper localhost:2181
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
集群版… 用dockercompose 的方式非常美好.但是我暂时忘记dockerfile的地址了,后面补充
kafka-manager
docker pull sheepkiller/kafka-manager
docker pull solsson/kafka-manager:latest
docker run -d -p 9999:9000 -e \
ZK_HOSTS=localhost:2181 solsson/kafka-manager
三. zookeeper
http://mirror.bit.edu.cn/apache/zookeeper/
RabbitMQ
docker pull rabbitmq
docker pull rabbitmq:management
docker run --name rabbitmq -d -p 15672:15672 -p 5672:5672 rabbitmq:management
pushgateway
docker pull prom/pushgateway
docker run -d -p 9091:9091 prom/pushgateway
验证:
http://localhost:9091/#
prometheus
docker pull prom/prometheus
docker run -d \
-p 9090:9090 \
-v $PWD/prometheus.yml:/etc/prometheus/prometheus.yml \
prom/prometheus
http://localhost:9090
grafana
docker pull grafana/grafana
docker run --rm -d \
-p 3000:3000 \
--name=grafana \
-v /data/docker/grafana:$PWD/grafana-storage \
grafana/grafana
clickhouse
docker pull yandex/clickhouse-server
docker pull yandex/clickhouse-client
docker run --cpus 8 --memory 20G --rm -it --name ck \
-v `pwd`/data/db:/var/lib/clickhouse \
-v `pwd`/data/logs:/var/log/clickhouse-server \
-v /etc/localtime:/etc/localtime \
-v /usr/share/zoneinfo:/usr/share/zoneinfo \
-v `pwd`/clickhouse-server:/etc/clickhouse-server \
-p8123:8123 -p9000:9000 -p9009:9009 docker.io/yandex/clickhouse-server
docker run -it --rm yandex/clickhouse-client \
--user root--password 123456 -m --host hostname
spark
http://
spark.apache.org/downloads.html
http://mirror.bit.edu.cn/apache/spark/spark-2.4.5/spark-2.4.5-bin-hadoop2.7.tgz
hive
http://mirror.bit.edu.cn/apache/hive/
hbase
http://mirrors.tuna.tsinghua.edu.cn/apache/hbase/
kafka
http://kafka.apache.org/downloads
http://mirror.bit.edu.cn/apache/kafka/2.4.0/kafka_2.11-2.4.0.tgz
flink
https://downloads.apache.org/flink/flink-1.10.0/apache-flink-1.10.0.tar.gz
flume
http://flume.apache.org/download.html