zookeeper集群上搭建clickhouse集群,傻瓜式教程
文章目录
环境:centos 7
一、zookeeper集群搭建
Ⅰ.下载zk
https://zookeeper.apache.org/releases.html#news 官网下载
Ⅱ.安装zk
以下每步操作zk集群每个zk均进行操作
- 创建目录并解压到对应目录下,这里是/opt/zookeeper/apache-zookeeper-3.7.0-bin
tar zxvf apache-zookeeper-3.7.0-bin.tar.gz
-
在apache-zookeeper-3.7.0-bin目录下创建 data与log文件夹,Ⅲ中使用
mkdir data
mkdir log
Ⅲ.zk配置修改
以下每步操作zk集群每个zk均进行操作
-
进入zk的conf目录, 并对zoo_sample.cfg文件重命名为zoo.cfg
cd /opt/zookeeper/apache-zookeeper-3.7.0-bin/conf
mv zoo_sample.cfg zoo.cfg
-
打开zoo.cfg文件,进行修改
-
配置dataDir、dataLogDir
dataDir=/opt/zookeeper/apache-zookeeper-3.7.0-bin/data dataLog=/opt/zookeeper/apache-zookeeper-3.7.0-bin/log
-
设置zk服务器地址(服务器zk1,zk2,zk3、选举端口2888、心跳端口3888)
server.1=zk1:2888:3888 server.2=zk2:2888:3888 server.3=zk3:2888:3888
-
-
data目录下,设置集群id
cd /opt/zookeeper/apache-zookeeper-3.7.0-bin/data/ touch myid vi myid #zk集群各个zk分别设置为1,2,3...
!!!此处设置的数字与2中设置的1,2,3对应,每个机器不同
-
修改各个zk机器hosts文件,增加别名(zk1,zk2,zk3)
vi /etc/hosts
格式为:192.168.19.123 zk1…
添加完成后,可以互相通过别名ping通,如:
Ⅳ.zk集群启动
-
切换到bin目录,启动zk
cd /opt/zookeeper/apache-zookeeper-3.7.0-bin/bin ./zkServer.sh start
-
查看status
./zkServer.sh status
至此,zk集群搭建完成!!
二、clickhouse集群搭建
Ⅰ.ck单机搭建
每台服务器都需要搭建单机ck,在此基础上进行集群搭建
-
确保服务器支持SSE指令集
grep -q sse4_2 /proc/cpuinfo && echo "SSE 4.2 supported" || echo "SSE 4.2 not supported"
![在这里插入图片描述](https://img-blog.csdnimg.cn/20210402235118402.png#pic_center)
- 安装clickhouse(这里使用rpm在线安装)
#添加官方存储库
sudo yum install yum-utils
sudo rpm --import https://repo.clickhouse.tech/CLICKHOUSE-KEY.GPG
sudo yum-config-manager --add-repo https://repo.clickhouse.tech/rpm/stable/x86_64
#安装
sudo yum install clickhouse-server clickhouse-client
-
切换到 /usr/bin/目录启动、停止clickhouse的服务端与客户端
cd /usr/bin/ systemctl start clickhouse-server systemctl start clickhouse-server clickhouse-client
可以查看到所有数据库
show databases;
Ⅱ.ck集群搭建
1.集群配置
- 集群配置可以在config.xml编辑,但推荐通过 编辑metrika.xml配置集群
cd /etc/clickhouse-server/config.d/
vim metrika.xml
<?xml version="1.0"?>
<yandex>
<clickhouse_remote_servers>
<cluster> <!--集群名称-->
<shard><!--定义分片节点,这里三个分片,每个分片一个副本-->
<internal_replication>true</internal_replication>
<replica>
<host>zk1</host>
<port>9000</port>
</replica>
</shard>
<shard>
<replica>
<internal_replication>true</internal_replication>
<host>zk2</host>
<port>9000</port>
</replica>
</shard>
<shard>
<internal_replication>true</internal_replication>
<replica>
<host>zk3</host>
<port>9000</port>
</replica>
</shard>
</cluster>
</clickhouse_remote_servers>
<!--zookeeper集群,对应前面搭建的zk集群信息-->
<zookeeper-servers>
<node index="1">
<host>zk1</host>
<port>2181</port>
</node>
<node index="2">
<host>zk2</host>
<port>2181</port>
</node>
<node index="3">
<host>zk3</host>
<port>2181</port>
</node>
</zookeeper-servers>
<macros> <!--定义宏变量-->
<replica>zk1</replica>
</macros>
<networks> <!--不限制访问来源ip地址-->
<ip>::/0</ip>
</networks>
<clickhouse_compression><!--数据压缩方式,默认为lz4-->
<case>
<min_part_size>10000000000</min_part_size>
<min_part_size_ratio>0.01</min_part_size_ratio>
<method>lz4</method>
</case>
</clickhouse_compression>
</yandex>
- 修改/etc/clickhouse-server/下的config.xml
<yandex>
<!--引入metrika.xml-->
<include_from>/etc/clickhouse-server/config.d/metrika.xml</include_from>
<!--引用Zookeeper配置的定义-->
<zookeeper incl="zookeeper-servers" optional="true" />
...
<listen_host>::</listen_host> <!--取消该注释,开启远程访问-->
...
<!--clickhouse_remote_servers节点引入-->
<remote_servers incl="clickhouse_remote_servers">
<!-- Test only shard config for testing distributed storage -->
<test_shard_localhost>
<!-- Inter-server per-cluster secret for Distributed queries
default: no secret (no authentication will be performed)
If set, then Distributed queries will be validated on shards, so at least:
- such cluster should exist on the shard,
- such cluster should have the same secret.
...
注意:此处remote_servers节点要引入clickhouse_remote_servers!!!
2.clickhouse集群启动
-
按照前面方式启动zk集群,确保zk集群成功启动
-
启动ck集群
sudo clickhouse start
启动失败可以查看日志,确定失败原因
-
进入clickhouse客户端,并查看集群是否搭建成功
cd /usr/bin/ clickhouse-client select * from system.clusters;
可以看到图中集群信息,若无则集群搭建失败,仔细查看cluster日志,找原因,日志地址在config.xml中,默认为:/var/log/clickhouse-server/clickhouse-server.log
至此,clickhouse集群搭建完毕~