clickhouse学习实践之分布式单副本集群部署

部署日期	参与人员	部署目标	版本
Mon 19 Sep 2022	@devops	部署一套本地化的分布式持久存储的Clickhouse集群	v0.1

集群信息

集群节点信息

主机名	IP地址	Clickhouse节点信息
bigdata1	10.0.10.195	clickhouse-node1
bigdata2	10.0.10.194	clickhouse-node2
Bigdata3	10.0.10.196	clickhouse-node3

节点基础信息

操作系统	内核版本	Docker版本	Clickhouse版本	安装方式
Ubuntu 20.04.3 LTS	5.4.0-91-generic	20.10.12	v22.8.5.29-lts	Apt包管理器

集群端口预留信息

端口	说明	备注
8123	http端口
9000	tcp端口

集群配置信息

配置项	配置信息
服务端口	- 8123: http端口 - 9000: tcp端口
数据目录	/data/clickhouse_bigdata
集群名称	bigdata_3shards_1replicas
配置文件路径	集群配置： - 默认配置文件：/etc/clickhouse-server/config.xml - zookeeper配置文件: /etc/clickhouse-server/config.d/zookeeper.xml - 宏配置文件: /etc/clickhouse-server/config.d/macros.xml - 集群配置切片副本配置文件: /etc/clickhouse-server/config.d/remote-servers.xml 用户授权认证配置: - 管理员clickhouse_manager：/etc/clickhouse-server/users.d/clickhouse_manager.xml - 普通用户clickhouse_operator: /etc/clickhouse-server/users.d/clickhouse_operator.xml
日志目录	/var/log/clickhouse-server/
分布式集群模式	3shards_1replicas
服务管理	sudo clickhouse [start/stop]
负载均衡地址	- HTTP: clickhouse.devopsman.cn - TCP: clickhouse-tcp.devopsman.cn

安装Clickhouse服务

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates dirmngr
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 8919F6BD2B48D754

echo "deb https://packages.clickhouse.com/deb stable main" | sudo tee \
    /etc/apt/sources.list.d/clickhouse.list
sudo apt-get update

apt-cache madison clickhouse-server | head -n5

sudo apt-get install -y clickhouse-server=22.8.5.29 clickhouse-client=22.8.5.29

# 修改clickhouse的家目录以及相关属主组
mkdir /home/clickhouse

chown -R clickhouse.clickhouse /home/clickhouse

usermod -d /home/clickhouse clickhouse

# 准备clickhouse的数据存储目录
mkdir /data/clickhouse_bigdata

chown -R clickhouse.clickhouse /data/clickhouse_bigdata

# 在clickhouse的默认配置文件中修改clickhouse的数据目录
sed -e 's#/var/lib/clickhouse#/data/clickhouse_bigdata#g' /etc/clickhouse-server/config.xml |grep -i '/data/clickhouse_bigdata'

sed -i 's#/var/lib/clickhouse#/data/clickhouse_bigdata#g' /etc/clickhouse-server/config.xml |grep -i '/data/clickhouse_bigdata'

# sudo service clickhouse-server start
# clickhouse-client # or "clickhouse-client --password" if you've set up a password.

配置节点域名解析

⚠️ 每个节点都需要配置

• 本地host配置

cat >> /etc/hosts << EOF
10.0.10.195 cknode1.devopsman.cn bigdata1 
10.0.10.194 cknode2.devopsman.cn bigdata2
10.0.10.196 cknode3.devopsman.cn bigdata3
EOF

• DNS解析平台上进行解析

ClickHouse 的集群配置

配置切片和副本remote-servers.xml

<clickhouse>
    <default_on_cluster_name>bigdata_3shards_1replicas</default_on_cluster_name>
    <default_replica_path>/clickhouse/tables/{cluster}/{shard}/{database}/{table}</default_replica_path>
    <default_replica_name>{replica}</default_replica_name>
    <remote_servers>
        <!-- User-specified clusters -->
        <bigdata_3shards_1replicas>
            <shard>
                <replica>
                    <host>cknode1.devopsman.cn</host>
                    <port>9000</port>
                </replica>
            </shard>
            <shard>
                <replica>
                    <host>cknode2.devopsman.cn</host>
                    <port>9000</port>
                </replica>
            </shard>
            <shard>
                <replica>
                    <host>cknode3.devopsman.cn</host>
                    <port>9000</port>
                </replica>
            </shard>
        </bigdata_3shards_1replicas>
    </remote_servers>
</clickhouse>

配置集群需要的Zookeeper信息zookeeper.xml

<clickhouse>
    <zookeeper>
        <node>
            <host>zookeeper-1.devopsman.cn</host>
            <port>2181</port>
        </node>
        <node>
            <host>zookeeper-2.devopsman.cn</host>
            <port>2181</port>
        </node>
        <node>
            <host>zookeeper-3.devopsman.cn</host>
            <port>2181</port>
        </node>
        <session_timeout_ms>30000</session_timeout_ms>
        <operation_timeout_ms>10000</operation_timeout_ms>
    </zookeeper>
    <distributed_ddl>
        <path>/clickhouse/bigdata/task_queue/ddl</path>
    </distributed_ddl>
</clickhouse>

宏macros.xml

<clickhouse>
    <macros>
        <installation>bigdata</installation>
        <cluster>bigdata_3shards_1replicas</cluster>
        <shard>2</shard>
	<replica>1</replica>
    </macros>
</clickhouse>

监听配置信息listen.xml

<clickhouse>
    <listen_try>1</listen_try>
    <listen_host>0.0.0.0</listen_host>
    <listen_host>127.0.0.1</listen_host>
</clickhouse>

ClickHouse 账号认证授权

管理员账号：clickhouse_manager.xml

<clickhouse>
    <users>
        <clickhouse_manager>
            <access_management>1</access_management>
            <networks>
                <ip>10.0.10.0/24</ip>
                <ip>127.0.0.1</ip>
                <ip>::/0</ip>
            </networks>
            <password_sha256_hex>ff3ebe0b384ba70a48ef866d0e4e8bfcc3104709f3858d95e51111111110e2e683cbb3da4</password_sha256_hex>
            <profile>clickhouse_manager</profile>
            <quota>default</quota>
        </clickhouse_manager>
    </users>
    <profiles>
        <clickhouse_manager>
            <log_queries>1</log_queries>
            <skip_unavailable_shards>1</skip_unavailable_shards>
            <http_connection_timeout>10</http_connection_timeout>
            <parallel_view_processing>1</parallel_view_processing>
        </clickhouse_manager>
    </profiles>
</clickhouse>

维护账号: clickhouse_operator.xml

<clickhouse>
    <users>
        <clickhouse_operator>
            <networks>
                <ip>10.0.10.0/24</ip>
                <ip>127.0.0.1</ip>
                <ip>::/0</ip>
            </networks>
            <password_sha256_hex>716b36073a90c6fe1d445ac1af85f4777c5b7a155cea35911111111111111</password_sha256_hex>
            <profile>clickhouse_operator</profile>
            <quota>default</quota>
        </clickhouse_operator>

    </users>
    <profiles>
        <clickhouse_operator>
            <log_queries>1</log_queries>
            <skip_unavailable_shards>1</skip_unavailable_shards>
            <http_connection_timeout>10</http_connection_timeout>
        </clickhouse_operator>
    </profiles>
</clickhouse>

在所有的配置文件配置完成后，切记要修改xml文件的所属属性

chown clickhouse.clickhouse /etc/clickhouse-server/config.d/*.xml /etc/clickhouse-server/users.d/*.xml

FQA

• 安装日志

Creating clickhouse group if it does not exist.
groupadd -r clickhouse
Creating clickhouse user if it does not exist.
useradd -r --shell /bin/false --home-dir /nonexistent -g clickhouse clickhouse
Will set ulimits for clickhouse user in /etc/security/limits.d/clickhouse.conf.
Creating config directory /etc/clickhouse-server/config.d that is used for tweaks of main server configuration.
Creating config directory /etc/clickhouse-server/users.d that is used for tweaks of users configuration.
Config file /etc/clickhouse-server/config.xml already exists, will keep it and extract path info from it.
/etc/clickhouse-server/config.xml has /var/lib/clickhouse/ as data path.
/etc/clickhouse-server/config.xml has /var/log/clickhouse-server/ as log path.
Users config file /etc/clickhouse-server/users.xml already exists, will keep it and extract users info from it.
Creating log directory /var/log/clickhouse-server/.
Creating data directory /var/lib/clickhouse/.
Creating pid directory /var/run/clickhouse-server.
chown -R clickhouse:clickhouse '/var/log/clickhouse-server/'
chown -R clickhouse:clickhouse '/var/run/clickhouse-server'
chown  clickhouse:clickhouse '/var/lib/clickhouse/'
groupadd -r clickhouse-bridge
useradd -r --shell /bin/false --home-dir /nonexistent -g clickhouse-bridge clickhouse-bridge
chown -R clickhouse-bridge:clickhouse-bridge '/usr/bin/clickhouse-odbc-bridge'
chown -R clickhouse-bridge:clickhouse-bridge '/usr/bin/clickhouse-library-bridge'
Enter password for default user: # 》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》此处是配置default的密码
Password for default user is saved in file /etc/clickhouse-server/users.d/default-password.xml.
Setting capabilities for clickhouse binary. This is optional.
chown -R clickhouse:clickhouse '/etc/clickhouse-server'

ClickHouse has been successfully installed.

Start clickhouse-server with:
sudo clickhouse start

Start clickhouse-client with:
clickhouse-client --password

Synchronizing state of clickhouse-server.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable clickhouse-server
Created symlink /etc/systemd/system/multi-user.target.wants/clickhouse-server.service → /lib/systemd/system/clickhouse-server.service.
Setting up clickhouse-client (22.8.5.29) ...
Processing triggers for systemd (245.4-4ubuntu3.13) ...

• 服务无法启动，通过debug方式启动clickhouse服务查看debug日志

sudo -u clickhouse clickhouse-server --config-file=/etc/clickhouse-server/config.xml

参考信息

• ClickHouse分布式集群部署
• ClickHouse实例部署
• ClickHouse配置文件
• 大数据ClickHouse进阶（五）：副本与分片