Clickhouse集群化（六）clickhosue-operator学习

1. Custom Resource元素

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "clickhouse-installation-test"

这是clickhouse operator自定义的资源ClickHouseInstallation

1.1. .spec.defaults

spec:
  defaults: - section represents default values for sections below.
    replicasUseFQDN: "no" - should replicas be specified by FQDN in <host></host>
    distributedDDL:  - reference to <yandex><distributed_ddl></distributed_ddl></yandex>
      profile: default
    storageManagement:
      # Specify PVC provisioner.
      # 1. StatefulSet. PVC would be provisioned by the StatefulSet
      # 2. Operator. PVC would be provisioned by the operator
      provisioner: StatefulSet

      # Specify PVC reclaim policy.
      # 1. Retain. Keep PVC from being deleted
      #    Retaining PVC will also keep backing PV from deletion. This is useful in case we need to keep data intact.
      # 2. Delete
      reclaimPolicy: Retain

1.2. .spec.configuration

表示ClickHouse配置文件的来源。无论是用户，远程服务器等配置文件。

1.2.1. .spec.configuration.zookeeper

表示标签 <yandex><zookeeper></zookeeper></yandex> 配置

    zookeeper:
      nodes:
        - host: zookeeper-0.zookeepers.zoo3ns.svc.cluster.local
          port: 2181
        - host: zookeeper-1.zookeepers.zoo3ns.svc.cluster.local
          port: 2181
        - host: zookeeper-2.zookeepers.zoo3ns.svc.cluster.local
          port: 2181
      session_timeout_ms: 30000
      operation_timeout_ms: 10000
      root: /path/to/zookeeper/node
      identity: user:password

1.2.2. .spec.configuration.profiles

    profiles:
      readonly/readonly: 1 表示<yandex><profiles></profiles></yandex> settings

1.2.3. .spec.configuration.users

    users:
      readonly/profile: readonly
      #     <users>
      #        <readonly>
      #          <profile>readonly</profile>
      #        </readonly>
      #     </users>
      test/networks/ip:
        - "127.0.0.1"
        - "::/0"
      #     <users>
      #        <test>
      #          <networks>
      #            <ip>127.0.0.1</ip>
      #            <ip>::/0</ip>
      #          </networks>
      #        </test>
      #     </users>
      test/profile: default
      test/quotas: default

1.2.4. .spec.configuration.settings

    profiles:
      readonly/readonly: "1"
      #      <profiles>
      #        <readonly>
      #          <readonly>1</readonly>
      #        </readonly>
      #      </profiles>
      default/max_memory_usage: "1000000000"

1.2.5. .spec.configuration.files

file允许通过YAML自定义ClickHouse文件。这可以用于创建复杂的自定义配置。

spec:
  configuration:
    settings:
      dictionaries_config: config.d/*.dict
    files:
      dict_one.dict: |
        <yandex>
          <dictionary>
        <name>one</name>
        <source>
            <clickhouse>
                <host>localhost</host>
                <port>9000</port>
                <user>default</user>
                <password/>
                <db>system</db>
                <table>one</table>
            </clickhouse>
        </source>
        <lifetime>60</lifetime>
        <layout><flat/></layout>
        <structure>
            <id>
                <name>dummy</name>
            </id>
            <attribute>
                <name>one</name>
                <expression>dummy</expression>
                <type>UInt8</type>
                <null_value>0</null_value>
            </attribute>
        </structure>
        </dictionary>
        </yandex>

1.2.6. .spec.configuration.clusters

集群配置

    clusters:

      - name: all-counts # 基础配置 使用layout
        templates:
          podTemplate: clickhouse-v23.8
          dataVolumeClaimTemplate: default-volume-claim
          logVolumeClaimTemplate: default-volume-claim
        schemaPolicy:
          replica: All
          shard: All
        layout:
          shardsCount: 3 分片数
          replicasCount: 2 副本数

      - name: shards-only
        templates:
          podTemplate: clickhouse-v23.8
          dataVolumeClaimTemplate: default-volume-claim
          logVolumeClaimTemplate: default-volume-claim
        layout:
          shardsCount: 3
          # replicasCount not specified, assumed = 1, by default

      - name: replicas-only
        templates:
          podTemplate: clickhouse-v23.8
          dataVolumeClaimTemplate: default-volume-claim
          logVolumeClaimTemplate: default-volume-claim
        layout:
          # shardsCount not specified, assumed = 1, by default
          replicasCount: 3

      - name: customized #自定义分片和副本 使用shards and replicas标签.
        templates:
          podTemplate: clickhouse-v23.8
          dataVolumeClaimTemplate: default-volume-claim
          logVolumeClaimTemplate: default-volume-claim
        schemaPolicy:
          replica: None
          shard: None
        layout:
          shards:
            - name: shard0
              replicasCount: 3
              weight: 1
              internalReplication: Disabled
              templates:
                podTemplate: clickhouse-v23.8
                dataVolumeClaimTemplate: default-volume-claim
                logVolumeClaimTemplate: default-volume-claim

            - name: shard1
              templates:
                podTemplate: clickhouse-v23.8
                dataVolumeClaimTemplate: default-volume-claim
                logVolumeClaimTemplate: default-volume-claim
              replicas:
                - name: replica0
                - name: replica1
                - name: replica2

            - name: shard2
              replicasCount: 3
              templates:
                podTemplate: clickhouse-v23.8
                dataVolumeClaimTemplate: default-volume-claim
                logVolumeClaimTemplate: default-volume-claim
                replicaServiceTemplate: replica-service-template
              replicas:
                - name: replica0
                  tcpPort: 9000
                  httpPort: 8123
                  interserverHTTPPort: 9009
                  templates:
                    podTemplate: clickhouse-v23.8
                    dataVolumeClaimTemplate: default-volume-claim
                    logVolumeClaimTemplate: default-volume-claim
                    replicaServiceTemplate: replica-service-template

          - name: with-secret
            # Insecure communication.
            # Opens/Closes insecure ports
            insecure: "yes"
            # Secure communication.
            # Opens/Closes secure ports
            # Translates into <secure>1</secure> ClickHouse setting for remote replicas
            secure: "yes"
            # Shared secret value to secure cluster communications
            secret:
              # Auto-generate shared secret value to secure cluster communications
              auto: "True"
              # Cluster shared secret value in plain text
              value: "plaintext secret"
              # Cluster shared secret source
              valueFrom:
                secretKeyRef:
                  name: "SecretName"
                  key: "Key"
            layout:
              shardsCount: 2

1.3. .spec.templates

1.3.1. .serviceTemplates

  templates:
    serviceTemplates:
      - name: chi-service-template
        # generateName understands different sets of macroses,
        # depending on the level of the object, for which Service is being created:
        #
        # For CHI-level Service:
        # 1. {chi} - ClickHouseInstallation name
        # 2. {chiID} - short hashed ClickHouseInstallation name (BEWARE, this is an experimental feature)
        #
        # For Cluster-level Service:
        # 1. {chi} - ClickHouseInstallation name
        # 2. {chiID} - short hashed ClickHouseInstallation name (BEWARE, this is an experimental feature)
        # 3. {cluster} - cluster name
        # 4. {clusterID} - short hashed cluster name (BEWARE, this is an experimental feature)
        # 5. {clusterIndex} - 0-based index of the cluster in the CHI (BEWARE, this is an experimental feature)
        #
        # For Shard-level Service:
        # 1. {chi} - ClickHouseInstallation name
        # 2. {chiID} - short hashed ClickHouseInstallation name (BEWARE, this is an experimental feature)
        # 3. {cluster} - cluster name
        # 4. {clusterID} - short hashed cluster name (BEWARE, this is an experimental feature)
        # 5. {clusterIndex} - 0-based index of the cluster in the CHI (BEWARE, this is an experimental feature)
        # 6. {shard} - shard name
        # 7. {shardID} - short hashed shard name (BEWARE, this is an experimental feature)
        # 8. {shardIndex} - 0-based index of the shard in the cluster (BEWARE, this is an experimental feature)
        #
        # For Replica-level Service:
        # 1. {chi} - ClickHouseInstallation name
        # 2. {chiID} - short hashed ClickHouseInstallation name (BEWARE, this is an experimental feature)
        # 3. {cluster} - cluster name
        # 4. {clusterID} - short hashed cluster name (BEWARE, this is an experimental feature)
        # 5. {clusterIndex} - 0-based index of the cluster in the CHI (BEWARE, this is an experimental feature)
        # 6. {shard} - shard name
        # 7. {shardID} - short hashed shard name (BEWARE, this is an experimental feature)
        # 8. {shardIndex} - 0-based index of the shard in the cluster (BEWARE, this is an experimental feature)
        # 9. {replica} - replica name
        # 10. {replicaID} - short hashed replica name (BEWARE, this is an experimental feature)
        # 11. {replicaIndex} - 0-based index of the replica in the shard (BEWARE, this is an experimental feature)
        generateName: "service-{chi}"
        # type ObjectMeta struct from k8s.io/meta/v1
        metadata:
          labels:
            custom.label: "custom.value"
          annotations:
            cloud.google.com/load-balancer-type: "Internal"
            service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
            service.beta.kubernetes.io/azure-load-balancer-internal: "true"
            service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
            service.beta.kubernetes.io/cce-load-balancer-internal-vpc: "true"
        # type ServiceSpec struct from k8s.io/core/v1
        spec:
          ports:
            - name: http
              port: 8123
            - name: client
              port: 9000
          type: LoadBalancer

1.3.2. podTemplates

zone和distribution一起定义节点上ClickHouse实例的分区布局。

    podTemplates:
      - name: clickhouse-v23.8
        # We may need to label nodes with clickhouse=allow label for this example to run
        # See ./label_nodes.sh for this purpose
        zone:
          key: "clickhouse"
          values:
            - "allow"
        # Shortcut version for AWS installations
        #zone:
        #  values:
        #    - "us-east-1a"

        # Possible values for podDistribution are:
        # Unspecified - empty value
        # ClickHouseAntiAffinity - AntiAffinity by ClickHouse instances.
        #   Pod pushes away other ClickHouse pods, which allows one ClickHouse instance per topologyKey-specified unit
        #   CH - (push away) - CH - (push away) - CH
        # ShardAntiAffinity - AntiAffinity by shard name.
        #   Pod pushes away other pods of the same shard (replicas of this shard),
        #   which allows one replica of a shard instance per topologyKey-specified unit.
        #   Other shards are allowed - it does not push all shards away, but CH-instances of the same shard only.
        #   Used for data loss avoidance - keeps all copies of the shard on different topologyKey-specified units.
        #   shard1,replica1 - (push away) - shard1,replica2 - (push away) - shard1,replica3
        # ReplicaAntiAffinity - AntiAffinity by replica name.
        #   Pod pushes away other pods of the same replica (shards of this replica),
        #   which allows one shard of a replica per topologyKey-specified unit.
        #   Other replicas are allowed - it does not push all replicas away, but CH-instances of the same replica only.
        #   Used to evenly distribute load from "full cluster scan" queries.
        #   shard1,replica1 - (push away) - shard2,replica1 - (push away) - shard3,replica1
        # AnotherNamespaceAntiAffinity - AntiAffinity by "another" namespace.
        #   Pod pushes away pods from another namespace, which allows same-namespace pods per topologyKey-specified unit.
        #   ns1 - (push away) - ns2 - (push away) - ns3
        # AnotherClickHouseInstallationAntiAffinity - AntiAffinity by "another" ClickHouseInstallation name.
        #   Pod pushes away pods from another ClickHouseInstallation,
        #   which allows same-ClickHouseInstallation pods per topologyKey-specified unit.
        #   CHI1 - (push away) - CHI2 - (push away) - CHI3
        # AnotherClusterAntiAffinity - AntiAffinity by "another" cluster name.
        #   Pod pushes away pods from another Cluster,
        #   which allows same-cluster pods per topologyKey-specified unit.
        #   cluster1 - (push away) - cluster2 - (push away) - cluster3
        # MaxNumberPerNode - AntiAffinity by cycle index.
        #   Pod pushes away pods from the same cycle,
        #   which allows to specify maximum number of ClickHouse instances per topologyKey-specified unit.
        #   Used to setup circular replication.
        # NamespaceAffinity - Affinity by namespace.
        #   Pod attracts pods from the same namespace,
        #   which allows pods from same namespace per topologyKey-specified unit.
        #   ns1 + (attracts) + ns1
        # ClickHouseInstallationAffinity - Affinity by ClickHouseInstallation name.
        #   Pod attracts pods from the same ClickHouseInstallation,
        #   which allows pods from the same CHI per topologyKey-specified unit.
        #   CHI1 + (attracts) + CHI1
        # ClusterAffinity - Affinity by cluster name.
        #   Pod attracts pods from the same cluster,
        #   which allows pods from the same Cluster per topologyKey-specified unit.
        #   cluster1 + (attracts) + cluster1
        # ShardAffinity - Affinity by shard name.
        #   Pod attracts pods from the same shard,
        #   which allows pods from the same Shard per topologyKey-specified unit.
        #   shard1 + (attracts) + shard1
        # ReplicaAffinity - Affinity by replica name.
        #   Pod attracts pods from the same replica,
        #   which allows pods from the same Replica per topologyKey-specified unit.
        #   replica1 + (attracts) + replica1
        # PreviousTailAffinity - Affinity to overlap cycles. Used to make cycle pod distribution
        #   cycle head + (attracts to) + previous cycle tail
        podDistribution:
          - type: ShardAntiAffinity
          - type: MaxNumberPerNode
            number: 2
            # Apply podDistribution on per-host basis
            topologyKey: "kubernetes.io/hostname"
            # Apply podDistribution on per-zone basis
            #topologyKey: "kubernetes.io/zone"

        # type ObjectMeta struct {} from k8s.io/meta/v1
        metadata:
          labels:
            a: "b"
        # type PodSpec struct {} from k8s.io/core/v1
        spec:
          containers:
            - name: clickhouse
              image: clickhouse/clickhouse-server:23.8
              volumeMounts:
                - name: default-volume-claim
                  mountPath: /var/lib/clickhouse
              resources:
                requests:
                  memory: "64Mi"
                  cpu: "100m"
                limits:
                  memory: "64Mi"
                  cpu: "100m"
            - name: clickhouse-log
              image: clickhouse/clickhouse-server:23.8
              command:
                - "/bin/sh"
                - "-c"
                - "--"
              args:
                - "while true; do sleep 30; done;"

      # pod template for ClickHouse v23.8
      - name: clickhouse-v23.8
        # type ObjectMeta struct {} from k8s.io/meta/v1
        metadata:
          labels:
            a: "b"
        # type PodSpec struct {} from k8s.io/core/v1
        spec:
          containers:
            - name: clickhouse
              image: clickhouse/clickhouse-server:23.8
              volumeMounts:
                - name: default-volume-claim
                  mountPath: /var/lib/clickhouse
              resources:
                requests:
                  memory: "64Mi"
                  cpu: "100m"
                limits:
                  memory: "64Mi"
                  cpu: "100m"
            - name: clickhouse-log
              image: clickhouse/clickhouse-server:23.8
              command:
                - "/bin/sh"
                - "-c"
                - "--"
              args:
                - "while true; do sleep 30; done;"

2. 建表

2.1. Manifest

创建一个双副本双飞分片的clickhouse集群

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"

metadata:
  name: "repl-05"

spec:
  defaults:
    templates: 
      dataVolumeClaimTemplate: default
      podTemplate: clickhouse:19.6
 
  configuration:
    zookeeper:
      nodes:
      - host: zookeeper.zoo1ns
    clusters:
      - name: replicated
        layout:
          shardsCount: 2
          replicasCount: 2

  templates:
    volumeClaimTemplates:
      - name: default
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 500Mi
    podTemplates:
      - name: clickhouse:19.6
        spec:
          containers:
            - name: clickhouse-pod
              image: clickhouse/clickhouse-server:23.8

2.2. Replicated table setup

2.2.1. 宏

Operator provides set of macros, which are:

1. {installation} -- ClickHouse Installation name
2. {cluster} -- primary cluster name
3. {replica} -- replica name in the cluster, maps to pod service name
4. {shard} -- shard id

ClickHouse also supports internal macros {database} and {table} that maps to current database and table respectively.

2.2.2. Create replicated table

Now we can create replicated table, using specified macros

CREATE TABLE events_local on cluster '{cluster}' (
    event_date  Date,
    event_type  Int32,
    article_id  Int32,
    title       String
) engine=ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/{database}/{table}', '{replica}')
PARTITION BY toYYYYMM(event_date)
ORDER BY (event_type, article_id);
第一个参数是分片的 zk_path 一般按照：/clickhouse/table/{shard}/{table_name} 的格式
第二个参数是副本名称，相同的分片副本名称不能相同。

CREATE TABLE events on cluster '{cluster}' AS events_local
ENGINE = Distributed('{cluster}', default, events_local, rand());
'{cluster}': 代表集群的名称，用于指定数据将如何在集群中分布。

default: 指定数据库的名称。在这个例子中，使用 default 数据库，这意味着表会在 default 数据库中创建。

events_local: 指定实际存储数据的本地表的名称。在查询 events 表时，数据将从集群中的 events_local 表中检索。events_local 表是在每个节点上本地存储数据的实际表。

rand(): 指定分布的方式。在这里，rand() 函数表示使用随机方式将数据分配到不同的分片上。这有助于实现数据的均匀分布。

We can generate some data:

INSERT INTO events SELECT today(), rand()%3, number, 'my title' FROM numbers(100);

And check how these data are distributed over the cluster

SELECT count() FROM events;
SELECT count() FROM events_local;

3. 用户

3.1. The 'default' user

“默认”用户用于从运行的pod连接到ClickHouse实例，也用于分布式查询。它使用空密码部署，这是ClickHouse开箱即用安装的长期默认密码。

为了确保它的安全，operator应用网络安全规则，限制连接到运行ClickHouse集群的pod，而不是其他。

下面的users.xml是由operator为具有两个节点的集群设置的。在此配置中，“默认”用户不能从集群外部连接。

3.2. 修改operate配置

clickhouse:
  access:
    secret:
      # Empty `namespace` means that k8s secret would be looked
      # in the same namespace where the operator's pod is running.
      namespace: ""
      name: "clickhouse-operator"

The following example shows a secret:

apiVersion: v1
kind: Secret
metadata:
  name: clickhouse-operator
type: Opaque
stringData:
  username: clickhouse_operator
  password: chpassword

3.3. Using hashed passwords

ClickHouseInstallation中的用户密码可以明文指定为sha256和双sha1哈希值。

当以明文形式指定密码时，操作员在部署到ClickHouse时将其散列，但在ClickHouse安装中仍以不安全的明文格式保留。

Altinity建议显式提供哈希值如下:

spec:
  useTemplates:
    - name: clickhouse-version
  configuration:
    users:
      user1/password: pwduser1  # This will be hashed in ClickHouse config files, but this NOT RECOMMENDED
      user2/password_sha256_hex: 716b36073a90c6fe1d445ac1af85f4777c5b7a155cea359961826a030513e448
      user3/password_double_sha1_hex: cbe205a7351dd15397bf423957559512bd4be395

3.4. Using secrets

spec:
  configuration:
    users:
      user1/password:
        valueFrom:
          secretKeyRef:
            name: clickhouse_secret
            key: pwduser1
      user2/password_sha256_hex:
        valueFrom:
          secretKeyRef:
            name: clickhouse_secret
            key: pwduser2          
      user3/password_double_sha1_hex:
        valueFrom:
          secretKeyRef:
            name: clickhouse_secret
            key: pwduser3    
 

apiVersion: v1
kind: Secret
metadata:
  name: clickhouse-secret
type: Opaque
stringData:
  pwduser1: pwduser1
  pwduser2: e106728c3541ec3694822a29d4b8b5f1f473508adc148fcb58a60c42bcf3234c
  pwduser3: cbe205a7351dd15397bf423957559512bd4be395

3.5. Securing the 'default' user

虽然“默认”用户受到网络规则的保护，但信息安全团队通常不允许无密码操作。可以像更改其他用户一样更改“默认”用户的密码。然而，ClickHouse也使用“默认”用户来运行分布式查询。如果密码更改，分布式查询可能会停止工作。

为了在不暴露密码的情况下保持分布式查询的运行，配置ClickHouse为集群间通信使用秘密令牌，而不是“默认”用户凭据。

3.5.1. 'auto' token

下面的示例展示了如何让ClickHouse自动生成秘密令牌。这是最简单和推荐的方法。

spec:
  configuration:
    users:
      default/password_sha256_hex: 716b36073a90c6fe1d445ac1af85f4777c5b7a155cea359961826a030513e448
    clusters:
      - name: default
        secret:
          auto: "true"

3.5.2. Custom token

The following example shows how to define a token.

spec:
  configuration:
    users:
      default/password_sha256_hex: 716b36073a90c6fe1d445ac1af85f4777c5b7a155cea359961826a030513e448
    clusters:
      - name: "default"
        secret:
          value: "my_secret"

3.5.3. Custom token from Kubernetes secret

The following example shows how to define a token within a secret.

spec:
  configuration:
    users:
      default/password_sha256_hex: 716b36073a90c6fe1d445ac1af85f4777c5b7a155cea359961826a030513e448
    clusters:
      - name: "default"
        secret:
          valueFrom:
            secretKeyRef:
              name: "secure-inter-cluster-communications"
              key: "secret"

4. zookeeper

This document describes how to setup ZooKeeper in k8s environment.

Zookeeper installation is available in two options:

1. Quick start - just run it quickly and ask no questions
2. Advanced setup - setup internal details, such as storage class, replicas number, etc

During ZooKeeper installation the following items are created/configured:

1. [OPTIONAL] Create separate namespace to run Zookeeper in
2. Create k8s resources (optionally, within namespace):

• Service - used to provide central access point to Zookeeper
• Headless Service - used to provide DNS namings
• Disruption Balance - used to specify max number of offline pods
• [OPTIONAL] Storage Class - used to specify storage class to be used by Zookeeper for data storage
• Stateful Set - used to manage and scale sets of pods

4.1. Quick start

Quick start is represented in two flavors:

1. With persistent volume - good for AWS. File are located in deploy/zookeeper/quick-start-persistent-volume
2. With local emptyDir storage - good for standalone local run, however has to true persistence.
   Files are located in deploy/zookeeper/quick-start-volume-emptyDir

Each quick start flavor provides the following installation options:

1. 1-node Zookeeper cluster (zookeeper-1- files). No failover provided.
2. 3-node Zookeeper cluster (zookeeper-3- files). Failover provided.

In case you'd like to test with AWS or any other cloud provider, we recommend to go with deploy/zookeeper/quick-start-persistent-volume persistent storage. In case of local test, you'd may prefer to go with deploy/zookeeper/quick-start-volume-emptyDir emptyDir.

4.1.1. Script-based Installation

In this example we'll go with simple 1-node Zookeeper cluster on AWS and pick deploy/zookeeper/quick-start-persistent-volume. Both create and delete shell scripts are available for simplification.

4.1.2. Manual Installation

In case you'd like to deploy Zookeeper manually, the following steps should be performed:

4.1.3. Namespace

Create namespace

kubectl create namespace zoo1ns

4.1.4. Zookeeper

Deploy Zookeeper into this namespace

kubectl apply -f zookeeper-1-node.yaml -n zoo1ns

Now Zookeeper should be up and running. Let's explore Zookeeper cluster.

IMPORTANT quick-start zookeeper installation are for test purposes mainly.
For fine-tuned Zookeeper setup please refer to advanced setup options.

4.2. Advanced setup

Advanced files are are located in deploy/zookeeper/advanced folder. All resources are separated into different files so it is easy to modify them and setup required options.

Advanced setup is available in two options:

1. With persistent volume
2. With emptyDir volume

Each of these options have both create and delete scripts provided

1. Persistent volume create and delete scripts
2. EmptyDir volume create and delete scripts

Step-by-step explanations:

4.2.1. Namespace

Create namespace in which all the rest resources would be created

kubectl create namespace zoons

4.2.2. Zookeeper Service

Create service. This service provides DNS name for client access to all Zookeeper nodes.

kubectl apply -f 01-service-client-access.yaml -n zoons

Should have as a result

service/zookeeper created

4.2.3. Zookeeper Headless Service

Create headless service. This service provides DNS names for all Zookeeper nodes

kubectl apply -f 02-headless-service.yaml -n zoons

Should have as a result

service/zookeeper-nodes created

4.2.4. Disruption Budget

Create budget. Disruption Budget instructs k8s on how many offline Zookeeper nodes can be at any time

kubectl apply -f 03-pod-disruption-budget.yaml -n zoons

Should have as a result

poddisruptionbudget.policy/zookeeper-pod-distribution-budget created

4.2.5. Storage Class

This part is not that straightforward and may require communication with k8s instance administrator.

First of all, we need to decide, whether Zookeeper would use Persistent Volume as a storage or just stick to more simple Volume (In doc emptyDir type is used)

In case we'd prefer to stick with simpler solution and go with Volume of type emptyDir, we need to go with emptyDir StatefulSet config 05-stateful-set-volume-emptyDir.yaml as described in next Stateful Set unit. Just move to it.

In case we'd prefer to go with Persistent Volume storage, we need to go with Persistent Volume StatefulSet config 05-stateful-set-persistent-volume.yaml

Shortly, Storage Class is used to bind together Persistent Volumes, which are created either by k8s admin manually or automatically by Provisioner. In any case, Persistent Volumes are provided externally to an application to be deployed into k8s. So, this application has to know Storage Class Name to ask for from the k8s in application's claim for new persistent volume - Persistent Volume Claim. This Storage Class Name should be asked from k8s admin and written as application's Persistent Volume Claim .spec.volumeClaimTemplates.storageClassName parameter in StatefulSet configuration. StatefulSet manifest with emptyDir 05-stateful-set-volume-emptyDir.yaml and/or StatefulSet manifest with Persistent Volume 05-stateful-set-persistent-volume.yaml.

4.2.6. Stateful Set

Edit StatefulSet manifest with emptyDir 05-stateful-set-volume-emptyDir.yaml and/or StatefulSet manifest with Persistent Volume 05-stateful-set-persistent-volume.yaml according to your Storage Preferences.

In case we'd go with Volume of type emptyDir, ensure .spec.template.spec.containers.volumes is in place and look like the following:

volumes:
      - name: datadir-volume
        emptyDir:
          medium: "" #accepted values:  empty str (means node's default medium) or Memory
          sizeLimit: 1Gi

and ensure .spec.volumeClaimTemplates is commented.

In case we'd go with Persistent Volume storage, ensure .spec.template.spec.containers.volumes is commented and ensure .spec.volumeClaimTemplates is uncommented.

volumeClaimTemplates:
  - metadata:
      name: datadir-volume
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
## storageClassName has to be coordinated with k8s admin and has to be created as a `kind: StorageClass` resource
      storageClassName: storageclass-zookeeper

and ensure storageClassName (storageclass-zookeeper in this example) is specified correctly, as described in Storage Class section

As .yaml file is ready, just apply it with kubectl

kubectl apply -f 05-stateful-set.yaml -n zoons

Should have as a result

statefulset.apps/zookeeper-node created

Now we can take a look into Zookeeper cluster deployed in k8s:

4.3. Explore Zookeeper cluster

4.3.1. DNS names

We are expecting to have ZooKeeper cluster of 3 pods inside zoons namespace, named as:

zookeeper-0
zookeeper-1
zookeeper-2

Those pods are expected to have short DNS names as:

zookeeper-0.zookeepers.zoons
zookeeper-1.zookeepers.zoons
zookeeper-2.zookeepers.zoons

where zookeepers is name of Zookeeper headless service and zoons is name of Zookeeper namespace.

and full DNS names (FQDN) as:

zookeeper-0.zookeepers.zoons.svc.cluster.local
zookeeper-1.zookeepers.zoons.svc.cluster.local
zookeeper-2.zookeepers.zoons.svc.cluster.local

4.3.2. Resources

List pods in Zookeeper's namespace

kubectl get pod -n zoons

Expected output is like the following

NAME             READY   STATUS    RESTARTS   AGE
zookeeper-0      1/1     Running   0          9m2s
zookeeper-1      1/1     Running   0          9m2s
zookeeper-2      1/1     Running   0          9m2s

List services

kubectl get service -n zoons

Expected output is like the following

NAME                   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
zookeeper              ClusterIP   10.108.36.44   <none>        2181/TCP                     168m
zookeepers             ClusterIP   None           <none>        2888/TCP,3888/TCP            31m

List statefulsets

kubectl get statefulset -n zoons

Expected output is like the following

NAME            READY   AGE
zookeepers      3/3     10m

In case all looks fine Zookeeper cluster is up and running