使用eck在k8s中部署es集群

1. ECK简介

Elastic Cloud on Kubernetes (ECK) 是一个官方提供的用于在 Kubernetes 集群中简化部署、管理和操作 Elastic Stack（包括 Elasticsearch 和 Kibana）的扩展。

ECK 是一个 Kubernetes Operator，它管理和自动化 Elastic Stack 的生命周期。通过使用 ECK，可以在 Kubernetes 环境中快速实现以下功能：

1. 部署和管理 Elasticsearch 和 Kibana 实例，包括创建、删除、扩展和升级。
2. 配置和调整 Elastic Stack 组件以满足特定需求。
3. 自动处理故障检测、恢复和备份。
4. 保护 Elasticsearch 集群，通过安全配置、证书管理和安全通信来确保数据安全。
5. 监控 Elastic Stack 的性能和资源使用，从而优化集群性能。

官方文档: https://www.elastic.co/guide/en/cloud-on-k8s/current/index.html

2. 版本说明

ECK版本: 2.8.0

适用于Kubernetes版本: 1.24~1.27 (本文使用1.27.2演示)

适用于Elasticsearch、Kibana版本: 6.8+、7.1+、8+ (本文演示部署8.8.0版本的es与kibana)

3. 部署ECK

3.1 创建ECK所需CRD

kubectl create -f https://download.elastic.co/downloads/eck/2.8.0/crds.yaml

输出

customresourcedefinition.apiextensions.k8s.io/agents.agent.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/apmservers.apm.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/beats.beat.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/elasticmapsservers.maps.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/elasticsearchautoscalers.autoscaling.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/elasticsearches.elasticsearch.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/enterprisesearches.enterprisesearch.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/kibanas.kibana.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/logstashes.logstash.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/stackconfigpolicies.stackconfigpolicy.k8s.elastic.co created

3.2 创建ECK opeartor

kubectl apply -f https://download.elastic.co/downloads/eck/2.8.0/operator.yaml

输出

namespace/elastic-system created
serviceaccount/elastic-operator created
secret/elastic-webhook-server-cert created
configmap/elastic-operator created
clusterrole.rbac.authorization.k8s.io/elastic-operator created
clusterrole.rbac.authorization.k8s.io/elastic-operator-view created
clusterrole.rbac.authorization.k8s.io/elastic-operator-edit created
clusterrolebinding.rbac.authorization.k8s.io/elastic-operator created
service/elastic-webhook-server created
statefulset.apps/elastic-operator created
validatingwebhookconfiguration.admissionregistration.k8s.io/elastic-webhook.k8s.elastic.co created

ECK operator在 elastic-system 命名空间中运行。生产环境中的工作负载选择专用命名空间，而不是使用 elastic-system 或 default 命名空间。

查看ECK operator

kubectl get pods -n elastic-system

输出

NAME                 READY   STATUS    RESTARTS   AGE
elastic-operator-0   1/1     Running   0          13m

4. 通过eck部署es集群

Kubernetes集群至少要有一个2GB可用内存的节点，否则Pod 将停留在 Pending 状态。

4.1 创建es集群es.yaml文件

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 8.8.0
  nodeSets:
  - name: default
    count: 3
    config:
      node.store.allow_mmap: false
    podTemplate:
      spec:
        volumes:
        - name: elasticsearch-data
          emptyDir: {
   }

kubectl apply -f es.yaml

4.2 查看es集群信息

查看es集群状态

kubectl get elasticsearch

输出

NAME         HEALTH    NODES   VERSION   PHASE             AGE
quickstart   unknown           8.8.0     ApplyingChanges   2m4s

此时看到的状态为unknown，可能是由于正在创建中

正常等待几分钟后应该显示为

NAME         HEALTH   NODES   VERSION   PHASE   AGE
quickstart   green    3       8.8.0     Ready   18m

查看es集群的pod

kubectl get pods --selector='elasticsearch.k8s.elastic.co/cluster-name=quickstart'

输出

NAME                      READY   STATUS    RESTARTS   AGE
quickstart-es-default-0   1/1     Running   0          19m
quickstart-es-default-1   1/1     Running   0          19m
quickstart-es-default-2   1/1     Running   0          19m

4.3 访问es集群

默认情况下为自动创建service

kubectl get service quickstart-es-http

输出

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
quickstart-es-http   ClusterIP   10.105.188.20   <none>        9200/TCP   33m

# 获取密码
PASSWORD=$(kubectl get secret quickstart-es-elastic-user -o go-template='{
    {.data.elastic | base64decode}}')
curl -u "elastic:$PASSWORD" -k "https://quickstart-es-http:9200"

输出

{
  "name" : "quickstart-es-default-0",
  "cluster_name" : "quickstart",
  "cluster_uuid" : "hPaILve1QCe2ig25RPErcg",
  "version" : {
    "number" : "8.8.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "c01029875a091076ed42cdb3a41c10b1a9a5a20f",
    "build_date" : "2023-05-23T17:16:07.179039820Z",
    "build_snapshot" : false,
    "lucene_version" : "9.6.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

5. 部署kibana

1. 创建文件kibana.yaml

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana