AWS EKS上使用OpenSearch部署EFK日志分析收集系统

本文链接：https://blog.csdn.net/weixin_41335923/article/details/121692400

本文档详细介绍了如何在AWS Elastic Kubernetes Service (EKS)上配置OpenSearch来搭建EFK日志分析收集系统。首先，创建OIDC身份提供商和IAM角色策略，接着配置访问权限，然后进行部署步骤，最后在OpenSearch仪表盘中查看并管理数据索引。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

一、为`Fluent Bit`配置`IRSA`

创建OIDC身份提供商

eksctl utils associate-iam-oidc-provider \
    --cluster {clustername} \
    --approve

创建IAM角色和策略

mkdir ~/environment/logging/

export ES_DOMAIN_NAME="eks-logging"

cat <<EoF > ~/environment/logging/fluent-bit-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "es:ESHttp*"
            ],
            "Resource": "arn:aws:es:${AWS_REGION}:${ACCOUNT_ID}:domain/${ES_DOMAIN_NAME}",
            "Effect": "Allow"
        }
    ]
}
EoF

aws iam create-policy   \
  --policy-name fluent-bit-policy \
  --policy-document file://~/environment/logging/fluent-bit-policy.json

创建角色

kubectl create namespace logging

eksctl create iamserviceaccount \
    --name fluent-bit \
    --namespace logging \
    --cluster {clustername} \
    --attach-policy-arn "arn:aws:iam::${ACCOUNT_ID}:policy/fluent-bit-policy" \
    --approve \
    --override-existing-serviceaccounts

二、配置`Amazon OpenSearch`访问

${ES_DOMAIN_NAME}为OpenSearch的名称

# We need to retrieve the Fluent Bit Role ARN
export FLUENTBIT_ROLE=$(eksctl get iamserviceaccount --cluster {clustername} --namespace logging -o json | jq '.[].status.roleARN' -r)

# Get the Amazon OpenSearch Endpoint
export ES_ENDPOINT=$(aws opensearch describe-domain --domain-name ${ES_DOMAIN_NAME} --output text --query "DomainStatus.Endpoint")

# Update the Elasticsearch internal database
curl -sS -u "${ES_DOMAIN_USER}:${ES_DOMAIN_PASSWORD}" \
    -X PATCH \
    https://${ES_ENDPOINT}/_opendistro/_security/api/rolesmapping/all_access?pretty \
    -H 'Content-Type: application/json' \
    -d'
[
  {
    "op": "add", "path": "/backend_roles", "value": ["'${FLUENTBIT_ROLE}'"]
  }
]
'

三、部署`Fluent Bit`

cd ~/environment/logging

# get the Amazon OpenSearch Endpoint
export ES_ENDPOINT=$(aws es describe-elasticsearch-domain --domain-name ${ES_DOMAIN_NAME} --output text --query "DomainStatus.Endpoint")

curl -Ss https://www.eksworkshop.com/intermediate/230_logging/deploy.files/fluentbit.yaml \
    | envsubst > ~/environment/logging/fluentbit.yaml
    
# deploy
kubectl apply -f ~/environment/logging/fluentbit.yaml

# watch
kubectl --namespace=logging get pods