K8S ELK日志分析

最新推荐文章于 2024-06-28 08:14:06 发布

wuxingge

最新推荐文章于 2024-06-28 08:14:06 发布

阅读量1k

点赞数

分类专栏： kubernetes

本文链接：https://blog.csdn.net/wuxingge/article/details/106657651

版权

kubernetes 专栏收录该内容

67 篇文章 9 订阅

订阅专栏

日志，对于任何系统来说都是及其重要的组成部分，在计算机系统中比较复杂，日志有不同的来源，如操作系统，应用服务，业务逻辑等，它们都在不停产生各种各样的日志
K8S系统里的业务应用是高度 "动态化"的，随着容器编排的进行，业务容器在不断的被创建、被销毁、被迁移、被扩缩容…
需要建立一套集中式的方法，把不同来源的数据集中整合到一个地方
- 收集—能够采集多种来源的日志数据（流式日志收集器）
- 传输—能够稳定的把日志数据传输到中央系统（消息队列）
- 存储—可以将日志以结构化数据的形式存储起来（搜索引擎）
- 分析—支持方便的分析、检索方法，最好有GUI管理系统（前端）
- 告警—能够提供错误报告，监控机制（监控工具）
开源解决方案—ELK Stack
- E – ElasticSearch
- L – LogStash
- K – Kibana

https://blog.stanley.wang/2019/01/18/%E5%AE%9E%E9%AA%8C%E6%96%87%E6%A1%A34%EF%BC%9Akubernetes%E9%9B%86%E7%BE%A4%E7%9A%84%E7%9B%91%E6%8E%A7%E5%92%8C%E6%97%A5%E5%BF%97%E5%88%86%E6%9E%90/

在这里插入图片描述

制作tomcat容器的底包镜像

wget https://mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-8/v8.5.56/bin/apache-tomcat-8.5.56.tar.gz
mkdir /data/dockerfile/tomcat
tar xf apache-tomcat-8.5.56.tar.gz -C /data/dockerfile/tomcat/

配置日志
vim conf/logging.properties

handlers = 1catalina.org.apache.juli.AsyncFileHandler, 2localhost.org.apache.juli.AsyncFileHandler, java.util.logg
ing.ConsoleHandler

...

1catalina.org.apache.juli.AsyncFileHandler.level = INFO
1catalina.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
1catalina.org.apache.juli.AsyncFileHandler.prefix = catalina.
1catalina.org.apache.juli.AsyncFileHandler.encoding = UTF-8

2localhost.org.apache.juli.AsyncFileHandler.level = INFO
2localhost.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
2localhost.org.apache.juli.AsyncFileHandler.prefix = localhost.
2localhost.org.apache.juli.AsyncFileHandler.encoding = UTF-8

#3manager.org.apache.juli.AsyncFileHandler.level = FINE
#3manager.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
#3manager.org.apache.juli.AsyncFileHandler.prefix = manager.
#3manager.org.apache.juli.AsyncFileHandler.encoding = UTF-8
#
#4host-manager.org.apache.juli.AsyncFileHandler.level = FINE
#4host-manager.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
#4host-manager.org.apache.juli.AsyncFileHandler.prefix = host-manager.
#4host-manager.org.apache.juli.AsyncFileHandler.encoding = UTF-8

java.util.logging.ConsoleHandler.level = INFO
java.util.logging.ConsoleHandler.formatter = org.apache.juli.OneLineFormatter
java.util.logging.ConsoleHandler.encoding = UTF-8
...

Dockerfile
/data/dockerfile/tomcat/目录下：
cat Dockerfile

From harbor.od.com/public/jre8:8u112
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\ 
    echo 'Asia/Shanghai' >/etc/timezone
ENV CATALINA_HOME /opt/tomcat
ENV LANG zh_CN.UTF-8
ADD apache-tomcat-8.5.56/ /opt/tomcat
ADD config.yml /opt/prom/config.yml
ADD jmx_javaagent-0.3.1.jar /opt/prom/jmx_javaagent-0.3.1.jar
WORKDIR /opt/tomcat
ADD entrypoint.sh /entrypoint.sh
CMD ["/entrypoint.sh"]

cat config.yml

—
rules:
  - pattern: '.*'

下载 jmx_javaagent-0.3.1.jar

wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar -O jmx_javaagent-0.3.1.jar

cat entrypoint.sh

#!/bin/bash
M_OPTS="-Duser.timezone=Asia/Shanghai -javaagent:/opt/prom/jmx_javaagent-0.3.1.jar=$(hostname -i):${M_PORT:-"12346"}:/opt/prom/config.yml"
C_OPTS=${C_OPTS}
MIN_HEAP=${MIN_HEAP:-"128m"}
MAX_HEAP=${MAX_HEAP:-"128m"}
JAVA_OPTS=${JAVA_OPTS:-"-Xmn384m -Xss256k -Duser.timezone=GMT+08  -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSClassUnloadingEnabled -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+PrintClassHistogram  -Dfile.encoding=UTF8 -Dsun.jnu.encoding=UTF8"}
CATALINA_OPTS="${CATALINA_OPTS}"
JAVA_OPTS="${M_OPTS} ${C_OPTS} -Xms${MIN_HEAP} -Xmx${MAX_HEAP} ${JAVA_OPTS}"
sed -i -e "1a\JAVA_OPTS=\"$JAVA_OPTS\"" -e "1a\CATALINA_OPTS=\"$CATALINA_OPTS\"" /opt/tomcat/bin/catalina.sh

cd /opt/tomcat && /opt/tomcat/bin/catalina.sh run 2>&1 >> /opt/tomcat/logs/stdout.log

build镜像

docker build . -t harbor.od.com/base/tomcat:v8.5.56
docker push harbor.od.com/base/tomcat:v8.5.56

jenkins构建

Add Parameter -> String Parameter

Name : app_name
Default Value :
Description : project name. e.g: dubbo-demo-web

Add Parameter -> String Parameter

Name : image_name
Default Value :
Description : project docker image name. e.g: app/dubbo-demo-web

Add Parameter -> String Parameter

Name : git_repo
Default Value :
Description : project git repository. e.g: git@gitee.com:stanleywang/dubbo-demo-web.git

Add Parameter -> String Parameter

Name : git_ver
Default Value : tomcat
Description : git commit id of the project.

Add Parameter -> String Parameter

Name : add_tag
Default Value :
Description : project docker image tag, date_timestamp recommended. e.g: 190117_1920

Add Parameter -> String Parameter

Name : mvn_dir
Default Value : ./
Description : project maven directory. e.g: ./

Add Parameter -> String Parameter

Name : target_dir
Default Value : ./dubbo-client/target
Description : the relative path of target file such as .jar or .war package. e.g: ./dubbo-client/target

Add Parameter -> String Parameter

Name : mvn_cmd
Default Value : mvn clean package -Dmaven.test.skip=true
Description : maven command. e.g: mvn clean package -e -q -Dmaven.test.skip=true

Add Parameter -> Choice Parameter

Name : base_image
Default Value :
base/tomcat:v7.0.94
base/tomcat:v8.5.40
base/tomcat:v9.0.17
Description : project base image list in harbor.od.com.

Add Parameter -> Choice Parameter

Name : maven
Default Value :
3.6.0-8u181
3.2.5-6u025
2.2.1-6u025
Description : different maven edition.

Add Parameter -> String Parameter

Name : root_url
Default Value : ROOT
Description : webapp dir.

pipeline {
  agent any 
    stages {
    stage('pull') { //get project code from repo 
      steps {
        sh "git clone ${params.git_repo} ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.app_name}/${env.BUILD_NUMBER} && git checkout ${params.git_ver}"
        }
    }
    stage('build') { //exec mvn cmd
      steps {
        sh "cd ${params.app_name}/${env.BUILD_NUMBER}  && /var/jenkins_home/maven-${params.maven}/bin/${params.mvn_cmd}"
      }
    }
    stage('unzip') { //unzip  target/*.war -c target/project_dir
      steps {
        sh "cd ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.target_dir} && mkdir project_dir && unzip *.war -d ./project_dir"
      }
    }
    stage('image') { //build image and push to registry
      steps {
        writeFile file: "${params.app_name}/${env.BUILD_NUMBER}/Dockerfile", text: """FROM harbor.od.com/${params.base_image}
ADD ${params.target_dir}/project_dir /opt/tomcat/webapps/${params.root_url}"""
        sh "cd  ${params.app_name}/${env.BUILD_NUMBER} && docker build -t harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag} . && docker push harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag}"
      }
    }
  }
}

在这里插入图片描述

附：简易版

=================================

cat Dockerfile

From harbor.od.com/public/jre8:8u112
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\ 
    echo 'Asia/Shanghai' >/etc/timezone
ENV CATALINA_HOME /opt/tomcat
ENV LANG zh_CN.UTF-8
ADD apache-tomcat-8.5.56/ /opt/tomcat
WORKDIR /opt/tomcat
ADD entrypoint.sh /entrypoint.sh
CMD ["/entrypoint.sh"]

cat entrypoint.sh

#!/bin/bash
cd /opt/tomcat && /opt/tomcat/bin/catalina.sh run 2>&1 >> /opt/tomcat/logs/stdout.log

tomcat镜像

harbor.od.com/app/tomcat:v8.5.56

pipeline {
  agent any 
    stages {
    stage('image') { //build image and push to registry
      steps {
        writeFile file: "${params.app_name}/${env.BUILD_NUMBER}/Dockerfile", text: """FROM harbor.od.com/${params.base_image}"""
        sh "cd  ${params.app_name}/${env.BUILD_NUMBER} && docker build -t harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag} . && docker push harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag}"
      }
    }
  }
}

/data/k8s-yaml/dubbo-demo-consumer目录下：

cat dp.yaml

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-demo-consumer
  namespace: test
  labels: 
    name: dubbo-demo-consumer
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-demo-consumer
  template:
    metadata:
      labels: 
        app: dubbo-demo-consumer
        name: dubbo-demo-consumer
    spec:
      containers:
      - name: dubbo-demo-consumer
        image: harbor.od.com/app/dubbo-demo-web:tomcat_200610_1556
        ports:
        - containerPort: 8080
          protocol: TCP
        - containerPort: 20880
          protocol: TCP
        env:
        - name: JAR_BALL
          value: dubbo-client.jar
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

cat svc.yaml

kind: Service
apiVersion: v1
metadata: 
  name: dubbo-demo-consumer
  namespace: test
spec:
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  selector: 
    app: dubbo-demo-consumer
  clusterIP: None
  type: ClusterIP
  sessionAffinity: None

cat ingress.yaml

kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: dubbo-demo-consumer
  namespace: test
spec:
  rules:
  - host: demo.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: dubbo-demo-consumer
          servicePort: 8080

kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-consumer/dp.yaml
kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-consumer/svc.yaml
kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-consumer/ingress.yaml

==================================

elasticsearch

https://www.elastic.co/

https://github.com/elastic/elasticsearch

https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.15.tar.gz

HDSS7-12.host.com上：

安装

tar xf elasticsearch-6.8.3.tar.gz -C /opt/
ln -s /opt/elasticsearch-6.8.3 /opt/elasticsearch

配置

mkdir -p /data/elasticsearch/{data,logs}

vim config/elasticsearch.yml

# 修改以下内容
cluster.name: es.od.com
node.name: hdss7-12.host.com
path.data: /data/elasticsearch/data
path.logs: /data/elasticsearch/logs
bootstrap.memory_lock: true
network.host: 10.4.7.12
http.port: 9200

vi config/jvm.options
修改以下

-Xms512m
-Xmx512m

创建普通用户

useradd -s /bin/bash -M es
chown -R es.es /opt/elasticsearch-6.8.3/
chown -R es.es /data/elasticsearch/

文件描述符

cat /etc/security/limits.d/es.conf

es hard nofile 65536
es soft fsize unlimited
es hard memlock unlimited
es soft memlock unlimited

调整内核参数

sysctl -w vm.max_map_count=262144
echo "vm.max_map_count=262144" >> /etc/sysctl.conf
sysctl -p

启动

su -c "/opt/elasticsearch/bin/elasticsearch -d" es

检查

netstat -lntup |grep 9200

调整ES日志模板

curl -H "Content-Type:application/json" -XPUT http://10.4.7.12:9200/_template/k8s -d '{
  "template" : "k8s*",
  "index_patterns": ["k8s*"],  
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 0
  }
}'

kafka

http://kafka.apache.org/

https://github.com/apache/kafka

http://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.1.1/kafka_2.12-2.1.1.tgz

HDSS7-11.host.com上：

安装

tar xf kafka_2.12-2.2.0.tgz -C /opt/
ln -s /opt/kafka_2.12-2.2.0 /opt/kafka

配置

vim config/server.properties
修改以下配置

log.dirs=/data/kafka/logs
zookeeper.connect=localhost:2181
log.flush.interval.messages=10000
log.flush.interval.ms=1000
delete.topic.enable=true
host.name=hdss7-11.host.com

mkdir -p /data/kafka/logs

启动

bin/kafka-server-start.sh -daemon config/server.properties

检查

netstat -lntup |grep 9092

kafka-manager

https://github.com/yahoo/kafka-manager

https://github.com/yahoo/kafka-manager/archive/2.0.0.2.tar.gz

HDSS7-200.host.com上：

mkdir /data/dockerfile/kafka-manager
cd /data/dockerfile/kafka-manager

方法1：
cat Dockerfile

FROM hseeberger/scala-sbt

ENV ZK_HOSTS=10.4.7.11:2181 \
     KM_VERSION=2.0.0.2

RUN mkdir -p /tmp && \
    cd /tmp && \
    wget https://github.com/yahoo/kafka-manager/archive/${KM_VERSION}.tar.gz && \
    tar xf ${KM_VERSION}.tar.gz && \
    cd /tmp/CMAK-${KM_VERSION} && \
    sbt clean dist && \
    unzip  -d / ./target/universal/kafka-manager-${KM_VERSION}.zip && \
    rm -fr /tmp/${KM_VERSION} /tmp/kafka-manager-${KM_VERSION}

WORKDIR /kafka-manager-${KM_VERSION}

EXPOSE 9000
ENTRYPOINT ["./bin/kafka-manager","-Dconfig.file=conf/application.conf"]

docker build . -t harbor.od.com/infra/kafka-manager:v2.0.0.2

方法2：
直接下载镜像
https://hub.docker.com/r/sheepkiller/kafka-manager/tags

资源配置清单

/data/k8s-yaml/kafka-manager目录
cat dp.yaml

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: kafka-manager
  namespace: infra
  labels: 
    name: kafka-manager
spec:
  replicas: 1
  selector:
    matchLabels: 
      app: kafka-manager
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600
  template:
    metadata:
      labels: 
        app: kafka-manager
    spec:
      containers:
      - name: kafka-manager
        image: harbor.od.com/infra/kafka-manager:v2.0.0.2
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9000
          protocol: TCP
        env:
        - name: ZK_HOSTS
          value: zk1.od.com:2181
        - name: APPLICATION_SECRET
          value: letmein
      imagePullSecrets:
      - name: harbor
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0

cat svc.yaml

kind: Service
apiVersion: v1
metadata: 
  name: kafka-manager
  namespace: infra
spec:
  ports:
  - protocol: TCP
    port: 9000
    targetPort: 9000
  selector: 
    app: kafka-manager

cat ingress.yaml

kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: kafka-manager
  namespace: infra
spec:
  rules:
  - host: km.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: kafka-manager
          servicePort: 9000

dns解析记录

km                 A    10.4.7.10

应用

kubectl apply -f http://k8s-yaml.od.com/kafka-manager/dp.yaml
kubectl apply -f http://k8s-yaml.od.com/kafka-manager/svc.yaml
kubectl apply -f http://k8s-yaml.od.com/kafka-manager/ingress.yaml

浏览器访问：
http://km.od.com/

在这里插入图片描述

filebeat

https://www.elastic.co/downloads/beats/filebeat

制作docker镜像

mkdir /data/dockerfile/filebeat
cd /data/dockerfile/filebeat

cat Dockerfile

FROM debian:jessie

ENV FILEBEAT_VERSION=7.4.0 \
    FILEBEAT_SHA1=c63bb1e16f7f85f71568041c78f11b57de58d497ba733e398fa4b2d071270a86dbab19d5cb35da5d3579f35cb5b5f3c46e6e08cdf840afb7c347777aae5c4e11

RUN set -x && \
  apt-get update && \
  apt-get install -y wget && \
  wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-${FILEBEAT_VERSION}-linux-x86_64.tar.gz -O /opt/filebeat.tar.gz && \
  cd /opt && \
  echo "${FILEBEAT_SHA1}  filebeat.tar.gz" | sha512sum -c - && \
  tar xzvf filebeat.tar.gz && \
  cd filebeat-* && \
  cp filebeat /bin && \
  cd /opt && \
  rm -rf filebeat* && \
  apt-get purge -y wget && \
  apt-get autoremove -y && \
  apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

COPY docker-entrypoint.sh /
ENTRYPOINT ["/docker-entrypoint.sh"]

cat docker-entrypoint.sh

#!/bin/bash

ENV=${ENV:-"test"}
PROJ_NAME=${PROJ_NAME:-"no-define"}
MULTILINE=${MULTILINE:-"^\d{2}"}

cat > /etc/filebeat.yaml << EOF
filebeat.inputs:
- type: log
  fields_under_root: true
  fields:
    topic: logm-${PROJ_NAME}
  paths:
    - /logm/*.log
    - /logm/*/*.log
    - /logm/*/*/*.log
    - /logm/*/*/*/*.log
    - /logm/*/*/*/*/*.log
  scan_frequency: 120s
  max_bytes: 10485760
  multiline.pattern: '$MULTILINE'
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 100
- type: log
  fields_under_root: true
  fields:
    topic: logu-${PROJ_NAME}
  paths:
    - /logu/*.log
    - /logu/*/*.log
    - /logu/*/*/*.log
    - /logu/*/*/*/*.log
    - /logu/*/*/*/*/*.log
    - /logu/*/*/*/*/*/*.log
output.kafka:
  hosts: ["10.4.7.11:9092"]
  topic: k8s-fb-$ENV-%{[topic]}
  version: 2.0.0
  required_acks: 0
  max_message_bytes: 10485760
EOF

set -xe

# If user don't provide any command
# Run filebeat
if [[ "$1" == "" ]]; then
     exec filebeat  -c /etc/filebeat.yaml 
else
    # Else allow the user to run arbitrarily commands like bash
    exec "$@"
fi

docker build . -t harbor.od.com/infra/filebeat:v7.4.0
docker push harbor.od.com/infra/filebeat:v7.4.0

资源配置清单

cat /data/k8s-yaml/dubbo-demo-consumer/dp.yaml

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-demo-consumer
  namespace: test
  labels: 
    name: dubbo-demo-consumer
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-demo-consumer
  template:
    metadata:
      labels: 
        app: dubbo-demo-consumer
        name: dubbo-demo-consumer
    spec:
      containers:
      - name: dubbo-demo-consumer
        image: harbor.od.com/app/dubbo-demo-web:tomcat_200610_1556
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          protocol: TCP
        env:
        - name: C_OPTS
          value: -Denv=fat -Dapollo.meta=http://config-test:8080
        volumeMounts:
        - mountPath: /opt/tomcat/logs
          name: logm
      - name: filebeat
        image: harbor.od.com/infra/filebeat:v7.4.0
        imagePullPolicy: IfNotPresent
        env:
        - name: ENV
          value: test
        - name: PROJ_NAME
          value: dubbo-demo-web
        volumeMounts:
        - mountPath: /logm
          name: logm
      volumes:
      - emptyDir: {}
        name: logm
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

边车模式

UTS  user   net  共享 
IPC  pid   fs   隔离

kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-consumer/dp.yaml

logstash

选型
https://www.elastic.co/support/matrix

准备docker镜像

docker pull logstash:6.8.3
docker tag 972bf55ad27b harbor.od.com/infra/logstash:v6.8.3
docker push harbor.od.com/infra/logstash:v6.8.3

mkdir /etc/logstash


cat /etc/logstash/logstash-test.conf
input {
  kafka {
    bootstrap_servers => "10.4.7.11:9092"
    client_id => "10.4.7.200"
    consumer_threads => 4
    group_id => "k8s_test"
    topics_pattern => "k8s-fb-test-.*"
  }
}

filter {
  json {
    source => "message"
  }
}

output {
  elasticsearch {
    hosts => ["10.4.7.12:9200"]
    index => "k8s-test-%{+YYYY.MM.DD}"
  }
}

启动logstash镜像

docker run -d --name logstash-test -v /etc/logstash:/etc/logstash harbor.od.com/infra/logstash:v6.8.3 -f /etc/logstash/logstash-test.conf

验证ElasticSearch里的索引

curl http://10.4.7.12:9200/_cat/indices?v
health status index                uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   k8s-test-2020.06.163 ksrdkQ2rTPK3D_EBNktOuQ   5   0         41            0      107kb          107kb

Kibana

https://hub.docker.com/_/kibana?tab=tags

准备docker镜像

docker pull kibana:6.8.3
docker tag 54db200915ee harbor.od.com/infra/kibana:v6.8.3
docker push harbor.od.com/infra/kibana:v6.8.3

资源配置清单

mkdir /data/k8s-yaml/kibana
cd /data/k8s-yaml/kibana

cat dp.yaml

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: kibana
  namespace: infra
  labels: 
    name: kibana
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: kibana
  template:
    metadata:
      labels: 
        app: kibana
        name: kibana
    spec:
      containers:
      - name: kibana
        image: harbor.od.com/infra/kibana:v6.8.3
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 5601
          protocol: TCP
        env:
        - name: ELASTICSEARCH_URL
          value: http://10.4.7.12:9200
      imagePullSecrets:
      - name: harbor
      securityContext: 
        runAsUser: 0
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

cat svc.yaml

kind: Service
apiVersion: v1
metadata: 
  name: kibana
  namespace: infra
spec:
  ports:
  - protocol: TCP
    port: 5601
    targetPort: 5601
  selector: 
    app: kibana

cat ingress.yaml

kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: kibana
  namespace: infra
spec:
  rules:
  - host: kibana.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: kibana
          servicePort: 5601

dns解析

kibana             A    10.4.7.10

应用

kubectl apply -f http://k8s-yaml.od.com/kibana/dp.yaml
kubectl apply -f http://k8s-yaml.od.com/kibana/svc.yaml
kubectl apply -f http://k8s-yaml.od.com/kibana/ingress.yaml

浏览器访问：
http://kibana.od.com

Monitoring

在这里插入图片描述

Management

在这里插入图片描述

Discover

在这里插入图片描述

时间选择器
在这里插入图片描述

环境选择器
在这里插入图片描述

项目选择器

在这里插入图片描述

关键字选择器

在这里插入图片描述

spinnaker
https://www.spinnaker.io/

https://github.com/spinnaker/spinnaker

wuxingge

关注

0
点赞
踩
2

收藏

觉得还不错? 一键收藏
打赏
0
评论
K8S ELK日志分析

日志，对于任何系统来说都是及其重要的组成部分，在计算机系统中比较复杂，日志由不同的来源，如操作系统，应用服务，业务逻辑等，它们都在不停产生各种各样的日志K8S系统里的业务应用是高度 "动态化"的，随着容器编排的进行，业务容器在不断的被创建、被销毁、被迁移、被扩缩容…需要建立一套集中式的方法，把不同来源的数据集中整合到一个地方https://blog.stanley.wang/2019/01/18/%E5%AE%9E%E9%AA%8C%E6%96%87%E6%A1%A34%EF%BC%9..
复制链接

扫一扫