Skywalking_k8s集群的Jvm监控

Skywalking针对jvm监控无入侵-k8s集群

介绍

咱中国人自己写的一个APM开源项目，成功被Apache认可进行孵化，具体的看Git吧。

git地址： https://github.com/apache/skywalking

很多大厂拿着个项目进行二次开发改为自己的云服务，我知道的就有阿里云、腾讯云、七牛云、华为【ps:开发者是华为的大佬】。此举是对着个项目的认可，但是很贵，“贵”懂吧，多的不说了。

Github开源的适合本地部署，要上云还是需要一定的操作的。Git上是有Helm来部署在云上面的【https://github.com/apache/skywalking-kubernetes】，但是人家说了着个不维护，而且坑多。

不知道别人是怎么部署的，我是自己把Skywalking看成两个部分部署的，server+agent.

server: busybox + elasticsearch + oap + ui

agent: 我是通过注入容器的方式把agent放到自己想监控的服务里。【ps:还有人是直接把探针包和自己的服务构建成一个war包】

server部署

所有的部署在skywalking这个namespace下,否则会有报错。想部署在其他命名空间下，修改大部分配置文件。

busybox

busybox.yml

我这里sleep时长设置的比较长，用来减少重启的频率

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox
spec:
  selector:
    matchLabels:
      app: busybox
  replicas: 1
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
      - name: busybox
        image: busybox:1.30
        resources:
          requests:
            cpu: 100m
            memory: 1Gi
          limits:
            cpu: 100m
            memory: 1Gi
        command:
        - sleep
        - "36000"
        imagePullPolicy: Always
        ports:
        - containerPort: 80
          protocol: TCP

elasticsearch

存储是根据官方推荐的es做的，在git上也看到有人mysql做存储的。应该其他的也可以，有需求去看源码再去做。

elasticsearch-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  labels:
    service: elasticsearch
spec:
  clusterIP: None
  ports:
  - port: 9200
    name: serving
  - port: 9300
    name: node-to-node
  selector:
    service: elasticsearch

elasticsearch-statefulset.yaml

这里es的资源限制根据自己需求去定制。具体多少agent需要多少内存暂时还没有测。

PV、PVC用的阿里云的。

印象中官方都是多副本的，我这里对数据保留没要求，内部使用不怕丢数据，只用了一个。

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  labels:
    service: elasticsearch
spec:
  serviceName: elasticsearch
  # NOTE: This is number of nodes that we want to run
  # you may update this
  replicas: 1
  selector:
    matchLabels:
      service: elasticsearch
  template:
    metadata:
      labels:
        service: elasticsearch
    spec:
      terminationGracePeriodSeconds: 300
      initContainers:
      # NOTE:
      # This is to fix the permission on the volume
      # By default elasticsearch container is not run as
      # non root user.
      # https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#_notes_for_production_use_and_defaults
      - name: fix-the-volume-permission
        image: busybox
        imagePullPolicy: IfNotPresent
        command:
        - sh
        - -c
        - chown -R 1000:1000 /usr/share/elasticsearch/data
        securityContext:
          privileged: true
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
      # NOTE:
      # To increase the default vm.max_map_count to 262144
      # https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-cli-run-prod-mode
      - name: increase-the-vm-max-map-count
        image: busybox
        imagePullPolicy: IfNotPresent
        command:
        - sysctl
        - -w
        - vm.max_map_count=262144
        securityContext:
          privileged: true
      # To increase the ulimit
      # https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#_notes_for_production_use_and_defaults
      - name: increase-the-ulimit
        image: busybox
        imagePullPolicy: IfNotPresent
        command:
        - sh
        - -c
        - ulimit -n 65536
        securityContext:
          privileged: true
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.3.2
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9200
          name: http
        - containerPort: 9300
          name: tcp
        # NOTE: you can increase this resources
        resources:
          requests:
            memory: 4Gi
          limits:
            memory: 4Gi
        env:
          # NOTE: the cluster name; update this
          - name: cluster.name
            value: elasticsearch-cluster
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          # NOTE: This will tell the elasticsearch node where to connect to other nodes to form a cluster
          - name: discovery.zen.ping.unicast.hosts
            value: elasticsearch:9300
          # NOTE: You can increase the heap size
          - name: ES_JAVA_OPTS
            value: -Xms3g -Xmx3g
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
        - ReadWriteOnce
      storageClassName: alicloud-disk-efficiency-shanghai-bdf
      # NOTE: You can increase the storage size
      resources:
        requests:
          storage: 50Gi

oap

oap就是skywalking的核心所在。

rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: skywalking-oap
  namespace: skywalking

---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: skywalking-oap
  namespace: skywalking
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: skywalking-oap
subjects:
-