k8s monitor

1.create AKS cluster

az group create -n aks2019rg -l eastus

az aks get-versions -l eastus -o table

az aks create -g aks2019rg -n aks2019 --node-count 2 --node-vm-size Standard_A2 --disable-rbac --generate-ssh-keys --kubernetes-version 1.11.5 --no-wait

2.install and config Helm

VER=v2.11.0
wget https://mirror.azure.cn/kubernetes/helm/helm-$VER-linux-amd64.tar.gztar-zxvfhelm-v2.11.0-linux-amd64.tar.gzsudomvlinux-amd64/helm/usr/local/binsudohelminit--tiller-imagegcr.azk8s.cn/kubernetes-helm/tiller:$VER --stable-repo-url https://mirror.azure.cn/kubernetes/charts/

3.install and config monitoring system

architecture:

prometheus vs zabbix

创建 ConfigMap，此ConfigMap将作为Prometheus Server的配置文件，用来记录需要监控的资源 metric endpoint 信息及配置。

kubectl create configmap prometheus-config --from-file ./demo/prometheus.yml -n monitoring

创建 Prometheus Deployment

kubectl create -f ./demo/prometheus-deploy.yml

shen@Azure:~$ cat prometheus-deploy.yml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: prometheus-deployment
namespace: monitoring
spec:
replicas: 1
template:
metadata:
labels:
app: prometheus-server
spec:
containers:
- name: prometheus
image: prom/prometheus
args:
- “–config.file=/etc/prometheus/prometheus.yml”
- “–storage.tsdb.path=/prometheus/”
ports:
- containerPort: 9090
volumeMounts:
- name: prometheus-config-volume
mountPath: /etc/prometheus/
- name: prometheus-storage-volume
mountPath: /prometheus/
volumes:
- name: prometheus-config-volume
configMap:
defaultMode: 420
name: prometheus-config

    - name: prometheus-storage-volume
      emptyDir: {}

---

apiVersion: v1
kind: Service
metadata:
labels:
app: prometheus-server
name: prometheus-deployment
namespace: monitoring
spec:
ports:

• port: 9090
  protocol: TCP
  targetPort: 9090
  selector:
  app: prometheus-server
  type: LoadBalancer

为Prometheus配置两个 metric endpoints, 即 kube-state-metrics，node-exporter, 来收AKS集群中资源使用情况的信息，实际应用中，我们可以通过更多的metric endpoints 收集不同层面的Metrics信息。

#清除刚创建的node-exporter
helm del --purge node-exporter

通过新创建的 Service Account tiller，重新初始化Helm
helm init --tiller-image gcr.azk8s.cn/kubernetes-helm/tiller:v2.11.0 --stable-repo-url https://mirror.azure.cn/kubernetes/charts/ --service-account tiller --upgrade

#重新创建 node-exporter
helm install --name node-exporter stable/prometheus-node-exporter --namespace monitoring

git clone https://github.com/kubernetes/kube-state-metrics.git

install kube-state-metrics

update prometheus.yml

add new targets to Prometheus:

then the monitor info has been collected by Prometheus:

install Grafana by helm:
helm install --name grafana stable/grafana --set service.type=LoadBalancer --set sidecar.datasources.enabled=true --set sidecar.dashboards.enabled=true --set sidecar.datasources.label=grafana_datasource --set sidecar.dashboards.label=grafana_dashboard --namespace monitoring

kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

configure Prometheus DataSource:

add dashboard:

install Log System:Elasticsearch & Fluent & Kibana

add Helm repo，install ElasticSearch Operator

helm repo add akomljen-charts https://raw.githubusercontent.com/komljen/helm-charts/master/charts/

install ElasticSearch

helm install --name efk --namespace logging akomljen-charts/efk

change Kibana Service type:LoadBalancer:

kubectl edit svc efk-kibana -n logging