Prometheus + Grafana + Pushgateway部署

部署Prometheus&Grafana

step1.下载项目git项目

	git clone https://github.com/coreos/kube-prometheus.git  

step2.拉取镜像

因为某些镜像被墙了，所以需要手动从阿里云上拉取镜像
该镜像是kube-state-metrics部署中所需要的一个镜像，下面进行镜像下载：

addon-resizer

sudo docker pull mirrorgooglecontainers/addon-resizer:1.8.4

---

    #显示以下信息则代表下载成功
	1.8.4: Pulling from mirrorgooglecontainers/addon-resizer
	Digest:sha256:737c634a95f04bffbcd8d3a833745cc26d4d42e8a759ca75b8b8b407ef40b780
	Status: Image is up to date for mirrorgooglecontainers/addon-resizer:1.8.4
	docker.io/mirrorgooglecontainers/addon-resizer:1.8.4

---

#然后将仓库名称：tag修改成部署的yaml文件对应的信息。
sudo docker tag mirrorgooglecontainers/addon-resizer:1.8.4 k8s.gcr.io/addon-resizer:1.8.4

prometheus

#从阿里云拉取镜像
sudo docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/prometheus

---

#显示以下信息代表镜像拉取成功

Using default tag: latest
latest: Pulling from google_containers/prometheus
Digest: sha256:a36643ecff25461626eb259e11309dbbff400839a0a90e2dea19c34267abd0a3
Status: Image is up to date for registry.cn-hangzhou.aliyuncs.com/google_containers/prometheus:latest
registry.cn-hangzhou.aliyuncs.com/google_containers/prometheus:latest

#修改tag
sudo docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/prometheus quay.io/prometheus/prometheus

其余的镜像在部署过程中可以正常下载，在后续的同学的部署中如果遇到镜像不能下载，可以自己寻找镜像网站进行下载并修改tag。

step3. 修改grafana-service.yaml

进入下载好的项目中修改yaml文件，使得外网可以通过nodePort方式访问Grafana：

cd ~/kube-prometheus/manifests
vim grafana-service.yaml

---

apiVersion: v1
kind: Service
metadata:
  labels:
    app: grafana
  name: grafana
  namespace: monitoring
spec:
  type: NodePort #增加类型NodePort
  ports:
  - name: http
    nodePort: 30006 #手动设定访问端口30006
    port: 3000
    targetPort: http
  selector:
    app: grafana

---

step4. 修改prometheus-service.yaml

进入下载好的项目中修改yaml文件，使得外网可以通过nodePort方式访问prometheus

---

apiVersion: v1
kind: Service
metadata:
  labels:
    prometheus: k8s
  name: prometheus-k8s
  namespace: monitoring
spec:
  type: NodePort #增加类型NodePort
  ports:
  - name: web
    port: 9090
    nodePort: 30007#手动设定访问端口30007
    targetPort: web
  selector:
    app: prometheus
    prometheus: k8s
  sessionAffinity: ClientIP

---

step5.开始部署prometheus + grafana

在开始之前，确认你的manifests目录下有这些yaml文件

	cd ~/kube-prometheus/manifests
	ls

00namespace-namespace.yaml                                         node-exporter-clusterRole.yaml
0prometheus-operator-0alertmanagerCustomResourceDefinition.yaml    node-exporter-daemonset.yaml
0prometheus-operator-0prometheusCustomResourceDefinition.yaml      node-exporter-serviceAccount.yaml
0prometheus-operator-0prometheusruleCustomResourceDefinition.yaml  node-exporter-serviceMonitor.yaml
0prometheus-operator-0servicemonitorCustomResourceDefinition.yaml  node-exporter-service.yaml
0prometheus-operator-clusterRoleBinding.yaml                       prometheus-adapter-apiService.yaml
0prometheus-operator-clusterRole.yaml                              prometheus-adapter-clusterRoleAggregatedMetricsReader.yaml
0prometheus-operator-deployment.yaml                               prometheus-adapter-clusterRoleBindingDelegator.yaml
0prometheus-operator-serviceAccount.yaml                           prometheus-adapter-clusterRoleBinding.yaml
0prometheus-operator-serviceMonitor.yaml                           prometheus-adapter-clusterRoleServerResources.yaml
0prometheus-operator-service.yaml                                  prometheus-adapter-clusterRole.yaml
alertmanager-alertmanager.yaml                                     prometheus-adapter-configMap.yaml
alertmanager-secret.yaml                                           prometheus-adapter-deployment.yaml
alertmanager-serviceAccount.yaml                                   prometheus-adapter-roleBindingAuthReader.yaml
alertmanager-serviceMonitor.yaml                                   prometheus-adapter-serviceAccount.yaml
alertmanager-service.yaml                                          prometheus-adapter-service.yaml
grafana-dashboardDatasources.yaml                                  prometheus-clusterRoleBinding.yaml
grafana-dashboardDefinitions.yaml                                  prometheus-clusterRole.yaml
grafana-dashboardSources.yaml                                      prometheus-prometheus.yaml
grafana-deployment.yaml                                            prometheus-roleBindingConfig.yaml
grafana-serviceAccount.yaml                                        prometheus-roleBindingSpecificNamespaces.yaml
grafana-serviceMonitor.yaml                                        prometheus-roleConfig.yaml
grafana-service.yaml                                               prometheus-roleSpecificNamespaces.yaml
kube-state-metrics-clusterRoleBinding.yaml                         prometheus-rules.yaml
kube-state-metrics-clusterRole.yaml                                prometheus-serviceAccount.yaml
kube-state-metrics-deployment.yaml                                 prometheus-serviceMonitorApiserver.yaml
kube-state-metrics-roleBinding.yaml                                prometheus-serviceMonitorCoreDNS.yaml
kube-state-metrics-role.yaml                                       prometheus-serviceMonitorKubeControllerManager.yaml
kube-state-metrics-serviceAccount.yaml                             prometheus-serviceMonitorKubelet.yaml
kube-state-metrics-serviceMonitor.yaml                             prometheus-serviceMonitorKubeScheduler.yaml
kube-state-metrics-service.yaml                                    prometheus-serviceMonitor.yaml
node-exporter-clusterRoleBinding.yaml                              prometheus-service.yaml

确认无误之后，开始下面的步骤

cd ~/kube-prometheus
kubectl create -f manifests/ || true
#由于部署所需的资源中存在竞争性，即前面的服务部署可能需要用到后面的安装的资源，所以下面的apply命令可能需要执行两遍
kubectl apply -f manifests/ 2>/dev/null || true  
#再次执行该命令
kubectl apply -f manifests/ 2>/dev/null || true
#查看部署的pod信息
kubectl  get pod -n monitoring -o wide
#节点部署信息

---

alertmanager-main-0                   2/2     Running   0          17h   10.244.1.29     hp-55   <none>           <none>
alertmanager-main-1                   2/2     Running   0          17h   10.244.4.26     hp-52   <none>           <none>
alertmanager-main-2                   2/2     Running   0          17h   10.244.3.30     hp-51   <none>           <none>
grafana-57bfdd47f8-sxxln              1/1     Running   0          17h   10.244.2.34     hp-50   <none>           <none>
kube-state-metrics-59c47cfcfb-nfc8c   4/4     Running   0          14h   10.244.4.28     hp-52   <none>           <none>
node-exporter-mn9xf                   2/2     Running   0          17h   192.168.30.55   hp-55   <none>           <none>
node-exporter-nqk7k                   2/2     Running   0          17h   192.168.30.51   hp-51   <none>           <none>
node-exporter-pxkgf                   2/2     Running   0          17h   192.168.30.54   hp-54   <none>           <none>
node-exporter-xv9nh                   2/2     Running   0          17h   192.168.30.50   hp-50   <none>           <none>
node-exporter-zwpw6                   2/2     Running   0          17h   192.168.30.52   hp-52   <none>           <none>
prometheus-adapter-668748ddbd-7w2j7   1/1     Running   0          17h   10.244.1.28     hp-55   <none>           <none>
prometheus-k8s-0                      3/3     Running   1          17h   10.244.2.35     hp-50   <none>           <none>
prometheus-k8s-1                      3/3     Running   6          17h   10.244.3.31     hp-51   <none>           <none>
prometheus-operator-55b978b89-8jz2l   1/1     Running   0          17h   10.244.4.25     hp-52   <none>           <none>

---

step6.开始部署pushgateway

在你的manifest目录下，创建这三个yaml文件。

cat > prometheus-pushgatewayServiceMonitor.yaml <<EOF
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    prometheus: k8s
  name: prometheus-pushgateway
  namespace: monitoring
spec:
  endpoints:
  - honorLabels: true
    port: http
  jobLabel: k8s-app
  selector:
    matchLabels:
      app: prometheus-pushgateway
EOF

cat > prometheus-pushgatewayService.yaml <<EOF
apiVersion: v1
kind: Service
metadata:
  labels:
    app: prometheus-pushgateway
  name: prometheus-pushgateway
  namespace: monitoring
spec:
  type: NodePort
  ports:
  - name: http
    port: 9091
    nodePort: 30008
    targetPort: metrics
  selector:
    app: prometheus-pushgateway
#  type: ClusterIP
EOF

cat >prometheus-pushgatewayDeployment.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prometheus-pushgateway
  name: prometheus-pushgateway
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-pushgateway
  template:
    metadata:
      labels:
        app: prometheus-pushgateway
    spec:
      containers:
      - image: prom/pushgateway:v0.8.0
        livenessProbe:
          httpGet:
            path: /#/status
            port: 9091
          initialDelaySeconds: 10
          timeoutSeconds: 10
        name: prometheus-pushgateway
        ports:
        - containerPort: 9091
          name: metrics
        readinessProbe:
          httpGet:
            path: /#/status
            port: 9091
          initialDelaySeconds: 10
          timeoutSeconds: 10
        resources:
          limits:
            cpu: 50m
            memory: 100Mi
          requests:
            cpu: 50m
            memory: 100Mi
EOF

然后使用

kubectl apply -f .