k8s 离线安装 kube-prometheus-stack

配置共享存储

Prometheus 需要配置持久化存储，防止数据丢失

服务端

服务端安装 NFS 服务

sudo apt install nfs-kernel-server

创建共享目录，在服务器端创建 /nfs 目录。

mkdir /nfs
chmod -R 777 /nfs # 设置文件权限

nfs目录下只给了默认权限，不设置权限，会报错

GF_PATHS_DATA='/var/lib/grafana' is not writable.
You may have issues with file permissions, more information here: http://docs.grafana.org/installation/docker/#migrate-to-v51-or-later
mkdir: can't create directory '/var/lib/grafana/plugins': Permission denied

编写配置文件

vim /etc/exports
#[任意主机所有权限]
/nfs   *(rw,sync,insecure,no_subtree_check,no_root_squash)

重启 NFS 服务

sudo service nfs-kernel-server restart

常用命令工具

#在安装 NFS 服务器时，已包含常用的命令行工具，无需额外安装
#显示已经 mount 到本机 NFS 目录的客户端机器
sudo showmount -e localhost
#将配置文件中的目录全部重新 export 一次，无需重启服务
sudo exportfs -rv
#查看 NFS 的运行状态
sudo nfsstat
#查看 rpc 执行信息，可以用于检测 rpc 运行情况
sudo rpcinfo

客户端

需要连接服务端的节点，例如 node 节点

安装客户端工具

sudo apt install nfs-common

查看 NFS 服务器上的共享目录

#显示指定的 NFS 服务器(假设 IP 地址为 192.168.58.29)上 export 出来的目录
sudo showmount -e 192.168.58.29

创建本地挂载目录

sudo mkdir -p /nfs

挂载共享目录

#假设 NFS 服务器 IP为 192.168.58.29，可以如下设置挂载  
sudo mount -t nfs 192.168.58.29:/nfs /nfs

开始安装 Prometheus

下载 Prometheus 压缩包

在 releases 中，找到自己想安装的版本：https://github.com/prometheus-community/helm-charts/releases/

wget https://github.com/prometheus-community/helm-charts/releases/download/kube-prometheus-stack-72.3.1/kube-prometheus-stack-72.3.1.tgz

配置持久化卷

创建 prometheus-storage-class.yaml 文件

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: prometheus-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

创建 prometheus-pv.yaml 文件

apiVersion: v1
kind: PersistentVolume
metadata:
  name: prometheus-pv
spec:
  capacity:
    storage: 40Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: prometheus-storage
  nfs:
    path: /nfs/prometheus
    server: 192.168.58.29
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: alertmanager-pv
spec:
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: prometheus-storage
  nfs:
    path: /nfs/alertmanager
    server: 192.168.58.29
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: grafana-pv
spec:
  capacity:
    storage: 8Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: prometheus-storage
  nfs:
    path: /nfs/grafana
    server: 192.168.58.29

执行安装 StorageClass、pv

kubectl apply -f prometheus-storage-class.yaml
kubectl apply -f prometheus-pv.yaml

验证安装

kubectl get StorageClass
kubectl get pv

配置 Prometheus 自定义配置

创建 values.yaml 文件，启用持久化存储并指定 StorageClass：

prometheus:
  prometheusSpec:
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: prometheus-storage
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 40Gi
alertmanager:
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: prometheus-storage
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 2Gi
grafana:
  persistence:
    enabled: true
    storageClassName: prometheus-storage
    accessModes: ["ReadWriteOnce"]
    size: 8Gi

创建共享存储文件夹

mkdir -p /nfs/alertmanager /nfs/grafana /nfs/prometheus

chmod -R 777 /nfs

helm 离线安装

helm install prometheus ./kube-prometheus-stack-72.3.1.tgz \
  --namespace monitoring \
  --create-namespace \
  -f values.yaml

安装完成，输出日志：

NAME: prometheus
LAST DEPLOYED: Wed May 14 15:41:31 2025
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
  kubectl --namespace monitoring get pods -l "release=prometheus"

Get Grafana 'admin' user password by running:

  kubectl --namespace monitoring get secrets prometheus-grafana -o jsonpath="{.data.admin-password}" | base64 -d ; echo

Access Grafana local instance:

  export POD_NAME=$(kubectl --namespace monitoring get pod -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=prometheus" -oname)
  kubectl --namespace monitoring port-forward $POD_NAME 3000

Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.

卸载 Prometheus

helm uninstall prometheus --namespace monitoring