Prometheus Node_exporter增加认证

背景

因安全需要，现在对 node_exporter 进行配置以支持 TLS 和 Basic Auth，顺便把 Prometheus 升级到最新版 2.40.4

Node_exporter 1.0 以上版本才支持 TLS 和 Basic Auth

Node_exporter 配置

准备工作

下载地址：

wget https://github.com/prometheus/node_exporter/releases/download/v1.5.0/node_exporter-1.5.0.linux-amd64.tar.gz

解压 Node_exporter

tar zxvf node_exporter-1.5.0.linux-amd64.tar.gz

tls 证书生成

openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout node_exporter.key -out node_exporter.crt -subj "/C=CN/ST=Beijing/L=Beijing/O=Moelove.info/CN=localhost"
Generating a RSA private key
...................+++++
.........................................................................................................................................................................................................................................................................+++++
writing new private key to 'node_exporter.key'
-----
root@zabbix:~/node_exporter# ll
total 16
drwxr-xr-x  2 root root 4096 Dec  1 14:58 ./
drwx------ 27 root root 4096 Dec  1 14:58 ../
-rw-r--r--  1 root root 1310 Dec  1 14:58 node_exporter.crt
-rw-------  1 root root 1704 Dec  1 14:58 node_exporter.key

通过上面的步骤，我们得到了 node_exporter.crt 和 node_exporter.key 这两个文件。

basic auth 认证生成

安装 htpasswd 来生成密码 hash

#Ubuntu
apt install apache2-utils -y
#centos
yum install httpd-tools -y

在 Node_exporter 目录下执行

# htpasswd -nBC 12 '' | tr -d ':\n'   
New password: 
Re-type new password: 
$2y$12$LWDB21oY/67kz3lu8Y7bGOcerRodeAK4SfOSmielNq.BzMlrjw1/q

配置

将前面生成的 node_exporter.crt 和 node_exporter.key 文件复制到 Node_exporter 解压目录下。

root@zabbix:/opt/node_exporter# cp /root/node_exporter/node_exporter.* .
root@zabbix:/opt/node_exporter# ll
total 19352
drwxr-xr-x 2 root root     4096 Dec  1 15:12 ./
drwxr-xr-x 5 root root     4096 Dec  1 15:07 ../
-rw-r--r-- 1 3434 3434    11357 Nov 30 03:05 LICENSE
-rw-r--r-- 1 3434 3434      463 Nov 30 03:05 NOTICE
-rwxr-xr-x 1 3434 3434 19779640 Nov 30 02:59 node_exporter*
-rw-r--r-- 1 root root     1310 Dec  1 15:12 node_exporter.crt
-rw------- 1 root root     1704 Dec  1 15:12 node_exporter.key
root@zabbix:/opt/node_exporter#

编写配置文件，并保存为 config.yaml (命名随意)：

tls_server_config:
  cert_file: node_exporter.crt
  key_file: node_exporter.key
basic_auth_users:
  # 当前设置的用户名为 prometheus ， 可以设置多个
  prometheus: $2y$12$LWDB21oY/67kz3lu8Y7bGOcerRodeAK4SfOSmielNq.BzMlrjw1/q

启动

nohup ./node_exporter --web.listen-address=:39100 --web.config.file=config.yaml &

验证

root@zabbix:/opt/node_exporter# curl http://localhost:39100/metrics 
Client sent an HTTP request to an HTTPS server.
root@zabbix:/opt/node_exporter# curl https://localhost:39100/metrics 
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
root@zabbix:/opt/node_exporter#

可以看到不能直接访问了，下面带上证书及用户密码再次测试

root@zabbix:/opt/node_exporter# curl -u prometheus -s  --cacert node_exporter.crt https://localhost:39100/metrics |grep node_exporter_build_info
Enter host password for user 'prometheus':
# HELP node_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which node_exporter was built.
# TYPE node_exporter_build_info gauge
node_exporter_build_info{branch="HEAD",goversion="go1.19.3",revision="1b48970ffcf5630534fb00bb0687d73c66d1c959",version="1.5.0"} 1

Prometheus 配置

下载最新版解压，并将前面生成的 node_exporter.crt 和 node_exporter.key 文件复制到该目录下。

wget https://mirrors.tuna.tsinghua.edu.cn/github-release/prometheus/prometheus/LatestRelease/prometheus-2.40.4.linux-amd64.tar.gz

在 prometheus.yml 加入如下内容：

global:
  scrape_interval:     15s 
  evaluation_interval: 15s 

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    scheme: https
    tls_config:
      ca_file: node_exporter.crt
      insecure_skip_verify: true
    basic_auth:
      username: prometheus
      password: 123456
    static_configs:
    - targets: ['localhost:9100']

启动 Prometheus 即可。

批量更新 Node_exporter

下载、解压

wget https://raw.githubusercontent.com/fsckzy/work-software-collection/main/node_exporter_ansible.zip
unzip node_exporter_ansible.zip
# 文件如下
[root@0001 yy]# ll
total 105612
-rw-r--r-- 1 root root       69 Dec  1 15:54 main.yaml
drwxr-xr-x 6 root root       59 Dec  1 15:44 node_exporter

这个 ansible roles 里包含了 Node_exporter 二进制文件， node_exporter.crt 和 node_exporter.key，config.yaml ，systemd 文件。

[root@0001 yy]# cd node_exporter/
[root@0001 node_exporter]# ls
default  files  tasks  vars
[root@0001 node_exporter]# ls *
default:

files:
node_exporter.service  node_exporter.zip

tasks:
main.yaml

vars:
[root@0001 node_exporter]# cat tasks/main.yaml 
---


- name: unrar node to agent.
  unarchive:
    src: node_exporter.zip
    dest: /opt/

- name: Copy systemd to agent.
  copy:
    src: node_exporter.service
    dest: /etc/systemd/system/node_exporter.service

- name: start service
  systemd:
    state: started
    daemon_reload: yes
    name: node_exporter
[root@0001 node_exporter]#

main.yaml

[root@0001 yy]# cat main.yaml 
- hosts: localhost
  gather_facts: no
  roles:
    - node_exporter

以后重启 Node_exporter，需要使用 systemd

systemctl restart node_exporter

建议先找一台机器进行测试。