PMM初体验

 

 

一、监控服务端

1、安装docker

yum remove docker docker-client docker-client-latest docker-common docker-latest docker-latest-logrotate docker-logrotate docker-selinux docker-engine-selinux docker-engine

yum install yum-utils device-mapper-persistent-data lvm2

yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

yum install docker-ce

systemctl start docker

systemctl enable docker

 

2、配置docker存储目录、国内docker源

配置文件：/etc/docker/daemon.json

{
    "data-root": "/data/docker",
    "registry-mirrors":["https://registry.docker-cn.com"]
}

data-root按需配置，默认在/var/lib/docker

systemctl daemon-reload

systemctl restart docker

 

3、拉取PMM Server镜像

docker pull percona/pmm-server:1.17.3

附：

查看已经拉取的镜像清单：

docker images

删除指定IMAGE ID的镜像：

docker rmi ada7dbc5a8bc

注：删除镜像前，先删除对应容器

    docker ps -a

    docker rm 容器ID

 

4、创建容器

创建一个容器用来保存PMM Server数据

docker create --name pmm-data percona/pmm-server:1.17.3 /bin/true

5、运行

运行以下命令：

docker run -d \
   -p 8080:80 \
   --volumes-from pmm-data \
   --name pmm-server \
   -e ORCHESTRATOR_ENABLED=true \
   -e DISABLE_TELEMETRY=true \
   --storage-opt size=30G \
   --restart always \
   percona/pmm-server:1.17.3

docker run的额外功能：

--storage-opt size=30G，默认10G，避免不够用再扩容

添加 “-e ORCHESTRATOR_ENABLED=true \”，启用Orchestrator

添加 “-e DISABLE_TELEMETRY=true \”，关闭自动上报部分用户数据到percona.com的功能

进入容器：docker exec -i -t pmm-server /bin/bash

修改时区：ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

退出容器，执行：docker restart pmm-server

 

二、客户端

1、安装

Download Percona Monitoring and Management

yum install pmm-client-1.17.3-1.el7.x86_64.rpm

 

2、配置

2.1、server信息

pmm-admin config --server 192.168.237.9:8080
OK, PMM server is alive.

server端信息将被记录进：/usr/local/percona/pmm-client/pmm.yml

2.2、按需开放如下端口

42000：基础系统监控

42002：MySQL监控

42003：MongoDB

42004：ProxySQL

 

3、配置监控项

3.1、添加监控账号：

GRANT SELECT, PROCESS, SUPER, REPLICATION CLIENT, RELOAD ON *.* TO 'pmm'@' 127.0.0.1' IDENTIFIED BY 'pass' WITH MAX_USER_CONNECTIONS 10;
GRANT SELECT, UPDATE, DELETE, DROP ON performance_schema.* TO 'pmm'@'127.0.0.1';

3.2、添加监控项：

[root@ceshi23710 tmp]# pmm-admin add mysql --host 127.0.0.1 --port 3306 --user pmm --password pass node1
[linux:metrics] OK, already monitoring this system.
[mysql:metrics] OK, now monitoring MySQL metrics using DSN pmm:***@tcp(127.0.0.1:3306)
[mysql:queries] OK, now monitoring MySQL queries from slowlog using DSN pmm:***@tcp(127.0.0.1:3306)

为了避免密码泄露：

（1）可以将上述命令放到shell脚本中执行：

#!/bin/bash
pmm-admin add mysql XXXX

（2）pmm-admin之后，执行history -c清空历史信息

查看本机已有监控项：pmm-admin list

[root@ceshi23710 tmp]# pmm-admin list
pmm-admin 1.17.3

PMM Server      | 192.168.237.9:8080 
Client Name     | ceshi23710
Client Address  | 192.168.237.10 
Service Manager | linux-systemd

-------------- ----------- ----------- -------- ---------------------------- --------------------------------------------------------------------------------------
SERVICE TYPE   NAME        LOCAL PORT  RUNNING  DATA SOURCE                  OPTIONS                                                                               
-------------- ----------- ----------- -------- ---------------------------- --------------------------------------------------------------------------------------
mysql:queries  node2       -           YES      pmm:***@tcp(127.0.0.1:3306)  query_source=slowlog, query_examples=true, slow_log_rotation=true, retain_slow_logs=1 
linux:metrics  ceshi23710  42000       YES      -                                                                                                                  
mysql:metrics  node2       42002       YES      pmm:***@tcp(127.0.0.1:3306)                                                                         

 

三、QAN功能

待更新

 

四、容器的热扩容

1、查看容器卷信息

获取容器ID
# docker ps -a
CONTAINER ID        IMAGE                       COMMAND                CREATED             STATUS              PORTS                           NAMES
776a56114f35        percona/pmm-server:1.17.3   "/opt/entrypoint.sh"   36 minutes ago      Up 35 minutes       443/tcp, 0.0.0.0:8080->80/tcp   pmm-server
f50034bbf093        percona/pmm-server:1.17.3   "/bin/true"            39 minutes ago      Created                                             pmm-data

根据容器ID提取卷id
# docker inspect -f '{{ .GraphDriver.Data.DeviceName }}' pmm-server
docker-253:2-272456627-e621a70d0729720b457978bded0819484fb0aa280405bb92e8f2e017d6063596

根据卷id提取thin信息
# dmsetup table
docker-253:2-272456627-e621a70d0729720b457978bded0819484fb0aa280405bb92e8f2e017d6063596: 0 62914560 thin 253:3 59
centos-home: 0 401743872 linear 8:17 2048
docker-253:2-272456627-pool: 0 209715200 thin-pool 7:1 7:0 128 32768 1 skip_block_zeroing 
centos-swap: 0 16515072 linear 8:2 2048
centos-root: 0 87310336 linear 8:2 16517120
centos-root: 87310336 17547264 linear 8:17 401745920

2、原30G调整为40G

计算公式如下：

echo $((40*1024*1024*1024/512))
83886080

3、根据1和2拼接出操作命令

# echo 0 83886080 thin 253:3 59 | dmsetup load docker-253:2-272456627-e621a70d0729720b457978bded0819484fb0aa280405bb92e8f2e017d6063596

# dmsetup resume docker-253:2-272456627-e621a70d0729720b457978bded0819484fb0aa280405bb92e8f2e017d6063596

XFS文件系统格式：xfs_growfs，ext3、ext4文件系统格式：resize2fs
# xfs_growfs /dev/mapper/docker-253:2-272456627-e621a70d0729720b457978bded0819484fb0aa280405bb92e8f2e017d6063596
meta-data=/dev/mapper/docker-253:2-272456627-e621a70d0729720b457978bded0819484fb0aa280405bb92e8f2e017d6063596 isize=256    agcount=49, agsize=163824 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=7864320, imaxpct=25
         =                       sunit=16     swidth=16 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 7864320 to 10485760

4、验证

进入容器，查看磁盘空间：

docker exec -i -t pmm-server /bin/bash

df -h

 

5、启动报错的处理

[root@test02 docker]# docker start pmm-server
Error response from daemon: Failed to mount; dmesg: <4>[32087502.063016] XFS (dm-4): last sector read failed
<6>[32087508.595214] attempt to access beyond end of device
<6>[32087508.595223] dm-4: rw=32, want=83886080, limit=62914560
<4>[32087508.595231] XFS (dm-4): last sector read failed
: mount /dev/mapper/docker-253:2-272456627-e621a70d0729720b457978bded0819484fb0aa280405bb92e8f2e017d6063596:/home/docker/devicemapper/mnt/e621a70d0729720b457978bded0819484fb0aa280405bb92e8f2e017d6063596, data: nouuid: input/output error
Error: failed to start containers: pmm-server
[root@test02 docker]# 
[root@test02 docker]# echo 0 83886080 thin 253:3 59 | dmsetup load docker-253:2-272456627-e621a70d0729720b457978bded0819484fb0aa280405bb92e8f2e017d6063596
[root@test02 docker]# dmsetup resume docker-253:2-272456627-e621a70d0729720b457978bded0819484fb0aa280405bb92e8f2e017d6063596
[root@test02 docker]# 
[root@test02 docker]# docker start pmm-server
pmm-server

 

6、一键扩容脚本

#!/bin/bash

CID=$1
SIZE=$2

if [ "$CID" != "" ] && [ "$SIZE" != "" ]; then
    DEV=$(basename $(echo /dev/mapper/docker-*-$CID));
    dmsetup table $DEV | sed "s/0 [0-9]* thin/0 $(($SIZE*1024*1024*1024/512)) thin/" | dmsetup load $DEV;
    dmsetup resume $DEV;
    xfs_growfs /dev/mapper/$DEV;
  echo "Resize $CID completed."
else
    echo "Usage: sh resize_container 459fd505311ad364309940ac24dcdb2bdfc68e3c3b0f291c9153fb54fbd46771 100";
fi

使用方法：

step1、根据容器ID提取卷id

# docker inspect -f '{{ .GraphDriver.Data.DeviceName }}' pmm-server
docker-253:2-272456627-e621a70d0729720b457978bded0819484fb0aa280405bb92e8f2e017d6063596

step2、扩容，45G为例

./resize_container e621a70d0729720b457978bded0819484fb0aa280405bb92e8f2e017d6063596 45

注：

1、重启后同样需要再次运行该脚本

2、每次调整的大小不能小于上次，否则报错：xfs_growfs: /dev/mapper/docker-XXX is not a mounted XFS filesystem

 

参考文档：

dockerhub: percona/pmm-server

两种方式迁移 Docker 的默认安装 (存储) 目录

Percona Monitoring and Management Documentation: Setting Up a Docker Container for PMM Server

Percona Monitoring and Management Documentation: Creating a MySQL User Account to Be Used with PMM

PMM监控MySQL

CentOS7修改时区的正确姿势

Understanding PMM QAN – Graphs & Metrics

Docker 环境 Storage Pool 用完解决方案：resize-device-mapper