Zabbix Server+Grafana安装手册

Zabbix Server+Grafana安装手册

Zabbix Server架构

Frontend：Web UI

DB：zabbix数据存储

Zabbix Server：数据搜集、计算触发器、发通知等

Zabbix Server安装步骤

Step 1: Install Zabbix server, frontend, and agent

wget https://repo.zabbix.com/zabbix/5.0/ubuntu/pool/main/z/zabbix-release/zabbix-release_5.0-1+$(lsb_release -sc)_all.deb
sudo dpkg -i zabbix-release_5.0-1+$(lsb_release -sc)_all.deb
sudo apt update
sudo apt -y install zabbix-server-mysql zabbix-frontend-php zabbix-apache-conf zabbix-agent

Step 2: Configure database

In this installation, I will use password rootDBpass as root password and zabbixDBpass as Zabbix password for DB. Consider changing your password for security reasons.

a. Reset root password for database

Secure MySQL/MariaDB by changing the default password for MySQL root:

sudo mysql_secure_installation

Enter current password for root (enter for none): Press the Enter
Set root password? [Y/n]: Y
New password: <Enter root DB password>
Re-enter new password: <Repeat root DB password>
Remove anonymous users? [Y/n]: Y
Disallow root login remotely? [Y/n]: Y
Remove test database and access to it? [Y/n]:  Y
Reload privilege tables now? [Y/n]:  Y

If you receive the error “mysql_secure_installation: command not found” then you need to install MariaDB using this command: “sudo apt -y install mariadb-common mariadb-server mariadb-client” and if that commands fails because of unmet dependencies, then install it like this “sudo apt -y install mariadb-common mariadb-server-10.3 mariadb-client-10.3“. Run “sudo mysql_secure_installation” command again after installing the database.

b. Create database

sudo mysql -uroot -p'rootDBpass' -e "create database zabbix character set utf8 collate utf8_bin;"
sudo mysql -uroot -p'rootDBpass' -e "grant all privileges on zabbix.* to zabbix@localhost identified by 'zabbixDBpass';"

c. Import initial schema and data

Temporary disable strict mode (ZBX-16465) to avoid MySQL error “ERROR 1118 (42000) at line 1284: Row size too large (> 8126)” :

sudo mysql -uroot -p'rootDBpass' zabbix -e "set global innodb_strict_mode='OFF';"

Import database shema for Zabbix server (could last up to 5 minutes):

zcat /usr/share/doc/zabbix-server-mysql*/create.sql.gz | mysql -uzabbix -p'zabbixDBpass' zabbix

Enable strict mode:

sudo mysql -uroot -p'rootDBpass' zabbix -e "set global innodb_strict_mode='ON';"

d. Enter database password in Zabbix configuration file

Open zabbix_server.conf file with command: “sudo vim /etc/zabbix/zabbix_server.conf” and add database password in this format anywhere in file:

DBPassword=zabbixDBpass

Step 3: Start Zabbix server and agent processes

sudo systemctl restart zabbix-server zabbix-agent 
sudo systemctl enable zabbix-server zabbix-agent

Step 4: Configure Zabbix frontend

a. Configure PHP for Zabbix frontend

Edit file /etc/zabbix/apache.conf:

sudo vim /etc/zabbix/apache.conf

Uncomment 2 lines in apache.conf that starts with “# php_value date.timezone Europe/Riga” by removing symbol # and set the right timezone for your country, for example:

php_value date.timezone Asia/Shanghai

b. Restart Apache web server and make it start at system boot

sudo systemctl restart apache2
sudo systemctl enable apache2

c. Configure web frontend

Connect to your newly installed Zabbix frontend using URL “http://server_ip_or_dns_name/zabbix” to initiate the Zabbix installation wizard.

In my case, that URL would be “http://192.168.1.161/zabbix” because I have installed Zabbix on the server with IP address 192.168.1.161 (you can find the IP address of your server by typing “ip a” command in the terminal).

Basically, in this wizard you only need to enter a password for Zabbix DB user and just click “Next step” for everything else. In this guide, I have used a zabbixDBpass as a database password, but if you set something else, be sure to enter the correct password when prompted by the wizard.

开始通过界面配置zabbix，登录密码是上面设置的数据库密码zabbixDBpass

1. Installation step: Welcome screen

2. Installation step: Pre-requisites check

3. Installation step: Configure DB connection

4. Installation step: Configure Zabbix server

5. Installation step: Pre-installation summary

6. Installation step: Finish

That’s it, you have installed Zabbix monitoring system!

Step 5: Login to frontend using Zabbix default login credentials

Use Zabbix default admin username “Admin” and password “zabbix” (without quotes) to login to Zabbix frontend at URL “http://server_ip_or_dns_name/zabbix” via your browser.

In my example, I have installed Zabbix on server 192.168.1.161 so I will enter in my browsers URL field http://192.168.1.161/zabbix (you can find the IP address of your server by typing “ip a” command in the terminal)

Step 6: Optimizing Zabbix Server (optional)

Don’t bother with this optimization if you are monitoring a small number of devices, but if you are planning to monitor a large number of devices then continue with this step.

Open “zabbix_server.conf” file with command: “sudo vim /etc/zabbix/zabbix_server.conf” and add this configuration anywhere in file:

StartPollers=100
StartPollersUnreachable=50
StartPingers=50
StartTrappers=10
StartDiscoverers=15
StartPreprocessors=15
StartHTTPPollers=5
StartAlerters=5
StartTimers=2
StartEscalators=2
CacheSize=128M
HistoryCacheSize=64M
HistoryIndexCacheSize=32M
TrendCacheSize=32M
ValueCacheSize=256M

Save and exit file

This is not a perfect configuration, keep in mind that you can optimize it even more. Let’s say if you don’t use ICMP checks then set the “StartPingers” parameter to 1 or if you don’t use active agents then set “StartTrappers” to 1 and so on. You can find out more about the parameters supported in a Zabbix server configuration file in the official documentation.

If you try to start the Zabbix server you will receive an error “[Z3001] connection to database 'Zabbix' failed: [1040] Too many connections” in the log “/var/log/zabbix/zabbix_server.log” because we are using more Zabbix server processes than MySQL can handle. We need to increase the maximum permitted number of simultaneous client connections and optimize MySQL – so move to the next step.

Step 7: Optimizing MySQL / MariaDB database (optional)

a. Create custom MySQL configuration file

Create file “10_my_tweaks.cnf" with “/etc/mysql/mariadb.conf.d/10_my_tweaks.cnf” and paste this configuration:

[mysqld]
max_connections                = 404
innodb_buffer_pool_size        = 800M

innodb-log-file-size           = 128M
innodb-log-buffer-size         = 128M
innodb-file-per-table          = 1
innodb_buffer_pool_instances   = 8
innodb_old_blocks_time         = 1000
innodb_stats_on_metadata       = off
innodb-flush-method            = O_DIRECT
innodb-log-files-in-group      = 2
innodb-flush-log-at-trx-commit = 2

tmp-table-size                 = 96M
max-heap-table-size            = 96M
open_files_limit               = 65535
max_connect_errors             = 1000000
connect_timeout                = 60
wait_timeout                   = 28800

Save and exit the file (ctrl+x, followed by y and enter) and set the correct file permission:

sudo chown mysql:mysql /etc/mysql/mariadb.conf.d/10_my_tweaks.cnf
sudo chmod 644 /etc/mysql/mariadb.conf.d/10_my_tweaks.cnf

Two things to remember!

Configuration parameter max_connections must be larger than the total number of all Zabbix proxy processes plus 150. You can use the command below to automatically check the number of Zabbix processes and add 150 to that number:

root@ubuntu:~ $ egrep "^Start.+=[0-9]"  /etc/zabbix/zabbix_server.conf  |  awk -F "=" '{s+=$2} END {print s+150}'
 404

The second most important parameter is innodb_buffer_pool_size, which determines how much memory can MySQL get for caching InnoDB tables and index data. You should set that parameter to 70% of system memory if only database is installed on server.

However, in this case, we are sharing a server with Zabbix and Apache processes so you should set innodb_buffer_pool_size to 40% of total system memory. That would be 800 MB because my Ubuntu server has 2 GB RAM.

I didn’t have any problems with memory, but if your Zabbix proxy crashes because of lack of memory, reduce “innodb_buffer_pool_size” and restart MySQL server.

Note that if you follow this configuration, you will receive “Too many processes on the Zabbix server” alarm in Zabbix frontend due to the new Zabbix configuration. It is safe to increase the trigger threshold or turn off that alarm (select “Problems” tab → left click on the alarm → select “Configuration” → remove the check from “Enabled” → hit the “Update” button)

b. Restart Zabbix Server and MySQL service

Stop and start the services in the same order as below:

sudo systemctl stop zabbix-server
sudo systemctl stop mysql
sudo systemctl start mysql
sudo systemctl start zabbix-server

Step 8: Create MySQL partitions on History and Events tables

Zabbix’s housekeeping process is responsible for deleting old trend and history data. Removing old data from the database using SQL delete query can negatively impact database performance. Many of us have received that annoying alarm “Zabbix housekeeper processes more than 75% busy” because of that.

That problem can be easily solved with the database partitioning. Partitioning creates tables for each hour or day and drops them when they are not needed anymore. SQL DROP is way more efficient than the DELETE statement.

You can partition MySQL tables in 5 minutes using this simple guide.

Step 9: How to manage Zabbix / MySQL / Apache service

Sometimes you will need to check or restart Zabbix, MySQL or Apache service – use commands below to do that.

Zabbix Server
sudo systemctl <status/restart/start/stop> zabbix-server

MySQL Server
sudo systemctl <status/restart/start/stop> mysql

Apache Server
sudo systemctl <status/restart/start/stop> apache2

Zabbix Agent
sudo systemctl <status/restart/start/stop> zabbix-agent 

Step 10: Upgrade between minor versions

I wrote about these upgrade procedures in my post about Zabbix upgrade. Zabbix’s team releases new minor versions at least once a month. The main purpose of minor upgrades is to fix bugs (hotfix) and sometimes even bring new functionality. Therefore, try to do a minor upgrade of Zabbix at least once a month.

There is no need for backups when doing a minor upgrade, they are completely safe. With this command you can easily upgrade smaller versions of 5.0.x (for example, from 5.0.1 to 5.0.3):

sudo apt install --only-upgrade 'zabbix.*'

And restart Zabbix server afterward:

sudo systemctl restart zabbix-server

解决zabbix图表中文乱码的问题

上传字体文件到/usr/share/zabbix/assets/fonts和/usr/share/zabbix/fonts/

vim /usr/share/zabbix/include/defines.inc.php
# 将关键字 “graphfont” 改为“simkai”
sudo service zabbix-server restart
sudo service apache2 restart

Zabbix Agent安装(被动方式)

只有Zabbix Server和被监控的服务器（agent端）网络互通才能被监控到，否则无法监控

# 1、创建zabbix组和zabbix用户
sudo groupadd zabbix
sudo useradd -g zabbix zabbix -s /sbin/nologin
sudo useradd zabbix

# 2、安装可执行文件
# 创建日志目录
sudo mkdir -p /usr/local/zabbix_agent/log
# 将可执行文件解压到安装目录
sudo tar -xvf zabbix_agent-5.0.3-linux-3.0-amd64-static.tar.gz -C /usr/local/zabbix_agent
# 修改目录权限，以免启动时报权限问题
sudo chown -R zabbix:zabbix   /usr/local/zabbix_agent
cd /usr/local/zabbix_agent
# 将可执行文件拷贝到系统bin和sbin目录
sudo cp bin/zabbix_*  /usr/bin/
sudo cp sbin/zabbix*  /usr/sbin/
# 默认请将配置文件必须放在这个目录下
sudo mkdir -p /usr/local/etc
sudo cp /usr/local/zabbix_agent/conf/zabbix_agentd.conf  /usr/local/etc
sudo vim /usr/local/etc/zabbix_agentd.conf
# LogFile=/usr/local/zabbix_agent/log/zabbix_agentd.log
# Server=192.168.33.12
# #ServerActive=127.0.0.1
# #Hostname=Zabbix server

# 启动
sudo zabbix_agentd

Grafana安装

按照官网命令安装即可：https://grafana.com/grafana/download

sudo apt-get install -y adduser libfontconfig1
wget https://dl.grafana.com/oss/release/grafana_7.2.0_amd64.deb
sudo dpkg -i grafana_7.2.0_amd64.deb

修改grafana使用mysql作为数据持久化

在此之前要把对象的数据库创建好

sudo vim /etc/grafana/grafana.ini
#################################### Database ####################################
[database]
# You can configure the database connection by specifying type, host, name, user and password
# as separate properties or as on string using the url properties.

# Either "mysql", "postgres" or "sqlite3", it's your choice
#type = mysql
#host = 127.0.0.1:3306
#name = grafana
#user = grafana
# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
#password = grafana

# Use either URL or the previous fields to configure the database
# Example: mysql://user:secret@host:port/database
url = mysql://grafana:grafana@127.0.0.1:3306/grafana

sudo vim /usr/share/grafana/conf/defaults.ini
#################################### Database ############################
[database]
# You can configure the database connection by specifying type, host, name, user and password
# as separate properties or as on string using the url property.

# Either "mysql", "postgres" or "sqlite3", it's your choice
#type = sqlite3
#host = 127.0.0.1:3306
#name = grafana
#user = root
# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
#password =
# Use either URL or the previous fields to configure the database
# Example: mysql://user:secret@host:port/database
url = mysql://grafana:grafana@127.0.0.1:3306/grafana

安装zabbix插件

grafana-cli plugins install alexanderzobnin-zabbix-app
sudo vim /usr/share/grafana/conf/defaults.ini

# 解决在grafana数据源中找不到zabbix的问题
allow_loading_unsigned_plugins = alexanderzobnin-zabbix-datasource
sudo service grafana-server start

附录

zabbix宏

自定义邮件内容会用到

https://www.zabbix.com/documentation/4.0/zh/manual/appendix/macros/supported_by_location

https://www.zabbix.com/documentation/current/manual/appendix/macros/supported_by_location

示例：

主题：Problem: {EVENT.NAME}故障{TRIGGER.STATUS},服务器:{HOSTNAME1}发生: {TRIGGER.NAME}故障!
内容：
告警主机: {HOSTNAME1}
告警时间: {EVENT.DATE} {EVENT.TIME}
告警等级: {TRIGGER.SEVERITY}
告警信息: {TRIGGER.NAME}
告警项目: {TRIGGER.KEY1}
问题详情: {ITEM.NAME}:{ITEM.VALUE}
当前状态: {TRIGGER.STATUS}:{ITEM.VALUE1}
事件ID: {EVENT.ID}


恢复主题：
恢复{TRIGGER.STATUS}, 服务器:{HOSTNAME1}: {TRIGGER.NAME}已恢复!
恢复信息：
告警主机: {HOSTNAME1}
告警时间: {EVENT.DATE} {EVENT.TIME}
告警等级: {TRIGGER.SEVERITY}
告警信息: {TRIGGER.NAME}
告警项目: {TRIGGER.KEY1}
问题详情: {ITEM.NAME}:{ITEM.VALUE}
当前状态: {TRIGGER.STATUS}:{ITEM.VALUE1}
事件ID: {EVENT.ID}