metrics和graphite监控

简介

  Graphite 是一个数据绘图工具，安装完成之后需要喂它数据，就可以根据你的需要将数据生成制定的格式（绘图、json、xml等），还可以和Nagios结合完成报警功能。具体的数据采集工具是由第三方工具完成。
  Graphite 支持的第三方工具很多，Bucky, collectd, Collectl, Charcoal, Diamond, Ganglia, Host sFlow, statsd, Tasseo 等等。本文第三方工具使用的elasticsearch的filter/metrics插件。

graphite安装

服务器环境为10.10.10.60 centos7
1.安装epel源并安装相关开发包

rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum -y install httpd gcc gcc-c++ git pycairo mod_wsgi
yum -y install python-pip python-devel blas-devel lapack-devel libffi-devel

2.下载并安装graphite

cd /usr/local/src/graphite
git clone https://github.com/graphite-project/graphite-web.git
git clone https://github.com/graphite-project/carbon.git
#检测安装依赖
pip install -r /usr/local/src/graphite/graphite-web/requirements.txt

若当前安装环境缺少相关组件，则会提示。例如我本机提示缺少django-tagging django等，我们可以通过pip安装：

pip install django-tagging
pip install 'Django==1.8'

注意：安装python相关组件要根据python版本进行，我本机为Python 2.7.5，但是安装django1.9和1.7后graphite启动报错，后来改成1.8才正常。

#安装carbon和graphite-web
cd /usr/local/src/graphite/carbon/
python setup.py install

cd /usr/local/src/graphite/graphite-web/
python setup.py install
#默认安装到/opt/graphite/

3.生成配置文件

 cp /opt/graphite/conf/carbon.conf.example /opt/graphite/conf/carbon.conf
 cp /opt/graphite/conf/storage-schemas.conf.example /opt/graphite/conf/storage-schemas.conf
 cp /opt/graphite/conf/storage-aggregation.conf.example /opt/graphite/conf/storage-aggregation.conf
 cp /opt/graphite/conf/relay-rules.conf.example /opt/graphite/conf/relay-rules.conf
 cp /opt/graphite/webapp/graphite/local_settings.py.example /opt/graphite/webapp/graphite/local_settings.py
 cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi
 #生成apache配置文件
 cp /opt/graphite/examples/example-graphite-vhost.conf /etc/httpd/conf.d/graphite.conf
 #配置启动脚本
 cp /usr/local/src/graphite/carbon/distro/redhat/init.d/carbon-* /etc/init.d/
 chmod +x /etc/init.d/carbon-*

4.配置graphite
graphite默认使用sqlit作为本地数据库，当然也可以使用mysql数据库。我们需要为graphite-webapp导入静态文件并且设置相关权限

cd /opt/graphite
PYTHONPATH=/opt/graphite/webapp/ django-admin.py syncdb --settings=graphite.settings
PYTHONPATH=/opt/graphite/webapp/ django-admin.py collectstatic --settings=graphite.settings

**注意：**django1.9不支持syncdb参数，需要使用migrate参数

chown -R apache:apache /opt/graphite/storage/
chown -R apache:apache /opt/graphite/static/
chown -R apache:apache /opt/graphite/webapp/

5.配置apache

<IfModule !wsgi_module.c>
    LoadModule wsgi_module modules/mod_wsgi.so
</IfModule>

# XXX You need to set this up!
# Read http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives#WSGISocketPrefix
# For example, create a directory /var/run/wsgi and use that.
WSGISocketPrefix run/wsgi

<VirtualHost *:80>
        ServerName graphite
        DocumentRoot "/opt/graphite/webapp"
        ErrorLog /opt/graphite/storage/log/webapp/error.log
        CustomLog /opt/graphite/storage/log/webapp/access.log common

        # I've found that an equal number of processes & threads tends
        # to show the best performance for Graphite (ymmv).
        WSGIDaemonProcess graphite processes=5 threads=5 display-name='%{GROUP}' inactivity-timeout=120
        WSGIProcessGroup graphite
        WSGIApplicationGroup %{GLOBAL}
        WSGIImportScript /opt/graphite/conf/graphite.wsgi process-group=graphite application-group=%{GLOBAL}

        # XXX You will need to create this file! There is a graphite.wsgi.example
        # file in this directory that you can safely use, just copy it to graphite.wgsi
        WSGIScriptAlias / /opt/graphite/conf/graphite.wsgi


        # XXX To serve static files, either:
        # * Install the whitenoise Python package (pip install whitenoise)
        # * Collect static files in a directory by running:
        #     django-admin.py collectstatic --noinput --settings=graphite.settings
        #   And set an alias to serve static files with Apache:
        Alias /static/ /opt/graphite/static/
        <Directory /opt/graphite/static>
        #       Order allow,deny
        #       Allow from all
                Require all granted
        </Directory>

        ########################
        # URL-prefixed install #
        ########################
        # If using URL_PREFIX in local_settings for URL-prefixed install (that is not located at "/"))
        # your WSGIScriptAlias line should look like the following (e.g. URL_PREFX="/graphite"

        # WSGIScriptAlias /graphite /srv/graphite-web/conf/graphite.wsgi/graphite
        # Alias /graphite/static /opt/graphite/webapp/content
        #  <Location "/graphite/static/">
        #        SetHandler None
        # </Location>


        # XXX In order for the django admin site media to work you
        # must change @DJANGO_ROOT@ to be the path to your django
        # installation, which is probably something like:
        # /usr/lib/python2.6/site-packages/django
        Alias /media/ "@DJANGO_ROOT@/contrib/admin/media/"

        # The graphite.wsgi file has to be accessible by apache. It won't
        # be visible to clients because of the DocumentRoot though.
        <Directory /opt/graphite/conf/>
                <IfVersion < 2.4>
                        Order deny,allow
                        Allow from all
                </IfVersion>
                <IfVersion >= 2.4>
                        Require all granted
                </IfVersion>
        </Directory>

</VirtualHost>

注意：
(1).默认情况下，apache访问限制Alias目录，因此我们需要在配置文件中对static目录进行更改：

        Alias /static/ /opt/graphite/static/
        <Directory /opt/graphite/static>
                Require all granted
        </Directory>

(2).apache版本不同，授权方式也不同，例如：

        Alias /static/ /opt/graphite/static/
        <Directory /opt/graphite/static>
                Order allow,deny
                Allow from all
                Require all granted
                Require all granted
        </Directory>

(3).若登录后发现css，js等静态内容无法显示，或日志返回403错误，则需要注意“Alias /static/ /opt/graphite/static/”最后的/是否存在。

6.启动graphite

service start carbon-cache
systemctl start httpd

7.登录10.10.10.60

由于没有数据输入，因此没有数据显示

实验

需求：在此我们通过ELKstack的logstash组件中的filter/metrics来采集nginx的日志，来检测访问日志中503每分钟的频率，若频率过高则表示nginx有异常访问。

1.编辑logstash配置文件

cd /usr/local/logstash/etc
vim reponse_503.conf

input {
        file {
                path => "/data/nginx/logs/access.log"
                type => "nginx-access"
                start_position => "beginning"
                sincedb_path => "/usr/local/logstash/sincedb"
        }
}
filter {
        if [type] == "nginx-access" {
                grok {
                        patterns_dir => "/usr/local/logstash/patterns"
                        match => {
                                "message" => "%{NGINXACCESS}"
                        }
                }
                date {
                        match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
                        }
                metrics {
                        meter => [ "http_%{status}" ]
                        add_tag => "metric"
                }
        }
}
output {
        if "metric" in [tags] {
                graphite {
                        host => "10.10.10.60"
                        metrics => [ "218-http_503.rate_1m", "%{[http_503][rate_1m]}" ]
                }
        }

}

2.运行logstash

/usr/local/logstash/bin/logstash -f reponse_503.conf
Default settings used: Filter workers: 4
Logstash startup completed

3.登录graphite就可以看出相关的数据曲线了

总结

  由于nagios不支持filter/metrics，因此我们在提供graphite进行画图。当然ELKstack支持很多组件，如果我们日常使用的nagios监控，我们也可以通过使用其他组件来配合nagios使用。