FIRST TOUCH - 20170223 - Collectd Graphite Grafana On Centos / Ubuntu

Collectd

apt-get install collectd

Graphite

Graphite 和 Grafana 简介

Graphite 是一款开源的监控绘图工具。

Graphite 可以实时收集、存储、显示时间序列类型的数据（time series data）。它主要有三个部分构成：
- carbon —— 基于 Twisted 的进程，用来接收数据；
- whisper —— 专门存储时间序列类型数据的小型数据库；
- graphite webapp —— 基于 Django 的网页应用程序。

Graphite详解

优点

1. Graphite非常快，它的架构是模块化可规模化的
2. Graphite很著名，有庞大的社区和广泛的支持
3. 有很多与Graphite相互协作的开源工具
4. Graphite完成单个工作且做得很好
5. Graphite采用Apache 2.0许可证

缺点

1. Graphite不能对数据进行分片，因此要解决这个问题就是采用多个Graphite实例
2. Graphite的安装是一个很复杂的任务，尽管目前有了完整的Docker映像可以一次性安装Graphite及其依赖

试错 Test Process 【瞎扯淡】

1. ssh: connect to host localhost port 22: Connection refused
   sudo apt-get install openssh-server
2. pip: command not found
   sudo apt-get install python-pip
3. command ‘x86_64-linux-gnu-gcc’ failed with exit status 1
   sudo apth-get install python-dev
4. Ubuntu 14.04 安装图形监控工具Graphite
   sudo pip install whisper; sudo pip install carbon; sudo pip install graphite-web

5. Graphite 学习文档
   Graphite的默认安装位置是/opt/graphite/。

   目录	介绍
   /opt/graphite/bin	二进制文件目录
   /opt/graphite/conf	配置文件目录
   /opt/graphite/lib	库目录
   /opt/graphite/storage	数据存放目录，包括log，whisper数据库，索引，rrd数据等
   /opt/graphite/webapp	webapp文件存放目录
   /opt/graphite/examples	获取数据的例子

6. DevOps实战：Graphite监控上手指南
   

   • Carbon【/opt/graphite/bin/ 目录下】
     
     • Cache：接受通过各种协议传输来的指标项数据并以尽可能高的效率将它们写入磁盘；在接收到指标项时，将指标项值缓存在RAM中，并用底层的Whisper库按照指定的时间间隔将这些值写入磁盘。
     • Relay：有两个不同的用途：将输入的指标项复制并分区。
     • Aggregator：运行于cache前方，在Whisper中记录指标项之前，缓存这些指标项一段时间。
       

       cd /opt/graphite/conf
       sudo cp aggregation-rules.conf.example aggregation-rules.conf
       sudo cp blacklist.conf.example blacklist.conf
       sudo cp carbon.conf.example carbon.conf
       sudo cp carbon.amqp.conf.example carbon.amqp.conf
       sudo cp relay-rules.conf.example relay-rules.conf
       sudo cp rewrite-rules.conf.example rewrite-rules.conf
       sudo cp storage-schemas.conf.example storage-schemas.conf
       sudo cp storage-aggregation.conf.example storage-aggregation.conf
       sudo cp whitelist.conf.example whitelist.conf
       
       vi carbon.conf
           [cache]
           LINE_RECEIVER_INTERFACE = 0.0.0.0
           LINE_RECEIVER_PORT = 2003

7. 出现以下报错：

   Traceback (most recent call last):
     File "./carbon-cache.py", line 32, in <module>
       run_twistd_plugin(__file__)
       ...
   AttributeError: 'module' object has no attribute 'OP_NO_TLSv1_1'

   参考 1

   安装 twisted：
   sudo pip install twisted

   参考 2

   在 CentOS 5.8 上安装 Graphite
   解决方法是修改 /opt/graphite/lib/carbon/util.py 文件，把其 from twisted.scripts._twistd_unix import daemonize 替换成 import daemonize 【可雷同】

   参考 3

   sudo apt-get install libssl-dev

   参考 4

   删掉含有 OP_NO_TLSv1_1 ／ OP_NO_TLSv1_2 的两行代码

   cd /opt/graphite/bin
   ./carbon-cache.py start
   
   
   # 运行结果看起来不太正常呀
   
   $ :0: UserWarning: You do not have a working installation of the service_identity module: 'No module named service_identity'.  Please install it from <https://pypi.python.org/pypi/service_identity> and make sure all of its dependencies are satisfied.  Without the service_identity module, Twisted can perform only rudimentary TLS client hostname verification.  Many valid certificate/hostname mappings may be rejected.
   Starting carbon-cache (instance a)
   
   ps -efla | grep carbon-cache
   
   # 运行结果
   
   $ 1 S root      18724      1  0  80   0 - 38876 ep_pol 13:21 ?        00:00:00 /usr/bin/python ./carbon-cache.py start
     0 S cgg       18735  16000  0  80   0 -  3978 pipe_w 13:40 pts/4    00:00:00 grep --color=auto carbon-cache
   
   netstat -nap |grep 2003
   
   # 运行结果大概没问题
   
   $ (Not all processes could be identified, non-owned process info
    will not be shown, you would have to be root to see it all.)
     tcp        0      0 0.0.0.0:2003            0.0.0.0:*               LISTEN      -

   • Whisper

     • 控制Whisper文件如何创建的配置文件如下：

       • /opt/graphite/conf/storage-schemas.conf
       • /opt/graphite/conf/storage-aggregation.conf

     • 存储模式

       [carbon]
       pattern = ^carbon\.
       retentions = 1min:180d,10min:180d
       
       [production_staging]
       pattern = ^(PRODUCTION|STAGING).*
       retentions = 10s:3d,1min:180d,10min:180d

     • 存储聚合

       [ratio]
       pattern = \.ratio$
       xFilesFactor = 0.1
       aggregationMethod = average
       
       [m1_rate]
       pattern = \.m1_rate$
       xFilesFactor = 0.1
       aggregationMethod = sum

   • Graphite Web

     参考 1

     Django 环境搭建

     sudo pip install Django

     sudo pip install --upgrade pip

     参考 2

     Configuration problems with django and mod_wsgi

     sudo pip install django-tagging

     参考 3

     python manage.py syncdb Unknown command: ‘syncdb’问题解决方法

     cd /opt/graphite

     export PYTHONPATH=$PYTHONPATH:pwd/webapp

     django-admin.py migrate –settings=graphite.settings

     参考 4

     from django.conf.urls import *找不到patterns模块？ Django-1.10，python27

完整过程 【Real Use】

Ubuntu

// [在 Ubuntu 12.04 上安装 Graphite 监控工具](http://www.vpsee.com/2012/05/install-graphite-on-ubuntu-12-04/)
sudo apt-get install collectd

sudo apt-get install apache2 libapache2-mod-wsgi python-django \
python-twisted python-cairo python-pip python-django-tagging

sudo pip install whisper
sudo pip install carbon
sudo pip install graphite-web

cd /opt/graphite/conf/
sudo cp carbon.conf.example carbon.conf
sudo cp storage-schemas.conf.example storage-schemas.conf
sudo cp graphite.wsgi.example graphite.wsgi

cd /opt/graphite/webapp/graphite/
sudo python manage.py syncdb
sudo chown -R www-data:www-data /opt/graphite/storage/
sudo cp local_settings.py.example local_settings.py

Centos

// [Install Graphite In CentOS 7](http://www.tuicool.com/articles/3uAney)
yum -y update
yum install collectd collectd-snmp
yum install -y httpd net-snmp perl pycairo mod_wsgi python-devel git gcc-c++
yum install -y python-pip

pip install 'django<1.6'
pip install 'Twisted<12'
// [graphite/carbon ImportError: No module named fields](http://stackoverflow.com/questions/31363276/graphite-carbon-importerror-no-module-named-fields)
pip install 'django-tagging<0.4'
pip install whisper
pip install graphite-web
pip install carbon

sudo cp /opt/graphite/examples/example-graphite-vhost.conf /etc/httpd/conf.d/graphite.conf

cd /opt/graphite/conf/
sudo cp carbon.conf.example carbon.conf
sudo cp storage-schemas.conf.example storage-schemas.conf
sudo cp graphite.wsgi.example graphite.wsgi

cd /opt/graphite/webapp/graphite/
sudo python manage.py syncdb
sudo chown -R apache:apache /opt/graphite/storage/
sudo cp local_settings.py.example local_settings.py

最后两者相同部分

# [DevOps实战：Graphite监控上手指南](http://www.infoq.com/cn/articles/graphite-intro)
vi /opt/graphite/webapp/graphite/local_settings.py
#########################
# General Configuration #
#########################
TIME_ZONE = 'UTC'
##########################
# Database Configuration #
##########################
DATABASES = {
    'default': {
        'NAME': '/opt/graphite/storage/graphite.db',
        'ENGINE': 'django.db.backends.sqlite3',
        'USER': '',
        'PASSWORD': '',
        'HOST': '',
        'PORT': ''
    }
}

cd /opt/graphite
PYTHONPATH=`pwd`/storage/whisper ./bin/run-graphite-devel-server.py --port=8085 --libs=`pwd`/webapp /opt/graphite 1>/opt/graphite/storage/log/webapp/process.log 2>&1 &
tail -f /opt/graphite/storage/log/webapp/process.log

cd /opt/graphite/
sudo ./bin/carbon-cache.py start

# [Ubuntu Server 14.04 x64安装Grafana](http://www.linuxdiyf.com/linux/21696.html)
cd /tmp
wget https://grafanarel.s3.amazonaws.com/builds/grafana-4.1.2-1486989747.linux-x64.tar.gz
tar -zxvf grafana-4.1.2-1486989747.linux-x64.tar.gz

'''
# 这部分没做，至今没搞懂参考内容这段的意义，remain to be thinking hard
# [安装 Grafana](http://www.vpsee.com/?s=grafana)
cd grafana-4.1.2-1486989747
cp config.sample.js config.js
vi config.js
...
      // InfluxDB example setup (the InfluxDB databases specified need to exist)
      datasources: {
        influxdb: {
          type: 'influxdb',
          url: "http://192.168.2.183:8086/db/collectd",
          username: 'root',
          password: 'root',
        },
        ...
      },
...
'''

/tmp/grafana-4.1.2-1486989747/bin/grafana-server start 1>/tmp/grafana-4.1.2-1486989747/grafana.log 2>&1 &