OpenTSDB的基本概念
OpenTSDB定义每个时间序列数据需要包含以下属性：
1. 指标名称（metric name）
2. 时间戳（UNIX timestamp，毫秒或者秒精度）
3. 值（64位整数或者单精度浮点数）
4. 一组标签（tags，用于描述数据属性，至少包含一个或多个标签，每个标签由tagKey和tagValue组成，tagKey和tagValue均为字符串）
1. 安装环境
1.1. 系统版本
lsb_release -a LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch Distributor ID: CentOS Description: CentOS release 6.4 (Final) Release: 6.4 Codename: Final |
1.2. 内核版本
uname -a Linux localhost.localdomain 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux |
1.3. 依赖安装
1.3.1. java安装
1.yum install java-1.6.0-openjdk-devel.x86_64
2. java环境变量设置
在~/.bash_profile文件中添加以下行。
vim ~/.bash_profile # java export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/ export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar |
1.3.2.gnuplot安装
yum install libotf yum install gnuplot-common.x86_64 yum install gnuplot.x86_64 yum install libtool |
2. hbase 安装
2.1. 源码下载
wget http://mirror.bit.edu.cn/apache/hbase/hbase-0.94.27/hbase-0.94.27.tar.gz
2.2. 配置修改
1.修改hbase-env.sh中的如下属性
export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/
export HBASE_MANAGES_ZK=true
2.单机配置修改hbase-site.xml：
<configuration>
<property>
<name>hbase.rootdir</name>
<value>/root/guochunyang/hbase_fs/</value>
</property>
</configuration>
2.3. 启动hbase
./bin/start-hbase.sh
4. 关闭hbase
./bin/stop-hbase.sh
2.4. habse命令
a.进入hbase console
bin/hbase shell |
b.查看所有表
hbase(main):009:0> list TABLE tsdb tsdb-meta tsdb-tree |
tsdb-meta是元数据表
这是用来存储时间序列索引和元数据的表。这也是一个可选特性，默认是不开启的，可以通过配置文件来启用该特性
具体操作命令可以参考：http://www.cnblogs.com/heyCoding/archive/2012/11/09/2762334.html
3. opentsdb 安装
1. 下载源码
git clone https://github.com/OpenTSDB/opentsdb.git
2.编译
./build.sh
3. 建表
env COMPRESSION=NONE HBASE_HOME=../hbase-0.94.27 src/create_table.sh
4. 配置文件 opentsdb.conf
[root@10.77.51.10 opentsdb]# cat /etc/opentsdb.conf
# --------- NETWORK ----------
# The TCP port TSD should use for communications
# *** REQUIRED ***
tsd.network.port = 4242
# The IPv4 network address to bind to, defaults to all addresses
# tsd.network.bind = 0.0.0.0
# Enables Nagel's algorithm to reduce the number of packets sent over the
# network, default is True
#tsd.network.tcpnodelay = true
# Determines whether or not to send keepalive packets to peers, default
# is True
#tsd.network.keepalive = true
# Determines if the same socket should be used for new connections, default
# is True
#tsd.network.reuseaddress = true
# Number of worker threads dedicated to Netty, defaults to # of CPUs * 2
#tsd.network.worker_threads = 8
# Whether or not to use NIO or tradditional blocking IO, defaults to True
#tsd.network.async_io = true
# ----------- HTTP -----------
# The location of static files for the HTTP GUI interface.
# *** REQUIRED ***
tsd.http.staticroot = build/staticroot
# Where TSD should write it's cache files to
# *** REQUIRED ***
tsd.http.cachedir = /tmp/tsd
# --------- CORE ----------
# Whether or not to automatically create UIDs for new metric types, default
# is False
#tsd.core.auto_create_metrics = false
tsd.core.auto_create_metrics = true
# --------- STORAGE ----------
# Whether or not to enable data compaction in HBase, default is True
#tsd.storage.enable_compaction = true
# How often, in milliseconds, to flush the data point queue to storage,
# default is 1,000
# tsd.storage.flush_interval = 1000
# Name of the HBase table where data points are stored, default is "tsdb"
#tsd.storage.hbase.data_table = tsdb
# Name of the HBase table where UID information is stored, default is "tsdb-uid"
#tsd.storage.hbase.uid_table = tsdb-uid
# Path under which the znode for the -ROOT- region is located, default is "/hbase"
#tsd.storage.hbase.zk_basedir = /hbase
# A comma separated list of Zookeeper hosts to connect to, with or without
# port specifiers, default is "localhost"
#tsd.storage.hbase.zk_quorum = localhost
5.启动
带参数模式
./build/tsdb tsd --port=4242 --staticroot=build/staticroot --cachedir="$tsdtmp" --zkquorum=127.0.0.1:2181
使用配置文件
./build/tsdb tsd
6. 创建指标
1. 手动创建指标
[root@10.77.51.10 opentsdb]# ./build/tsdb mkmetric mysql.bytes_received mysql.bytes_sent
metrics mysql.bytes_received: [0, 0, 1]
metrics mysql.bytes_sent: [0, 0, 2]
2.自动创建指标
需要修改配置文件中tsd.core.auto_create_metrics = true,默认为false。
需要向tsdb的监听端口发送消息。比如：
put tsd.connectionmgr.connections 1443170747 6 type=open host=localhost.localdomain
3.向OpenTSDB添加数据
(1)telnet或者nc
语法：
put <metric> <timestamp> <value> <tagk1=tagv1[ tagk2=tagv2 ...tagkN=tagvN]>
例如:
put sys.cpu.user 1356998400 42.5 host=webserver01 cpu=0
[root@10.77.51.10 ~]# telnet 127.0.0.1 4242
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
put proc.test.telnet 1443426262 12 telnet_host=localhost
[root@10.77.51.10 ~]# nc 127.0.0.1 4242
put proc.test.nc 1443426262 3 nc_host=localhost
(2)HTTP API
curl -H "Content-Type: application/json" -X POST http://127.0.0.1:4242/api/put/ -d '{"metric":"proc.test.curlpost","timestamp":1443427323,"value":18,"tags":{"host":"web01","dc":"lga"}}'
批量投递:
curl -H "Content-Type: application/json" -X POST http://127.0.0.1:4242/api/put/ -d '[{"metric":"proc.test.curlpost","timestamp":1443427323,"value":18,"tags":{"host":"web01","dc":"lga"}},{"metric":"proc.test.curlpost","timestamp":1443427323,"value":18,"tags":{"host":"web01","dc":"lga"}}]'
或者通过以下python脚本
[root@10.77.51.10 opentsdb_test]# cat test_post.py #!/usr/bin/env python # -*- coding:utf-8 -*- import httplib import urllib import urllib2 import json import time import random def my_post(): base_url = '10.77.51.10' headerdata = {'Content-type':'application/json'} time_cur = int(time.time()) va = random.randint(1,100) values = { "metric": "proc.test.httppost", "timestamp": time_cur, "value": va, "tags": { "host": "web01", "dc": "lga" } } jdata = json.dumps(values) conn = httplib.HTTPConnection(base_url,4242) conn.request(method="POST",url="/api/put/",body=jdata,headers=headerdata) res = conn.getresponse() res.read() # get response /api/put?details conn.request(method="POST",url="/api/put?details",body=jdata,headers=headerdata) res = conn.getresponse() return res.read() while(True): try: resp = my_post() print(resp) time.sleep(1) except Exception as e: exit(0) except KeyboardInterrupt as e: exit(0) [root@10.77.51.10 opentsdb_test]# python test_post.py {"errors":[],"failed":0,"success":1} {"errors":[],"failed":0,"success":1} {"errors":[],"failed":0,"success":1} {"errors":[],"failed":0,"success":1} ^C[root@10.77.51.10 opentsdb_test]#
参考http://opentsdb.net/docs/build/html/api_http/put.html
(3)批量导入
参考http://opentsdb.net/docs/build/html/user_guide/cli/import.html
3. tcollector安装
1. 下载源码
https://github.com/OpenTSDB/tcollector
2. 启动
[root@10.77.51.10 tcollector]# pwd
/root/guochunyang/tcollector
[root@10.77.51.10 tcollector]# python tcollector.py
4. opentsdb_test
1. 启动opentsdb,配置支持tsd.core.auto_create_metrics = true,启动hbase,启动tcollector。用浏览器打开,打开后如图
2. 测试
Metric:test1.test1.test1
测试程序：
[root@10.77.51.10 opentsdb_test]# cat test2.py
import subprocess
import time
import logging
import random
logging.basicConfig(level=logging.DEBUG,filename="test.log")
def put_data(metric,value,**kw):
t = int(time.time())
cmd = "echo put %s %d %d host=test1 tag_k1=tag_v1 | nc -w 30 127.0.0.1 4242" %(metric,t,value)
subp = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE)
random.seed()
while(True):
i = random.randint(1,100)
time.sleep(1)
put_data("test1.test1.test1",i)
i += 1
监控结果如图：
资源：
http://opentsdb.net/docs/build/html/installation.html
http://www.searchtb.com/2012/07/opentsdb-monitoring-system.html
http://www.ttlsa.com/opentsdb/opentsdb-comments/
http://opentsdb.net/docs/build/html/user_guide/utilities/tcollector.html
http://www.ttlsa.com/opentsdb/opentsdb-tcollector-source-code-analysis/
http://blog.sina.com.cn/s/blog_63cae59301016wfd.html
http://opentsdb.net/docs/build/html/user_guide/writing.html
https://yq.aliyun.com/articles/54785 数据表设计
https://blog.csdn.net/u012965373/article/details/79361781
1562
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
