scribe日志系统安装笔记

scribe日志系统安装笔记

目前web访问日志为crontab定时清空，而且负载均衡后，访问随机分配到一台服务器。程序出错后需要分析日志时，需要登录几台机器查看，于是想用一台主机作日志主机，负责收集（按时间顺序而不用sort?）、分析日志。下面的是年前安装scribe日志系统的记录。

一、软件下载 

wget http://www.monkey.org/~provos/libevent-2.0.10-stable.tar.gz
wget http://nchc.dl.sourceforge.net/project/boost/boost/1.45.0/boost_1_45_0.tar.bz2
wget http://apache.etoak.com//incubator/thrift/0.5.0-incubating/thrift-0.5.0.tar.gz
wget https://download.github.com/facebook-scribe-2ee14d3.tar.gz
wget http://cloud.github.com/downloads/facebook/scribe/scribe-2.2.tar.gz

二、安装步骤 

1～4系统已经安装过，下面安装略过，未安装的可yum install libevent进行安装。

1、gcc> 3.3.5 
gcc -v
which gcc
2、安装ruby 
3、安装python 
4、安装libevent 
5、安装 boost 
tar jxvf boost_1_45_0.tar.bz2
cd boost_1_45_0
./bootstrap.sh
./bjam -s HAVE_ICU=1 --prefix=/usr/local/boost --includedir=/usr/local/boost/include --libdir=/usr/local/boost/lib
./bjam install --prefix=/usr/local/boost

6、安装facebook 服务 

安装thrift
tar zxvf thrift-0.5.0.tar.gz
cd thrift-0.5.0
./configure --with-boost=/usr/local/boost --with-php-config=/usr/local/php5/bin/php-config
make

(出现如下错误，编辑文件lib/cpp/src/concurrency/ThreadManage.h修改为
24 #include <boost/tr1/tr1/functional>后，重新make)

src/concurrency/ThreadManager.h:24:26: tr1/functional: No such file or
directory
In file included from src/concurrency/ThreadManager.cpp:20:
src/concurrency/ThreadManager.h:61: error: `std::tr1' has not been declared
src/concurrency/ThreadManager.h:61: error: ISO C++ forbids declaration
of `function' with no type
src/concurrency/ThreadManager.h:61: error: expected `;' before '<' token
src/concurrency/ThreadManager.h:182: error: `ExpireCallback' has not
been declared
src/concurrency/ThreadManager.h:182: error: ISO C++ forbids declaration
of `expireCallback' with no type
src/concurrency/ThreadManager.cpp:135: error: `ExpireCallback' has not
been declared
src/concurrency/ThreadManager.cpp:135: error: ISO C++ forbids
declaration of `expireCallback' with no type
src/concurrency/ThreadManager.cpp:145: error: `ExpireCallback' does not
name a type
src/concurrency/ThreadManager.cpp: In member function `virtual void
apache::thrift::concurrency::ThreadManager::Impl::removeExpiredTasks()':
src/concurrency/ThreadManager.cpp:533: error: `expireCallback_' was not
declared in this scope
src/concurrency/ThreadManager.cpp: At global scope: 

make install
安装fb303
cd contrib/fb303

(目录下没有Makefile.in，将thrift目录下的Makefile.in至fb303目录下)
./bootstrap.sh
./configure --with-boost=/usr/local/boost
make
make install
7、安装scribe 
环境变量
export BOOST_ROOT=/usr/local/boost
export LD_LIBRARY_PATH=/usr/local/boost/lib::/usr/lob:/usr/local/lib
ldconfig -v
tar zxvf facebook-scribe-2ee14d3.tar.gz (scribe需要在github上下载最新版，scribe2.2发布版与thrift0.5.0不兼容，Ying-Yi Liang搞定并更新了)

cd facebook-scribe-2ee14d3
./bootstrap.sh
./configure --with-boost=/usr/local/boost --prefix=/usr/local/scribe
make
make install

8、配置测试 
mkdir /usr/local/scribe/conf
cp examples/example1.conf /usr/local/scribe/conf
/usr/local/scribe/bin/scribed -c /usr/local/scribe/conf/example1.conf
/usr/local/scribe/bin/scribed -c /usr/local/scribe/conf/example1.conf &
nohup /usr/local/scribe/bin/scribed -c /usr/local/scribe/conf/example1.conf &
exit

测试程序：
mkdir  /tmp/scribetest
测试写入：
echo  "hello world"| /usr/local/softs/facebook-scribe-2ee14d3/examples/scribe_cat test
查看结果：
cat  /tmp/scribetest/test/test_current

 

9、配置PHP接口

 1）生成PHP库及配置目录

cd /usr/local/scribe
/usr/local/bin/thrift -o . -I /usr/local/share/ --gen php /usr/local/share/fb303/if/fb303.thrift /usr/local/bin/thrift -o . -I /usr/local/share/ --gen php /usr/local/soft/facebook-scribe-2ee14d3/if/scribe.thrift cp /usr/local/soft/thrift-0.5.0/lib/php/src includes/ -r mkdir -p includes/packages/fb303 mkdir -p includes/packages/scribe mv gen-php/fb303/FacebookService.php gen-php/fb303/fb303_types.php includes/packages/fb303/ mv gen-php/scribe/scribe_types.php includes/packages/scribe/ mv gen-php/scribe/scribe.php includes/ rm -rf gen-php
2）编写测试程序：
vi test.php <?php //参考http://www.ruturaj.net/scribe-php-logging $GLOBALS['THRIFT_ROOT'] = './includes'; include_once $GLOBALS['THRIFT_ROOT'] . '/scribe.php'; include_once $GLOBALS['THRIFT_ROOT'] . '/transport/TSocket.php'; include_once $GLOBALS['THRIFT_ROOT'] . '/transport/TFramedTransport.php'; include_once $GLOBALS['THRIFT_ROOT'] . '/protocol/TBinaryProtocol.php'; $msg1['category'] = 'keyword'; $msg1['message'] = "This is some message for the category/n"; $msg2['category'] = 'keyword'; $msg2['message'] = "Some other message for the category/n"; //$log_entry = new LogEntry( array('category'=>$category, 'category'=>$category) ) $entry1 = new LogEntry($msg1); $entry2 = new LogEntry($msg2); $messages = array($entry1, $entry2); $socket = new TSocket('localhost', 1463, true); $transport = new TFramedTransport($socket); //$protocol = new TBinaryProtocol($trans, $strictRead=false, $strictWrite=true) $protocol = new TBinaryProtocol($transport, false, false); //$scribe_client = new scribeClient($iprot=$protocol, $oprot=$protocol) $scribe_client = new scribeClient($protocol, $protocol); $transport->open(); $scribe_client->Log($messages); $transport->close(); ?>
3）执行程序：

/usr/local/php5/bin/php test.php
4）查看结果：
cat /tmp/scribetest/keyword/keyword_current

三、相关文章 
多个服务器日志的排序合并： clfmerge
scribe的安装与使用 
scribe-日志收集器分析
使用Scribe来监控大型网络系统
nginx-scribe-log
Scribe PHP logging
Pipe Apache (or any) Logs to Scribe

 

参考：

scribe日志系统安装笔记
Facebook scribe在Ubuntu下面安装笔记