日志归集系统搭建（1）——需求背景及环境搭建

---

需求背景

公司需要搭建一个日志收集服务器，用于将公司的项目日志汇总到一台服务器上面，方便查看和减轻各项目服务器压力。但是由于目前资源不够充足，所以放弃使用ELK、EFK。最后在调研尝试过Linux自带的Syslog和fluentd之后，Linux自带的Syslog虽然更简单，但是汇总的数据中会有乱码的情况（怀疑是Appender问题），并且将日志信息组合成Json格式比较麻烦，因此选择使用fluentd来做日志收集。

安装各个组件

选用fluentd来做日志收集选择使用源码来进行安装，并且需要安装一些依赖的插件，如Ruby等，下面依次介绍需要安装的组件。

安装ruby

下载ruby压缩包 ruby-2.6.5.tar.gz，版本：2.6.5，并解压

tar -zxvf ruby-2.6.5.tar.gz
cd ruby-2.6.5

安装ruby，安装需要gcc

yum install -y gcc
./configure prefix=/export/source/ruby
make && make install

安装完成后修改系统配置文件 /etc/profile，在最后添加ruby目录并加到path中

export RUBY_HOME=/export/source/ruby
export PATH=$PATH:$JAVA_HOME/bin:$RUBY_HOME/bin

验证是否安装成功

[root@localhost ruby-2.6.5]# ruby -v
ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]

显示ruby版本号，安装成功

获取fluentd源码

从github上拉取fluentd源码，github地址:fluentd

git clone https://github.com/fluent/fluentd.git
cd fluentd

修改gem源

先查看当前默认源

gem source
*** CURRENT SOURCES ***

https://rubygems.org/

移除默认源

gem sources -r https://rubygems.org/

添加源，目前可用的有ruby-china的源，添加ruby-china源

gem sources -a https://gems.ruby-china.com
https://gems.ruby-china.com added to sources
# 查看当前源
gem source
*** CURRENT SOURCES ***

https://gems.ruby-china.com

更新缓存

gem sources -u

安装Bundle

gem install bundler

等待提示安装成功

构建fluentd

进入fluentd文件夹后，构建项目

[root@localhost fluentd]# bundle install
Fetching gem...
...
Bundle complete! 11 Gemfile dependencies, 37 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.

[root@localhost fluentd]# bundle exec rake build
fluentd 1.7.3 built to pkg/fluentd-1.7.3.gem.

[root@localhost fluentd]# gem install pkg/fluentd-1.7.3.gem
Successfully installed fluentd-1.7.3
Parsing documentation for fluentd-1.7.3
Installing ri documentation for fluentd-1.7.3
Done installing documentation for fluentd after 3 seconds
1 gem installed

安装完成

运行fluentd

[root@localhost fluentd]# fluentd --setup ./fluent
[root@localhost fluentd]# fluentd -c ./fluent/fluent.conf -vv &

最后在控制台显示fluentd读取的配置文件信息并且打印相关日志。至此日志收集服务器所需要的fluentd已经搭建完成，接下来就是配置fluentd以完成日记收集功能。

安装过程遇到的问题

1.安装bundler没有zlib包

ERROR: Loading command: install (LoadError)
 cannot load such file -- zlib
ERROR: While executing gem ... (NoMethodError)
    undefined method `invoke_with_build_args' for nil:NilClass

解决方案
安装zlib-devel

yum -y install zlib-devel

进入ruby源码文件夹,安装ruby自身提供的zlib包ruby-2.5.1/ext/zlib

cd ruby-2.6.5/ext/zlib
ruby ./extconf.rb
make
make install

在make时会报错

make: *** No rule to make target `/include/ruby.h', needed by `zlib.o'.  Stop.

修改目录中Makefile文件

zlib.o: $(top_srcdir)/include/ruby.h

改成

zlib.o: …/…/include/ruby.h

到这里就可以make成功了。

2. 没有openssl

错误描述：cannot load such file – openssl

/export/source/ruby/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require': cannot load such file -- openssl (LoadError)
	29: from /export/source/ruby/bin/gem:21:in `<main>'
	28: from /export/source/ruby/lib/ruby/2.6.0/rubygems/gem_runner.rb:59:in `run'
	27: from /export/source/ruby/lib/ruby/2.6.0/rubygems/command_manager.rb:148:in `run'
	26: from /export/source/ruby/lib/ruby/2.6.0/rubygems/command_manager.rb:178:in `process_args'
...

/export/source/ruby/lib/ruby/2.6.0/rubygems/request.rb:84:in `rescue in configure_connection_for_https': Unable to require openssl, install OpenSSL and rebuild Ruby (preferred) or use non-HTTPS sources (Gem::Exception)
	22: from /export/source/ruby/bin/gem:21:in `<main>'
	21: from /export/source/ruby/lib/ruby/2.6.0/rubygems/gem_runner.rb:59:in `run'
	20: from /export/source/ruby/lib/ruby/2.6.0/rubygems/command_manager.rb:148:in `run'
	19: from /export/source/ruby/lib/ruby/2.6.0/rubygems/command_manager.rb:178:in `process_args'
	...

解决方案

yum install openssl-devel -y

在ruby自身提供的额外文件夹中ruby-2.6.5/ext/openssl，执行ruby ./extconf.rb，再make

ruby ./extconf.rb
make
make install

make时出错

make: *** No rule to make target `/include/ruby.h', needed by `ossl.o'.  Stop.

修改Makefile文件，在文件顶部添加top_srcdir = …/…