datahub同步到mysql_mysql数据通过fluent同步到阿里云datahub填坑过程

1、ruby安装

#wget http://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.1.tar.gz

#tar zxvf ruby-2.1.1.tar.gz

#cd ruby-2.1.1

#./configure --prefix=/root/bi/ruby --with-opessl-dir=/root/bi/openssl-1.0.0l

#make && make install

添加export PATH=/root/bi/ruby/bin:$PATH 到/root/.bash_profile文件末尾

2、安装gem并更新gem源

#wget http://production.cf.rubygems.org/rubygems/rubygems-2.2.2.tgz

#tar zxvf rubygems-2.2.2.tgz

#cd rubygems-2.2.2

#ruby setup.rb

#gem --version

#gem sources --remove https://rubygems.org/

#gem sources -a https://ruby.taobao.org/

#gem sources -l

gem sources -a https://ruby.taobao.org/这个步骤可能会报错：ERROR: While executing gem … (Gem::Exception)

Unable to require openssl, install OpenSSL and rebuild ruby (preferred) or use non-HTTPS sources

[root@Source ~]# gem sources -u ; gem source -l

ERROR: While executing gem … (Gem::Exception)

Unable to require openssl, install OpenSSL and rebuild ruby (preferred) or use non-HTTPS sources

解决这个问题需要

yum -y install openssl openssl-devel ; make clean ; 然后重新安装

3、安装依赖

gem install protobuf

gem install fluentd -v "~> 0.12.0" --no-ri --no-rdoc --指定版本

4、安装插件

gem install fluent-plugin-aliyun-odps

gem install fluent-plugin-datahub

gem install fluent-plugin-sql

5、配置fluent.conf

把/root/bi/ruby/lib/ruby/gems/2.4.0/gems/fluentd-0.14.13/fluent.conf拷贝到/et c/fluent目录

配置source和match

source配置：

type sql

host xxxxx

database xxxx

adapter mysql2

username xxxx

password xxxx

select_interval 10s

select_limit 1000

state_file /path/sql_state

table logininfo

tag logininfo

update_column loginInfoId

match配置

@type datahub

access_id LTAI6Tr4838pXfPO

access_key Rj7PboFRfbEOhIUKpxju7l0WvJIluT

endpoint http://dh-cn-hangzhou.aliyuncs.com

buffer_chunk_limit 3m

buffer_queue_limit 128

flush_interval 5s

project_name mysql_datahub

topic_name logininfo

column_names ["logininfoid","version","lastmodifieddate","email","loanpoint"]

dirty_data_continue true

dirty_data_file /root/bi/dirtyfile.txt

put_data_batch_size 1000

6、启动fluent

这步可能会报/usr/local/lib/ruby/gems/2.2.0/gems/activerecord- 4.2.6/lib/active_record/connection_adapters/connection_specification.rb:177:in rescue in spec': Specified 'mysql' for database adapter, but the gem is not loaded. Addgem ‘mysql’` to your Gemfile (and ensure its version is at the minimum required by ActiveRecord). (Gem::LoadError)

这个问题是mysql插件需要用到mysql adapter适配器，需要安装mysql adapter适配器，执行以下命令：

yum install mysql-devel

gem install mysql2

注意是运行gem install mysql2 而不是gem install mysql，同时在source配置里面也要把adapter 设为mysql2。在这一步也可能遇到mysql包冲突的问题，所以建议在一个没有安装mysql的服务器上安装。

7、最后的问题

因为阿里云datahub是大小写敏感的，所以要把所有字段设为小写，不然字段会变为Null或空白。那么这需要怎么解决呢，查找fluent的官方文档，有record_transformer Filter Plugin这个东西，对于有大写字母的字段我们需配置一个filter,同时注意filter必须配置在match前面。

filter配置

@type record_transformer

logininfoid ${record["loginInfoId"]}