flume使用（二）：采集远程日志数据到MySql数据库

本文内容可查看目录

本文内容包含单节点（单agent）和多节点（多agent，采集远程日志）说明

一、环境

linux系统：Centos7
Jdk：1.7

Flume：1.7.0

二、安装

linux中jdk、mysql的安装不多赘述

flume1.7的安装：进入官网：http://flume.apache.org/

然后找到1.7版本下载放到centos系统解压即可

三、准备数据库表

注，本文flume的event是execSource来源。即通过执行linux命令获得执行结果作为flume的数据源。通过自定义MysqlSink作为flume的sink。

创建sql语句：

CREATE TABLE `flume_test` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `name` varchar(255) DEFAULT NULL,
  `age` int(11) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

四、MysqlSink编写

4.1.maven创建项目（打包方式为jar）

pom.xml文件：

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>yichao.mym</groupId>
  <artifactId>flumeDemo</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>
  
     <dependencies>
        <dependency>
            <groupId>org.apache.flume</groupId>
            <artifactId>flume-ng-core</artifactId>
            <version>1.7.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flume</groupId>
            <artifactId>flume-ng-configuration</artifactId>
            <version>1.7.0</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/mysql/mysql-connector-java -->
		<dependency>
			<groupId>mysql</groupId>
			<artifactId>mysql-connector-java</artifactId>
			<version>5.1.15</version>
		</dependency>
    </dependencies>
</project>

4.2 准备java Bean

与数据库表对应的javabean，方便处理event的body（event的body就是execSource的命令读取的内容）

package yichao.mym.base.bean;

public class Person {

	private String name;
	
	private Integer age;

	public String getName() {
		return name;
	}

	public void setName(String name) {
		this.name = name;
	}

	public Integer getAge() {
		return age;
	}

	public void setAge(Integer age) {
		this.age = age;
	}
	
}

4.3 自定义的sink编写

说明都在代码中

package yichao.mym.base.bean;

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.util.List;

import org.apache.flume.Channel;
import org.apache.flume.Context;
import org.apache.flume.Event;
import org.apache.flume.EventDeliveryException;
import org.apache.flume.Transaction;
import org.apache.flume.conf.Configurable;
import org.apache.flume.sink.AbstractSink;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import com.google.common.base.Preconditions;
import com.google.common.base.Throwables;
import com.google.common.collect.Lists;

public class MysqlSink extends AbstractSink implements Configurable {

    private Logger LOG = LoggerFactory.getLogger(MysqlSink.class);
    
    private String hostname;
    private String port;
    private String databaseName;
    private String tableName;
    private String user;
    private String password;
    private PreparedStatement preparedStatement;
    private Connection conn;
    private int batchSize;		//每次提交的批次大小

    public MysqlSink() {
        LOG.info("MySqlSink start...");
    }

    /**实现Configurable接口中的方法：可获取配置文件中的属性*/
    public void configure(Context context) {
        hostname = context.getString("hostname");
        Preconditions.checkNotNull(hostname, "hostname must be set!!");
        port = context.getString("port");
        Preconditions.checkNotNull(port, "port must be set!!");
        databaseName = context.getString("databaseName");
        Preconditions.checkNotNull(databaseName, "databaseName must be set!!");
        tableName = context.getString("tableName");
        Preconditions.checkNotNull(tableName, "tableName must be set!!");
        user = context.getString("user");
        Preconditions.checkNotNull(user, "user must be set!!");
        password = context.getString("password");
        Preconditions.checkNotNull(password, "password must be set!!");
        batchSize = context.getInteger("batchSize", 100);		//设置了batchSize的默认值
        Preconditions.checkNotNull(batchSize > 0, "batchSize must be a positive number!!");
    }

    /**
     * 服务启动时执行的代码，这里做准备工作
     */
    @Override
    public void start() {
        super.start();
        try {
            //调用Class.forName()方法加载驱动程序
            Class.forName("com.mysql.jdbc.Driver");
        } catch (ClassNotFoundException e) {
            e.printStackTrace();
        }

        String url = "jdbc:mysql://" + hostname + ":" + port + "/" + databaseName;
        //调用DriverManager对象的getConnection()方法，获得一个Connection对象

        try {
            conn = DriverManager.getConnection(url, user, password);
            conn.setAutoCommit(false);
            //创建一个Statement对象
            preparedStatement = conn.prepareStatement("insert into " + tableName +
                    " (name,age) values (?,?)");

        } catch (SQLException e) {
            e.printStackTrace();
            System.exit(1);
        }

    }

    /**
     * 服务关闭时执行
     */
    @Override
    public void stop() {
        super.stop();
        if (preparedStatement != null) {
            try {
                preparedStatement.close();
            } catch (SQLException e) {
                e.printStackTrace();
            }
        }

        if (conn != null) {
            try {
                conn.close();
            } catch (SQLException e) {
                e.printStackTrace();
            }
        }
    }

    /**
     *  执行的事情：<br/>
          	（1）持续不断的从channel中获取event放到batchSize大小的数组中<br/>
          	（2）event可以获取到则进行event处理，否则返回Status.BACKOFF标识没有数据提交<br/>
          	（3）batchSize中有内容则进行jdbc提交<br/>
     */
    public Status process() throws EventDeliveryException {
        Status result = Status.READY;
        Channel channel = getChannel();
        Transaction transaction = channel.getTransaction();
        Event event;
        String content;

        List<Person> persons = Lists.newArrayList();
        transaction.begin();
       
        
        try {
        	/*event处理*/
            for (int i = 0; i < batchSize; i++) {
                event = channel.take();
                if (event != null) {//对事件进行处理
                    //event 的 body 为   "exec tail-event-$i , $i"
                    content = new String(event.getBody());
                    Person person=new Person();
                    if (content.contains(",")) {
                        //存储 event 的 content
                    	person.setName(content.substring(0, content.indexOf(",")));
                        //存储 event 的 create  +1 是要减去那个 ","
                    	person.setAge(Integer.parseInt(content.substring(content.indexOf(",")+1).trim()));
                    }else{
                    	person.setName(content);
                    }
                    persons.add(person);
                } else {
                    result = Status.BACKOFF;
                    break;
                }
            }

            /*jdbc提交*/
            if (persons.size() > 0) {
                preparedStatement.clearBatch();
                for (Person temp : persons) {
                    preparedStatement.setString(1, temp.getName());
                    preparedStatement.setInt(2, temp.getAge());
                    preparedStatement.addBatch();
                }
                preparedStatement.executeBatch();
                conn.commit();
            }
            transaction.commit();
        } catch (Exception e) {
            try {
                transaction.rollback();
            } catch (Exception e2) {
                LOG.error("Exception in rollback. Rollback might not have been.successful.", e2);
            }
            LOG.error("Failed to commit transaction.Transaction rolled back.", e);
            Throwables.propagate(e);
        } finally {
            transaction.close();
        }
        return result;
    }
}

编写好后打包成jar，发送到 flume安装目录下的lib文件夹中。同时 把mysql的驱动包mysql-connector-java一起放过去

4.4 conf配置：编写mysqlSink.conf（单agent的测试）

在flume的conf 文件夹下新建配置文件 mysqlSink.conf 内容如下:

agent1.sources=source1
agent1.channels=channel1
agent1.sinks=mysqlSink

# describe/configure source1
# type:exec is through linux command like 'tail -F' to collect logData
agent1.sources.source1.type=exec
agent1.sources.source1.command=tail -F /usr/local/tomcat/logs/ac.log
agent1.sources.source1.channels=channel1

# use a channel which buffers events in memory
# type:memory or file is to temporary to save buffer data which is sink using
agent1.channels.channel1.type=memory
agent1.channels.channel1.capacity=5000
agent1.channels.channel1.transactionCapacity=1000

# describe sink. there are using mysqlSink that is a jar
agent1.sinks.mysqlSink.type=yichao.mym.base.bean.MysqlSink
agent1.sinks.mysqlSink.hostname=localhost
agent1.sinks.mysqlSink.port=3306
agent1.sinks.mysqlSink.databaseName=firstflume
agent1.sinks.mysqlSink.tableName=flume_test
agent1.sinks.mysqlSink.user=root
agent1.sinks.mysqlSink.password=123456
agent1.sinks.mysqlSink.channel=channel1
agent1.sinks.mysqlSink.batchSize=5

说明：

(1)localhost 为mysql 数据库所在的服务器IP；

(2)/usr/local/tomcat/logs/ac.log；

(3)yichao.mym.base.bean.MysqlSink是自定义sink的mysqlsink的全称；

重点：capacity（channel大小） > transactionCapacity（大小是每次flume的事务大小） > batchSize（sink会一次从channel中取多少个event去发送）。

这些数值应根据实时性要求、并发量、占用系统资源等方面权衡设计，但必须遵循以上标准。flume官方却没有这样的说明，一旦没有遵循，执行过程中就会报错！

五、准备测试

启动flume：在flume安装目录下的bin目录中：

./flume-ng agent -c ../conf -f ../conf/mysqlSink.conf -n agent1 -Dflume.root.logger=INFO,console

启动服务后，可以模拟log文件的动态增长，新开终端，通过shell命令：

for i in {1..100};do echo "exec tail-name-$i,$i" >> /usr/local/tomcat/logs/ac.log;sleep 1;done;

此时可以快速刷新数据库的数据表，可以看到数据正在动态增长：

-----------------------------------------------------------------------------------------------------

六、多节点多agent

1.说明架构方式

两台可互相通信的linux机器：

201机器：安装好jdk1.7，mysql，flume1.7

202机器：安装好jdk1.7，flume1.7

结构：

不过本案例中，agent1、agent2、agent3都是execSource源，即直接读取磁盘上的log文件，而不是log4j直接作为agent的source。

那么对于本案例，202机器就作为其中一个agent收集者（agent1、agent2、agent3），把从本机上收集的log内容发送到远程的201机器。他们之间就是使用avro作为传输协议。

所以本案例202机器的：

source：exec （tail -F /usr/local/tomcat/logs/ac.log）

channel：memory

sink：avro

本案例201机器的:

source：avro

channel：memory

sink：自定义的mysqlSink

注：表、自定义的sink的jar、javaBean都和之前的一致

2.两个agent的配置文件conf

202机器的flume配置文件：tail-avro.conf

agent1.sources=source1  
agent1.channels=channel1  
agent1.sinks=mysqlSink  
  
# describe/configure source1  
# type:exec is through linux command like 'tail -F' to collect logData  
agent1.sources.source1.type=exec  
agent1.sources.source1.command=tail -F /usr/local/tomcat/logs/ac.log  
agent1.sources.source1.channels=channel1  
  
# use a channel which buffers events in memory  
# type:memory or file is to temporary to save buffer data which is sink using  
agent1.channels.channel1.type=memory  
agent1.channels.channel1.capacity=5000  
agent1.channels.channel1.transactionCapacity=1000  

agent1.sinks.mysqlSink.type=avro
agent1.sinks.mysqlSink.channel=channel1
agent1.sinks.mysqlSink.hostname=192.168.216.201
agent1.sinks.mysqlSink.port=4545
agent1.sinks.mysqlSink.batch-size=5

201机器的flume配置文件：avro-mysql.conf

agent1.sources=source1  
agent1.channels=channel1  
agent1.sinks=mysqlSink  
  
# describe/configure source1  
# type:avro is through net-protocal-transport to collect logData  
agent1.sources.source1.type = avro
agent1.sources.source1.channels = channel1
agent1.sources.source1.bind = 192.168.216.201
agent1.sources.source1.port = 4545
  
# use a channel which buffers events in memory  
# type:memory or file is to temporary to save buffer data which is sink using  
agent1.channels.channel1.type=memory  
agent1.channels.channel1.capacity=5000  
agent1.channels.channel1.transactionCapacity=1000  

# describe sink. there are using mysqlSink that is a jar  
agent1.sinks.mysqlSink.type=yichao.mym.base.bean.MysqlSink  
agent1.sinks.mysqlSink.hostname=localhost  
agent1.sinks.mysqlSink.port=3306  
agent1.sinks.mysqlSink.databaseName=firstflume  
agent1.sinks.mysqlSink.tableName=flume_test  
agent1.sinks.mysqlSink.user=root  
agent1.sinks.mysqlSink.password=123456  
agent1.sinks.mysqlSink.channel=channel1  
agent1.sinks.mysqlSink.batchSize=5  

分别配置好并且启动服务。（可先启动机器201，因为机器202需要连接机器201）

3.启动测试

机器 201 的flume启动命令：在flume目录下的bin目录中执行

./flume-ng agent -c ../conf -f ../conf/avro-mysql.conf -n agent1 -Dflume.root.logger=INFO,console

机器 202 的flume启动命令：在flume目录下的bin目录中执行

./flume-ng agent -c ../conf -f ../conf/tail-avro.conf -n agent1 -Dflume.root.logger=INFO,console

启动完之后在机器202上进行模拟log文件数据动态生成：

for i in {1..150};do echo "exec tail-name-$i,$i" >> /usr/local/tomcat/logs/ac.log;sleep 1;done;

此时可以查看机器201上的数据库表的数据是否有动态添加：

至此多节点agent的测试完成！