java flume hbase_flume将数据导入到hbase中

安装flume：

[hadoop@h71 ~]$ tar -zxvf flume-ng-1.6.0-cdh5.5.2.tar.gz

修改 flume-env.sh 配置文件,主要是JAVA_HOME变量设置

[hadoop@h71 apache-flume-1.6.0-cdh5.5.2-bin]$ cp conf/flume-env.sh.template conf/flume-env.sh

[hadoop@h71 apache-flume-1.6.0-cdh5.5.2-bin]$ vi conf/flume-env.sh

添加：

export JAVA_HOME=/usr/jdk1.7.0_25

(这里添加的是你的Java安装目录，我这里安装的是jdk1.7.0_25，注意Java版本不要过低)

验证是否安装成功：

[hadoop@h71 apache-flume-1.6.0-cdh5.5.2-bin]$ bin/flume-ng version

Flume 1.6.0-cdh5.5.2

Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git

Revision: f65a722cd2d1e7ceeda972570a5a6ee01c3a0a3d

Compiled by jenkins on Mon Jan 25 16:38:11 PST 2016

From source with checksum 028a2c6b035a03df1dfa91a3feda3424

[hadoop@h71 ~]$ cd hbase-1.0.0-cdh5.5.2/lib/

然后将以下文件复制到flume中：

[hadoop@h71 lib]$ cp protobuf-java-2.5.0.jar /home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin/lib/

[hadoop@h71 lib]$ cp hbase-protocol-1.0.0-cdh5.5.2.jar /home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin/lib/

[hadoop@h71 lib]$ cp hbase-client-1.0.0-cdh5.5.2.jar /home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin/lib/

[hadoop@h71 lib]$ cp hbase-common-1.0.0-cdh5.5.2.jar /home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin/lib/

[hadoop@h71 lib]$ cp hbase-server-1.0.0-cdh5.5.2.jar /home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin/lib/

[hadoop@h71 lib]$ cp hbase-hadoop2-compat-1.0.0-cdh5.5.2.jar /home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin/lib/

[hadoop@h71 lib]$ cp hbase-hadoop-compat-1.0.0-cdh5.5.2.jar /home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin/lib/

[hadoop@h71 lib]$ cp htrace-core-3.2.0-incubating.jar /home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin/lib/

(也可以直接把hbase-1.0.0-cdh5.5.2/lib下的jar包全部复制到flume的lib目录下)

确保test_idoall_org表在hbase中已经存在

hbase(main):002:0> create 'test_idoall_org','uid','name'

0 row(s) in 0.6730 seconds

=> Hbase::Table - test_idoall_org

hbase(main):003:0> put 'test_idoall_org','10086','name:idoall','idoallvalue'

0 row(s) in 0.0960 seconds

[hadoop@h71 apache-flume-1.6.0-cdh5.5.2-bin]$ vi conf/hbase_simple.conf

a1.sources = r1

a1.sinks = k1

a1.channels = c1

# Describe/configure the source

a1.sources.r1.type = exec

a1.sources.r1.command = tail -F /home/hadoop/data.txt

a1.sources.r1.port = 44444

a1.sources.r1.host = 192.168.8.71

a1.sources.r1.channels = c1

# Describe the sink

a1.sinks.k1.type = logger

a1.sinks.k1.type = hbase

a1.sinks.k1.table = test_idoall_org

a1.sinks.k1.columnFamily = name

a1.sinks.k1.column = idoall

a1.sinks.k1.serializer = org.apache.flume.sink.hbase.RegexHbaseEventSerializer

a1.sinks.k1.channel = memoryChannel

# Use a channel which buffers events in memory

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000

a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel

a1.sources.r1.channels = c1

a1.sinks.k1.channel = c1

启动flume agent：

[hadoop@h71 apache-flume-1.6.0-cdh5.5.2-bin]$ bin/flume-ng agent -c . -f conf/hbase_simple.conf -n a1 -Dflume.root.logger=INFO,console

。。。。。。。。。(前面省略，太多了)

:/home/hadoop/hive-1.1.0-cdh5.5.2/lib/logredactor-1.0.3.jar:/home/hadoop/hive-1.1.0-cdh5.5.2/lib/commons-dbcp-1.4.jar:/home/hadoop/hive-1.1.0-cdh5.5.2/lib/jcommander-1.32.jar

12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:java.library.path=:/home/hadoop/hadoop-2.6.0-cdh5.5.2/lib/native

12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp

12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:java.compiler=12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux

12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:os.arch=i386

12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.18-194.el5

12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:user.name=hadoop

12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/hadoop

12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin

12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x1d630160x0, quorum=localhost:2181, baseZNode=/hbase

12/12/13 00:21:09 INFO zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)

12/12/13 00:21:09 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /127.0.0.1:47668, server: 127.0.0.1/127.0.0.1:2181

12/12/13 00:21:09 INFO zookeeper.ClientCnxn: Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x3b8fd59e830000, negotiated timeout = 90000

12/12/13 00:21:10 INFO Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available

12/12/13 00:21:10 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: k1: Successfully registered new MBean.

12/12/13 00:21:10 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: k1 started

(启动成功。。。)

产生数据：

[hadoop@h71 ~]$ touch data.txt

[hadoop@h71 ~]$ echo "hello idoall.org from flume" >> data.txt

这时登录到hbase中，可以发现新数据已经插入：

hbase(main):005:0> scan 'test_idoall_org'

ROW COLUMN+CELL

10086 column=name:idoall, timestamp=1355329032253, value=idoallvalue

1355329550628-0EZpfeEvxG-0 column=name:payload, timestamp=1355329383396, value=hello idoall.org from flume

2 row(s) in 0.0140 seconds