flume mysql hbase_【flume】5.采集日志进入hbase

设置我们的flume配置信息

# Licensed to the Apache Software Foundation (ASF) under one

# or more contributor license agreements. See the NOTICE file

# distributed withthis work foradditional information

# regarding copyright ownership. The ASF licensesthisfile

# to you under the Apache License, Version2.0(the

#"License"); you may not use thisfile except in compliance

# with the License. You may obtain a copy of the License at

#

# http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing,

# software distributed under the License is distributed on an

#"AS IS"BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY

# KIND, either express or implied. See the Licenseforthe

# specific language governing permissions and limitations

# under the License.

# The configuration file needs to define the sources,

# the channels and the sinks.

# Sources, channels and sinks are defined per agent,

# inthis case called 'agent'agent1.sources=r1

agent1.channels=c1

agent1.sinks=s1

# For each one of the sources, the type is defined

agent1.sources.r1.type=exec

#tail-F /home/oss/cloud_iom/ktpt/iom-cloud-service/logs/iom-app-debug.log

agent1.sources.r1.command= tail -F /home/oss/cloud_iom/ktpt/iom-cloud-service/logs/iom-app-debug.log

# The channel can be defined as follows.

#agent.sources.seqGenSrc.channels=memoryChannel

agent1.sources.r1.channels=c1

# Each sink's type must be defined

agent1.sinks.s1.type =hbase2

agent1.sinks.s1.table=iom_app_debug

agent1.sinks.s1.columnFamily=log

agent1.sinks.s1.serializer=org.apache.flume.sink.hbase2.RegexHBase2EventSerializer

#agent1.sinks.s1.serializer.regex= \\[(.*?)\\]\\ \\[(.*?)\\]\\ \\[(.*?)\\]\\ \\[(.*?)\\]#Specify the channel the sink should use

agent1.sinks.s1.channel=c1

# Each channel's type is defined.

agent1.channels.c1.type =memory

# Other config values specific to each type of channel(sink or source)

# can be defined as well

# Inthis case, it specifies the capacity of the memory channel

agent1.channels.c1.capacity= 100

这个脚本配置好，设置启动命令,使用nohup是为了之后采集器自己后期自动运行

nohup flume-ng --conf hadoop/flume/conf -f hadoop/flume/conf/flume-conf.properties -n agent1 -Dflume.root.logger=DEBUG,console &

我的flume目录：

采集截图

日志文件截图

当然，这里是按行进行采集的(用的tail -F)，但是shell脚本可以自己定义,只要type配置的是exec，后面sink对象也可以自己配置

第一步数据采集，第二步应该是想想如何进行数据分析，当然这样采集的数据直接分析的可能性也不太大，而且数据杂乱无序，我们还需要定义相应的逻辑先对数据进行清洗，然后再采集进去

这里这样采集是有问题的，正确的做法应该是

1.flume采集数据进入hdfs

2.MapReduce对采集进入的数据进行数据清洗，整理数据

3.MapReduce分析数据，解析入库进入hbase，或者直接保存到hdfs

4.sqoop 迁移数据到对应的数据库(mysql，Oracle)

5.根据解析之后的数据，查询Oracle制作报表图像，分析趋势，预测，或者定位问题关系