2-网站日志分析案例-基于Flume采集WEB日志-windows版本

2-网站日志分析案例-基于Flume采集WEB日志-windows版本

1.Flume简介

Flume is a distributed, reliable, and available service for
efficiently collecting, aggregating, and moving large amounts of log
data. It has a simple and flexible architecture based on streaming
data flows. It is robust and fault tolerant with tunable reliability
mechanisms and many failover and recovery mechanisms. It uses a simple
extensible data model that allows for online analytic application.

译文：Flume是一种分布式的、可靠的、可用的服务，用于高效地收集、聚合和移动大量的日志数据。它具有基于流数据流的简单而灵活的架构。它具有可调的可靠性机制和许多故障转移和恢复机制，具有健壮性和容错能力。它使用一个简单的可扩展数据模型，允许在线分析应用程序。

2.在Windows环境下安装Flume

1.本地需要配置JAVA_HOME

2.下载flume
https://flume.apache.org/download.html页面，我下载的版本为1.9.0

3.启动命令测试

D:\apache\apache-flume-1.9.0-bin\apache-flume-1.9.0-bin\bin>flume-ng version

安装是很简单的。

3.基于Flume完成Windows下的日志采集

3.1流程

sources类型选择：因为window下没有tail命令，所以无法监控单个文件，需要通过spooldir监控日志目录
channels类型选择：为了快，便捷，选择memory
sinks类型选择：采用logger和file_roll两种，其中logger为了查看是否成功，file_roll实现日志文件迁移

3.2具体配置

# 配置agent1的三个组件
agent1.sources = source1
agent1.sinks = sink1 sink2 sink3
agent1.channels = channel1
 
# Describe/configure spooldir source1
agent1.sources.source1.channels = channel1
agent1.sources.source1.type = spooldir
agent1.sources.source1.spoolDir = E://log
agent1.sources.source1.inputCharset = GBK
agent1.sources.source1.fileHeader = true

 
#configure host for source
agent1.sources.source1.interceptors = i1
agent1.sources.source1.interceptors.i1.type = host
agent1.sources.source1.interceptors.i1.hostHeader = hostname
 
# Describe sink1 file_roll
agent1.sinks.sink1.channel = channel1
agent1.sinks.sink1.type = file_roll
agent1.sinks.sink1.sink.directory = D://flume-collection
agent1.sinks.sink1.sink.rollInterval = 60

# Describe sink2
agent1.sinks.sink2.channel = channel1
agent1.sinks.sink2.type = logger

# Describe sink3
agent1.sinks.sink3.channel = channel1
agent1.sinks.sink3.type = hdfs
agent1.sinks.sinks.hdfs.useLocalTimeStamp = true
agent1.sinks.sink3.hdfs.path = /sx/logtable/%Y-%m-%d
agent1.sinks.sink3.hdfs.filePrefix = logevent
agent1.sinks.sink3.hdfs.rollInterval = 10
agent1.sinks.sink3.hdfs.rollSize = 134217728
agent1.sinks.sink3.hdfs.rollCount = 0

 
# Use a channel which buffers events in memory
agent1.channels.channel1.type = memory
agent1.channels.channel1.keep-alive = 120
agent1.channels.channel1.capacity = 500000
agent1.channels.channel1.transactionCapacity = 600

3.3 启动

D:\apache\apache-flume-1.9.0-bin\apache-flume-1.9.0-bin>.\bin\flume-ng agent --conf conf --conf-file  conf\log2file.conf --name agent1 -property flume.root.logger=INFO,console

D:\apache\apache-flume-1.9.0-bin\apache-flume-1.9.0-bin>.\bin\flume-ng agent --conf conf --conf-file conf\log2file.conf --name agent1 -property flume.root.logger=INFO,console

3.4 注意事项

1.需要确定读取日志文件的编码格式，默认读取格式为UTF-8，如果编码不为默认，需要手动修改，如果编码格式不正确，可能出现：

FATAL: Spool Directory source source1: { spoolDir: E://log }: Uncaught exception in SpoolDirectorySource thread. Restart or reconfigure Flume to continue processing.
java.nio.charset.MalformedInputException: Input length = 1

2.agent1.sinks.sink1.sink.directory = D://flume-collection，要注意配置的key为sink.directory，否则会出现

Directory may not be null

的错误

4.总结

本文主要基于Flume实现了日志的采集，本文案例不复杂，但由于基于windows实现的案例不多，笔者尽量把自己遇到的问题描述在博客中，包括编码问题和配置的注意事项，减少大家的试错成本。