Flume环境搭建_五种案例
http://flume.apache.org/FlumeUserGuide.html
A simple example
Here, we give an example configuration file, describing a single-node Flume deployment. This configuration lets a user generate events and subsequently logs them to the console.
# example.conf: A single-node Flume configuration
# Name the components on this agent
a1.sources = r1
a1.sinks = k1 a1.channels = c1 # Describe/configure the source a1.sources.r1.type = netcat a1.sources.r1.bind = localhost a1.sources.r1.port = 44444 # Describe the sink a1.sinks.k1.type = logger # Use a channel which buffers events in memory a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 # Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1
This configuration defines a single agent named a1. a1 has a source that listens for data on port 44444, a channel that buffers event data in memory, and a sink that logs event data to the console. The configuration file names the various components, then describes their types and configuration parameters. A given configuration file might define several named agents; when a given Flume process is launched a flag is passed telling it which named agent to manifest.
Given this configuration file, we can start Flume as follows:
$ bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger=INFO,console
Note that in a full deployment we would typically include one more option: --conf=<conf-dir>. The <conf-dir> directory would include a shell script flume-env.sh and potentially a log4j properties file. In this example, we pass a Java option to force Flume to log to the console and we go without a custom environment script.
From a separate terminal, we can then telnet port 44444 and send Flume an event:
$ telnet localhost 44444
Trying 127.0.0.1...
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.
Hello world! <ENTER>
OK
The original Flume terminal will output the event in a log message.
12/06/19 15:32:19 INFO source.NetcatSource: Source starting
12/06/19 15:32:19 INFO source.NetcatSource: Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444] 12/06/19 15:32:34 INFO sink.LoggerSink: Event: { headers:{} body: 48 65 6C 6C 6F 20 77 6F 72 6C 64 21 0D Hello world!. }
Congratulations - you’ve successfully configured and deployed a Flume agent! Subsequent sections cover agent configuration in much more detail.
以下为具体搭建流程
Flume搭建_案例一:单个Flume
安装node2上
1. 上传到/home/tools,解压,解压后移动到/home下
2. 重命名,并修改flume-env.sh
![](https://i-blog.csdnimg.cn/blog_migrate/c517ae2abcd01e352c621ac57aea1390.png)
![](https://i-blog.csdnimg.cn/blog_migrate/efcb3559ea3628de5241f4aceb602ceb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/930b6cacfdc5e28f7c25a090c12aec40.png)
![](https://i-blog.csdnimg.cn/blog_migrate/b80f47aff9e4f04713a69aa1d727c4c7.png)
![](https://i-blog.csdnimg.cn/blog_migrate/68eec79010c3d1e1afb96399b40bfc5f.png)
![](https://i-blog.csdnimg.cn/blog_migrate/ae3d4c76b2c4f65b5bd8ee3db1266654.png)
![](https://i-blog.csdnimg.cn/blog_migrate/a74fd26df6c5668a75d81ea30d531b18.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e09e9cb7a9c9f1fae020cd1d1836e689.png)
![](https://i-blog.csdnimg.cn/blog_migrate/8ec6fd8ec62d4289e74b3e4a28115432.png)
![](https://i-blog.csdnimg.cn/blog_migrate/7ae723d4ff62dda892040229eeff9577.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5a00a2794b2e8aef5796dd60ebfd5386.png)
Flume搭建_案例二:两个Flume做集群
![](https://i-blog.csdnimg.cn/blog_migrate/22866c1314d4408ab8a1f5d09d9276bb.png)
MemoryChanel配置
capacity:默认该通道中最大的可以存储的event数量是100,
trasactionCapacity:每次最大可以从source中拿到或者送到sink中的event数量也是100
keep-alive:event添加到通道中或者移出的允许时间
byte**:即event的字节量的限制,只包括eventbody
![](https://i-blog.csdnimg.cn/blog_migrate/bacf9827a50e53ba70d24faaa50518ea.png)
![](https://i-blog.csdnimg.cn/blog_migrate/c089ee11bdd0be9073a89626de0e7f72.png)
![](https://i-blog.csdnimg.cn/blog_migrate/8af34f2e6d7b288ded8fdddd54093664.png)
![](https://i-blog.csdnimg.cn/blog_migrate/33b0ab846a326a51c6aa6ded2afcc670.png)
先启动node02的Flume
flume-ng agent -n a1 -c conf -f avro.conf -Dflume.root.logger=INFO,console
flume-ng agent -n a1 -c conf -f /home/test_flume/flume22 -Dflume.root.logger=INFO,console
再启动node01的Flume
flume-ng agent -n a1 -c conf -f simple.conf2 -Dflume.root.logger=INFO,console
flume-ng agent -n a1 -c conf -f /home/test_flume/flume21 -Dflume.root.logger=INFO,console
![](https://i-blog.csdnimg.cn/blog_migrate/a31f3abbe4ad474edcaa1e58335abbf5.png)
![](https://i-blog.csdnimg.cn/blog_migrate/1180b77c3db75cd0a2df25b96e737e1f.png)
![](https://i-blog.csdnimg.cn/blog_migrate/d6a4a9eaf9d61469db100c49fa98a16a.png)
![](https://i-blog.csdnimg.cn/blog_migrate/1167f329be3e056d99d1d96dbe0ab59a.png)
![](https://i-blog.csdnimg.cn/blog_migrate/08f164d3fd51a7648503fd0153d2f71b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/36772f1609c03f2fece5ffc300ab1433.png)
Flume搭建_案例三:如何监控一个文件的变化?
![](https://i-blog.csdnimg.cn/blog_migrate/c7a9e56296ba39c84dda884899c1b3c2.png)
![](https://i-blog.csdnimg.cn/blog_migrate/69cb3b15067df02e82e181b58b27d164.png)
启动Flume
flume-ng agent -n a1 -c conf -f exec.conf -Dflume.root.logger=INFO,console
flume-ng agent -n a1 -c conf -f /home/test_flume/flume3 -Dflume.root.logger=INFO,console
![](https://i-blog.csdnimg.cn/blog_migrate/bbd53efd7a63211145e07058c709d059.png)
![](https://i-blog.csdnimg.cn/blog_migrate/40d89452582640ad239874b55c25b984.png)
Flume搭建_案例四: 如何监控一个文件:目录的变化?
![](https://i-blog.csdnimg.cn/blog_migrate/2eb1ddf2667f8ab9b864a3ef6930665b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/6e311ecbf98d18dac54f3f76515811d7.png)
![](https://i-blog.csdnimg.cn/blog_migrate/fcf458f575dc3f1e35ca1128f82dad87.png)
![](https://i-blog.csdnimg.cn/blog_migrate/0f5249eb52a2ec029223be0dcc01f2cc.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e041cd3e38ad8d565d6c463d917e0265.png)
![](https://i-blog.csdnimg.cn/blog_migrate/4e638189ac4b6b6151032e99be5369de.png)
Flume搭建_案例五: 如何定义一个HDFS类型的Sink?
![](https://i-blog.csdnimg.cn/blog_migrate/8f2c782e5dea392f378c72fb1c814fdc.png)
Flume搭建_案例五_配置项解读
![](https://i-blog.csdnimg.cn/blog_migrate/9ccc7303090b295df317ba2656eb40a9.png)
![](https://i-blog.csdnimg.cn/blog_migrate/f48eca68416eb2c29482bdeab8a1569c.png)
hdfs.rollInterval | 30 | Number of seconds to wait before rolling current file (0 = never roll based on time interval) |
hdfs.rollSize | 1024 | File size to trigger roll, in bytes (0: never roll based on file size) |
hdfs.rollCount | 10 | Number of events written to file before it rolled (0 = never roll based on number of events) |
4. 多长时间没有操作,Flume将一个临时文件生成新文件?
hdfs.idleTimeout | 0 | Timeout after which inactive files get closed (0 = disable automatic closing of idle files) |
5. 多长时间生成一个新的目录?(比如每10s生成一个新的目录)
四舍五入,没有五入,只有四舍
(比如57分划分为55分,5,6,7,8,9在一个目录,10,11,12,13,14在一个目录)
hdfs.round | false | Should the timestamp be rounded down (if true, affects all time based escape sequences except %t) |
hdfs.roundValue | 1 | Rounded down to the highest multiple of this (in the unit configured using hdfs.roundUnit), less than current time. |
hdfs.roundUnit | second | The unit of the round down value - second, minute or hour. |
![](https://i-blog.csdnimg.cn/blog_migrate/9ce89fd0ee158e90c819e0244a2b16de.png)
![](https://i-blog.csdnimg.cn/blog_migrate/3011b1916b7621fb8929526337e32554.png)