SparkFlumeEvent:spark streaming连接flume，从SparkFlumeEvent中获取记录内容

最新推荐文章于 2021-02-21 19:06:21 发布

KingBoxing

最新推荐文章于 2021-02-21 19:06:21 发布

阅读量3.7k

点赞数

分类专栏： spark

本文链接：https://blog.csdn.net/aaa1117a8w5s6d/article/details/43086883

版权

spark 专栏收录该内容

12 篇文章 0 订阅

订阅专栏

JavaReceiverInputDStream<SparkFlumeEvent> flumeStream =  FlumeUtils.createStream(jssc, args[0], Integer.parseInt(args[1]));
	    
		JavaDStream<String> words = flumeStream
				.flatMap(new FlatMapFunction<SparkFlumeEvent, String>() {
					@Override
					public Iterable<String> call(SparkFlumeEvent t)
							throws Exception {
						System.out.println("t.event():"+ t.event());
						System.out.println(new String(t.event().getBody().array()));
						FlumeVo vo = (FlumeVo) JSON.parseObject(t.event().toString(),FlumeVo.class);
						System.out.println(vo.getBody().getBytes());						
						return Arrays.asList(t.event().toString().split(" "));
					}
				});
		words.print();

spark streaming连接flume，得到的是SparkFlumeEvent对象，但是怎么能从SparkFlumeEvent对象里得到flume传过来的数据呢，开始我各种get函数去试，但是都没有get出来，t.event()输出的内容是：

{"headers": {}, "body": {"bytes": "[10.4.1.24:nginx]"UfaNode" "123.134.176.33" "10.4.2.30:31001" "[24/Jan/2015:13:53:54 +0800]" "0.034" "0.006" "-" "200" "-" "65536" "190" "xiaomi" "appback" "-" "POST /xiaomi/upload_block_chunk?chunk_pos=524288&body_sum=-39296692&upload_id=066af4a9_7mcrh73iSzu%28kzfGK%28p9Xho3qxtK58xo0lUldabQt1RuCMczzYM5YupnOVWFy133D0g%2959%29T5zWwUnrpyAhoNQ-0_2 HTTP/1.1" "MI 2S/MI 2S; MIUI/4.12.5 V5" "101.71.27.24""}}

结果我就建了一个对象FlumeVo来解析这个字符串，但是放到线上环境后又出问题了，因为flume传过来的真实数据中是有引号的，所以JSON解析异常，没办法又得重新尝试，在百度查了半天没结果，还得靠强大的google，给大家推荐一个网址：http://www.fqgoogle.com/，解决办法就是上面的：