flume1.6自定义elasticsearch2.3的sink插件

最新推荐文章于 2022-09-02 14:41:27 发布
最新推荐文章于 2022-09-02 14:41:27 发布
阅读量1.7k 收藏 2
点赞数
分类专栏： # flume 文章标签： flume
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/liuxiao723846/article/details/78683544
版权

目前官方Flume最新的版本仍然不支持Elasticsearch2.x版本，flume的老版本（1.6）只支持Elasticsearch1.7.x的版本，因为Elasticsearch2.x版本做了比较大的改动，很多API都已经废弃不用了。

在github上找了一个项目：https://github.com/lucidfrontier45/ElasticsearchSink2  也是按照flume源码中sink的模式编写的，其实也很简单。最近，公司有个项目也是需要将日志手机到es上，供实时查询，索性自己写了一个。


一、整体架构：

1、线上业务代码将日志数据发送到kafka，数据结构是pb，kafka中存放的是byte流；

2、flume-source对接kafka，自定义一个intercept，将byte流解成结构化数据；

3、自定义es-sink，将结构化数据存放到es中。

二、代码：

1、pb：

option java_package = "XXX.base.proto";
option java_outer_classname="ApiLogPB";

message ApiLog {
    optional string puid = 1;
    optional string uId = 2;
    optional string reqId = 3;
    optional int32 fNum = 4;
    optional int32 cost = 5;
    optional string chId = 6 [ default = "default"];
    optional string strategy = 7;
    repeated int64 recId = 8;
    optional string txt = 9;
    optional string vedio = 10;
    optional string gallery = 11;
    optional string pMap = 12;
    optional string paramMap = 13;
}

2、线上发送数据到kafka： 

public Integer call() throws Exception {
		Builder newBuilder = ApiLogPB.ApiLog.newBuilder();
		
		newBuilder.setReqId(reqId);
		newBuilder.setUId(uId);
		newBuilder.setPuid(puid);
		newBuilder.setChId(channelId);
		newBuilder.setFNum(fNum);
		newBuilder.setCost(Integer.parseInt((System.currentTimeMillis()-a)+""));
		newBuilder.setStrategy(personalResponse.getStrategy());
		
		List<Long> recIdSet = personalResponse.getRecIdSet();
		for (Long fid:recIdSet) {
			newBuilder.addRecId(fid);
		}
		
		Map<String, List<Long>> typeFeeds = personalResponse.getTypeFeeds();
		List<Long> newList = typeFeeds.get("NEWS");
		newBuilder.setTxt(newList==null ? "":newList.toString());
		List<Long> garleryList = typeFeeds.get("GALLERY");
		newBuilder.setGallery(garleryList==null ? "":garleryList.toString());
		List<Long> vedioList = typeFeeds.get("VIDEO");
		newBuilder.setVedio(vedioList==null?"":vedioList.toString());
		
		Map<String, List<Long>> mutilFeeds = personalResponse.getMutilFeeds();
		StringBuilder pSb = new StringBuilder();
		Set<Entry<String, List<Long>>> entrySet = mutilFeeds.entrySet();
		int i = 0;
		for (Entry<String, List<Long>> engry : entrySet) {
			String key = engry.getKey();
			List<Long> value = engry.getValue();
			if (CollectionUtils.isNotEmpty(value)) {
				String v1 = StringUtils.strip(value.toString(),"[]");
				
				pSb.append(key).append(":").append(v1);
				if (i<entrySet.size()-1) {
					pSb.append(";");
				}
				i++;
			}
		}
		newBuilder.setPMap(pSb.toString());
		//newBuilder.setPMap(mutilFeeds.toString());
		
		newBuilder.setParamMap(paraMap.toString());
		
		KafkaDao.getInstance().sendMessage(uId,newBuilder.build().toByteArray());
		return null;
	}

3、日志实体类： 

package XXX.log.common.entity;

import java.util.HashMap;
import java.util.List;
import java.util.Map;
import com.google.protobuf.InvalidProtocolBufferException;
import com.iqiyi.ttbrain.base.proto.ApiLogPB;
import com.iqiyi.ttbrain.base.proto.ApiLogPB.ApiLog;


public class ApiEntity {
	public static final int P_N_SIZE = 50;
	private static final String flag = "\t";
	
	
	private String reqId="";
	private String uid="";
	private String ppuid="";
	private String channel="";
	private int feedNum=10;
	private int cost=0;
	private String strategy="";
	
	private String timeStamp="";
	private String host="";
	
	private String recFeedId="";
	private String txt="";
	private String gallery="";
	private String vedio="";
	
	private String pMap = "";
	private String paraMap = "";
	
	
	@Override
	public String toString() {
		StringBuilder sb = new StringBuilder(768);
		sb.append(reqId).append(flag).append(uid).append(flag).append(ppuid).append(flag).append(channel).append(flag)
		.append(feedNum).append(flag).append(cost).append(flag).append(strategy).append(flag)
		
		.append(timeStamp).append(flag).append(host).append(flag)
		
		.append(recFeedId).append(flag)
		.append(txt).append(flag).append(gallery).append(flag).append(vedio).append(flag)
		
		.append(pMap).append(flag)
		.append(paraMap).append(flag)
	 
		.append("end");
		
		return sb.toString();
	}
	
	public static ApiEntity parseFromPB(byte[] bytes) throws InvalidProtocolBufferException{
		ApiEntity an = null;
		if (bytes != null) {
			an = new ApiEntity();
			ApiLog apiLog = ApiLogPB.ApiLog.parseFrom(bytes);
			
			an.setReqId(apiLog.getReqId());
			an.setUid(apiLog.getUId());
			an.setPpuid(apiLog.getPuid());
			an.setChannel(apiLog.getChId());
			an.setFeedNum(apiLog.getFNum());
			an.setCost(apiLog.getCost());
			an.setStrategy(apiLog.getStrategy());
			
			StringBuilder sb = new StringBuilder();
			List<Long> recIdList = apiLog.getRecIdList();
			for (int i=0;i<recIdList.size();i++) {
				sb.append(recIdList.get(i));
				if (i<recIdList.size()-1) {
					sb.append(",");
				}
			}
			an.setRecFeedId(sb.toString());
			
			an.setTxt(apiLog.getTxt());
			an.setGallery(apiLog.getGallery());
			an.setVedio(apiLog.getVedio());
			
			an.setParaMap(apiLog.getParamMap());
			an.setpMap(apiLog.getPMap());
		}
		
		return an;
	}
	
	public static Map<String,Object> parseToMap(String line) {
		Map<String,Object> map = new HashMap<>();
		if (line != null) {
			String[] split = line.split(flag);
			if (split.length>=15) {
				map.put("reqId", split[0]);
				map.put("uid", split[1]);
				map.put("ppuid", split[2]);
				map.put("channel", split[3]);
				map.put("feedNum", split[4]);
				map.put("cost", split[5]);
				map.put("strategy", split[6]);
				map.put("timeStamp", split[7]);
				map.put("host", split[8]);
				map.put("recFeedId", split[9]);
				map.put("txt", split[10]);
				map.put("gallery", split[11]);
				map.put("vedio", split[12]);
				map.put("pMap", split[13]);
				map.put("paraMap", split[14]);
			}
		}
		/*BeanMap beanMap = BeanMap.create(an);  
        for (Object key : beanMap.keySet()) {
        	map.put((String)key, beanMap.get(key));
        }*/

		return map;
	}
	
	
	public static void main(String...strings) {
		ApiEntity le = new ApiEntity();
		le.setChannel("default");
		le.setParaMap("sdfsdfdsfdsfds");
		le.setpMap("12321321321321");
		String line = le.toString();
		System.out.println(line);
		
		String[] split = line.split(flag);
		System.out.println(split.length);
		
		Map<String, Object> parseToMap = parseToMap(line);
		System.out.println(parseToMap);
		
		System.out.println(split[15]);
	}
}

4、flume-intercept： 

package XXX.log.flume.interceptor;

import java.util.List;
import java.util.Map;
import org.apache.flume.Context;
import org.apache.flume.Event;
import org.apache.flume.interceptor.Interceptor;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.google.common.collect.Lists;
import com.iqiyi.ttbrain.log.common.entity.ApiEntity;


public class ApiInterceptor implements Interceptor {
	private static final Logger logger = LoggerFactory.getLogger(ApiInterceptor.class);
	
	@Override
	public void close() {
		// TODO Auto-generated method stub
		logger.info("flume ApiInterceptor is close");
	}

	@Override
	public void initialize() {
		// TODO Auto-generated method stub
		logger.info("flume ApiInterceptor is initialize");
	}

	@Override
	public Event intercept(Event event) {
		try {
			Map<String, String> headers = event.getHeaders();
			byte[] body = event.getBody();
			if (body != null) {
				ApiEntity apiLog = null;
				try{
				apiLog = ApiEntity.parseFromPB(body);
				}catch(Exception e){
					logger.info("apiLog:{}",apiLog);
				}
				if (apiLog != null) {
					String hostName = headers.get("hostname");
					String timeStamp = headers.get("timestamp");
					apiLog.setHost(hostName);
					apiLog.setTimeStamp(timeStamp);
//					logger.info(apiLog.toString());
					event.setBody(apiLog.toString().getBytes());
					return event;
				}
			}
		} catch (Exception e ) {
			logger.error("intercept:",e);
		}
		return null;
	}
	
	@Override
	public List<Event> intercept(List<Event> events) {
		List<Event> intercepted = Lists.newArrayListWithCapacity(events.size());
		for (Event event : events) {
            Event interceptedEvent = intercept(event);
            if (interceptedEvent != null) {
                intercepted.add(interceptedEvent);
            }
        }
        return intercepted;
	}
	
	public static class Builder implements Interceptor.Builder {
        //使用Builder初始化Interceptor
        @Override
        public Interceptor build() {
            return new ApiInterceptor();
        }

		@Override
		public void configure(Context arg0) {
			// TODO Auto-generated method stub
			
		}
    }
}
5、esDao工具类： 

package XXX.log.flume.sink.db;

import java.net.InetSocketAddress;
import java.util.ArrayList;
import java.util.Date;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.elasticsearch.action.bulk.BulkRequestBuilder;
import org.elasticsearch.action.bulk.BulkResponse;
import org.elasticsearch.action.index.IndexResponse;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.transport.InetSocketTransportAddress;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;


public class EsDao {
	private static final Logger logger = LoggerFactory.getLogger(EsDao.class);
	// ip:port
	private static final String clusterHost = "1.1.1.1:9300";
	private static final String clusterName = "test";

	private static TransportClient transportClient = null;

	static {
		Settings settings = Settings.settingsBuilder()
				.put("cluster.name", clusterName).build();

		transportClient = TransportClient.builder().settings(settings).build();

		String[] hostNames = clusterHost.split(",");
		InetSocketTransportAddress[] serverAddresses = new InetSocketTransportAddress[hostNames.length];
		for (int i = 0; i < hostNames.length; i++) {
			String[] hostPort = hostNames[i].trim().split(":");
			String host = hostPort[0].trim();
			int port = hostPort.length == 2 ? Integer.parseInt(hostPort[1]
					.trim()) : 9300;
			serverAddresses[i] = new InetSocketTransportAddress(
					new InetSocketAddress(host, port));
		}

		for (InetSocketTransportAddress host : serverAddresses) {
			transportClient.addTransportAddress(host);
		}

	}

	private EsDao() {
	}
	
	public static void closeClient() {
		transportClient.close();
	}
	
	
	/**
	 * bulk——在一个请求中添加、更新和删除多个文档
	 * @param indexName
	 * @param indexType
	 * @param datas
	 * @return
	 */
	public static boolean bulk(String indexName,String indexType,List<Map<String,Object>> datas){
		BulkRequestBuilder bulkRequestBuilder = transportClient.prepareBulk();
		for (Map<String,Object> map: datas) {
			bulkRequestBuilder.add(transportClient.prepareIndex(indexName,
					indexType).setSource(map));
		}
		BulkResponse bulkResponse = bulkRequestBuilder.execute().actionGet();
		
		return bulkResponse.hasFailures();
    }
}

6、esSink： 

package XXX.log.flume.sink;

import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.apache.flume.Channel;
import org.apache.flume.Context;
import org.apache.flume.Event;
import org.apache.flume.EventDeliveryException;
import org.apache.flume.Transaction;
import org.apache.flume.conf.Configurable;
import org.apache.flume.sink.AbstractSink;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.google.common.collect.Lists;
import com.iqiyi.ttbrain.log.flume.sink.db.EsDao;


public class EsSink extends AbstractSink implements Configurable {
	
	private static final Logger logger = LoggerFactory.getLogger(EsSink.class);
	
	private String indexName = "api";
	private String indexType =  "api_type";
	private int batchSize = 100;
	
	
	public EsSink() {
		logger.info("EsSink start...");
    }
	
	@Override
    public void start() {
        super.start();
        
    }
	
	@Override
    public void stop() {
        super.stop();
        
        //DataSourceUtils.closeDs();
    }
	
	@Override
	public Status process() throws EventDeliveryException {
		Status result = Status.READY;
        Transaction transaction = null;
        Event event = null;
        String content = "";
        List<Map<String,Object>> actions = Lists.newArrayList();
        
        try {
        	//flume
        	Channel channel = getChannel();
        	transaction = channel.getTransaction();
        	transaction.begin();
        	
        	for (int i = 0; i < batchSize; i++) {
        		event = channel.take();
            	if (event == null) {
            		result = Status.BACKOFF;
            		break;
            	} else {
            		content = new String(event.getBody(),"UTF-8");
            		String[] split = content.split("\t");
            		if (split.length < 15) {
            			continue;
            		}
            		//logger.info(content);
            		actions.add(generateMap(content));
            	}
        	}
        	
        	if (actions.size() > 0) {
                //logger.info(actions.get(0));
        		EsDao.bulk(indexName, indexType, actions);
            }
        	
            transaction.commit();
        } catch (Throwable e) {
            /*try {
            	if (transaction != null) {
            		transaction.rollback();
            	}
            } catch (Exception e2) {
            	logger.error("flume transaction rollback error.", e2);
            }*/
            logger.error("Failed to commit flume transaction," +"Transaction rolled back.", e);
        } finally {
        	if (transaction != null) {
        		transaction.close();
        	}
        }
        return result;
	}

	@Override
	public void configure(Context context) {
		
	}
	
	private static Map<String,Object> generateMap(String line) {
		Map<String,Object> map = new HashMap<>();
		if (line != null) {
			String[] split = line.split("\t");
			if (split.length>=15) {
				map.put("reqId", split[0]);
				map.put("uid", split[1]);
				map.put("ppuid", split[2]);
				map.put("channel", split[3]);
				map.put("feedNum", split[4]);
				map.put("cost", split[5]);
				map.put("strategy", split[6]);
				map.put("timeStamp", split[7]);
				map.put("host", split[8]);
				map.put("recFeedId", split[9]);
				map.put("txt", split[10]);
				map.put("gallery", split[11]);
				map.put("vedio", split[12]);
				map.put("pMap", split[13]);
				map.put("paraMap", split[14]);
			}
		}
		/*BeanMap beanMap = BeanMap.create(an);  
        for (Object key : beanMap.keySet()) {
        	map.put((String)key, beanMap.get(key));
        }*/

		return map;
	}
}

三、flume配置：
1）将上面的代码打包，放到flume的lib下


2）flume的配置如下：

agent1.sources = logsource
agent1.channels = mc1 mc2
agent1.sinks = avro-sink sink2


agent1.sources.logsource.channels = mc1 mc2
agent1.sinks.avro-sink.channel = mc1
agent1.sinks.sink2.channel = mc2


#source
agent1.sources.logsource.type = org.apache.flume.source.kafka.KafkaSource
agent1.sources.logsource.zookeeperConnect = ttAlgorithm-kafka-online001-jyltqbs.qiyi.virtual:2181,ttAlgorithm-kafka-online002-jyltqbs.qiyi.virtual:2181,ttAlgorithm-kafka-online003-jyltqbs.qiyi.virtual:2181,ttAlgorithm-kafka-online004-jyltqbs.qiyi.virtual:2181,ttAlgorithm-kafka-online005-jyltqbs.qiyi.virtual:2181
agent1.sources.logsource.topic = topic_predict
agent1.sources.logsource.groupId = flume
agent1.sources.logsource.kafka.consumer.timeout.ms = 100

#interceptor
agent1.sources.logsource.interceptors=filt2 filt3 filt4
agent1.sources.logsource.interceptors.filt2.type=host
agent1.sources.logsource.interceptors.filt2.hostHeader=hostname
agent1.sources.logsource.interceptors.filt2.useIP=true
agent1.sources.logsource.interceptors.filt3.type=timestamp
agent1.sources.logsource.interceptors.filt4.type=com.iqiyi.ttbrain.log.flume.interceptor.PredictInterceptor$Builder

agent1.sources.logsource.selector.type = multiplexing
agent1.sources.logsource.selector.header = isCool
agent1.sources.logsource.selector.mapping.0 = mc1
agent1.sources.logsource.selector.mapping.1 = mc2
agent1.sources.logsource.selector.default = mc1

#channel1
agent1.channels.mc1.type = memory
agent1.channels.mc1.capacity = 10000
agent1.channels.mc1.transactionCapacity = 10000
agent1.channels.mc1.keep-alive = 60

#channel2
agent1.channels.mc2.type = memory
agent1.channels.mc2.capacity = 10000
agent1.channels.mc2.transactionCapacity = 10000
agent1.channels.mc2.keep-alive = 60


#sink1
#agent1.sinks.avro-sink.type = file_roll
#agent1.sinks.avro-sink.sink.directory = /data/mysink
#agent1.sinks.avro-sink.sink.rollInterval = 10000000
agent1.sinks.avro-sink.type = hdfs
agent1.sinks.avro-sink.hdfs.path = hdfs://hadoop-jy-namenode/data/qytt/flume/ttengine_predict/dt=%Y-%m-%d/hour=%H/
agent1.sinks.avro-sink.hdfs.writeFormat = Text
agent1.sinks.avro-sink.hdfs.fileType = DataStream
agent1.sinks.avro-sink.hdfs.fileSuffix = .log
agent1.sinks.avro-sink.hdfs.filePrefix = %Y-%m-%d_%H
agent1.sinks.avro-sink.hdfs.rollInterval = 3600
agent1.sinks.avro-sink.hdfs.rollSize = 0
agent1.sinks.avro-sink.hdfs.rollCount = 0
agent1.sinks.avro-sink.hdfs.batchSize = 1000
agent1.sinks.avro-sink.hdfs.callTimeout = 60000
agent1.sinks.avro-sink.hdfs.appendTimeout = 60000

#sink2
agent1.sinks.sink2.type = hdfs
agent1.sinks.sink2.hdfs.path = hdfs://hadoop-jy-namenode/data/qytt/flume/ttengine_predict_cool_start/dt=%Y-%m-%d/hour=%H/
agent1.sinks.sink2.hdfs.writeFormat = Text
agent1.sinks.sink2.hdfs.fileType = DataStream
agent1.sinks.sink2.hdfs.fileSuffix = .log
agent1.sinks.sink2.hdfs.filePrefix = %Y-%m-%d_%H
agent1.sinks.sink2.hdfs.rollInterval = 3600
agent1.sinks.sink2.hdfs.rollSize = 0
agent1.sinks.sink2.hdfs.rollCount = 0
agent1.sinks.sink2.hdfs.batchSize = 1000
agent1.sinks.sink2.hdfs.callTimeout = 60000
agent1.sinks.sink2.hdfs.appendTimeout = 60000

3）注意事项：

flume1.6和es2.3版本有jar冲突问题，在编写flume-es-sink时，需要引入：

<dependency>  
        <groupId>org.elasticsearch</groupId>  
        <artifactId>elasticsearch</artifactId>  
        <version>2.3.2</version>  
</dependency>
然后发现会有java.lang.NoSuchMethodError:com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concu‌rrent/Executor 报错，和 Exception in thread "main" java.lang.NoSuchFieldError: FAIL_ON_SYMBOL_HASH_OVERFLOW 两个问题，解决方法，在maven中加上 

<dependency>  
    <groupId>com.google.guava</groupId>  
    <artifactId>guava</artifactId>  
    <version>18.0</version>  
</dependency>
和 

<dependency>    
     <groupId>com.fasterxml.jackson.core</groupId>    
     <artifactId>jackson-core</artifactId>    
     <version>2.6.2</version>    
 </dependency>


同时，在把jar包放到flume的lib中，启动flume也会报上面的错误，这时需要把flume的lib目录下guava-11.0.2.jar和jackson-core-2.3.1.jar 两个包去掉。

具体可以参考：http://blog.csdn.net/liuxiao723846/article/details/78531916


参考文档：

http://tech.lede.com/2017/02/08/rd/server/flumeToEs/

赶路人儿
关注
  • 0
    点赞
  • 踩
  • 2
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
专栏目录
flume自定义sink写入es
lvhong1234的博客
07-21 1049
flume elasticsearch sink
flume + elasticsearch
洛冰翼的专栏
11-24 1201
flume + elasticsearch flume-1.9.0 + elasticsearch-7.7.1 + kibana-7.7.1 下载地址 flume-ng http://flume.apache.org/download.html elasticsearch https://www.elastic.co/cn/downloads/past-releases#elasticsearch kibana https://www.elastic.co/c...
Flume+ElasticSearch+Kibana实现分布式日志收集
gszzb的博客
01-27 765
Flume+ElasticSearch+Kibana实现分布式日志收集前言背景Flume的优缺点优点缺点Flume简介概述核心组件ES简介Kibana简介环境准备搭建步骤安装Flume1.7.0安装ES1.5.0安装Kibana4.0.1联调FAQ参考文档 前言 本人小白一枚，第一次分享，想把自己懂的一丢丢东西分享出来，希望可以帮到一些朋友。本人只做了一些浅显的研究，如发现有问题，欢迎指正~ 背景 许多项目是需要分布式部署的，日志分布在不同服务器上，查看管理日志比较痛苦。当前已有几种成熟的方案可以实现分布式
flume-ng-elasticsearch-sink-1.8.0.jar
09-18
在Flume中的ElasticSearchSink支持Flume与Elasticsearch整合，可以将Flume采集的数据传输到Elasticsearch中，Flume版本：1.8.0，ElasticSearchSink版本：6.2.4
flume-logstash-sink:Flume sink 使用 logstash V2 格式将日志事件发送到 ES 集群
07-07
水槽-logstash-sink Flume sink 使用 logstash V2 格式将日志事件发送到 ES 集群 看到 要启用事件序列化器，您需要将此行添加到水槽接收器配置中： [agent_name].sinks.[sink_name].indexNameBuilder = com.gilt.flume.elasticsearch.TimeBasedIndexNameBuilderV12 并设置序列化程序： [agent_name].sinks.[sink_name].serializer = com.gilt.flume.elasticsearch.ElasticSearchLogStashEventSerializerV12 您还需要将 jar 添加到您的水槽 lib 文件夹中，以便可以访问这些类。
flume1.6+es5.5.1
08-22
标题 "flume1.6+es5.5.1" 暗示我们要讨论的是Apache Flume 1.6版本与Elasticsearch (ES) 5.5.1版本的集成。Flume是Apache Hadoop项目的一个子项目，主要用于收集、聚合和移动大量日志数据。而Elasticsearch是一种...
flume1.6+es551
08-22
本文将深入探讨Flume 1.6版本与Elasticsearch 5.5.1的集成，以及如何利用`flume-ng-elasticsearch-sink`实现数据流的高效导入。 Flume的核心概念包括源头（Source）、通道（Channel）和接收器（Sink）。源头负责从...
es1.6_es_sink
08-24
标题“es1.6_es_sink”和描述“es1.6 elasticsearch5 sink”都指向一个关键主题，即使用Flume 1.6版本将数据流传输到Elasticsearch 5.0版本的接收器（sink）。Flume是Apache开发的一个分布式、可靠且可用于有效收集...
es551+flume1.6
08-22
标题“es551+flume1.6”和描述“es5.5.1+flume1.6”指的是Elasticsearch版本5.5.1与Apache Flume版本1.6的集成应用。这两个组件在大数据处理和日志收集领域中发挥着重要作用。 **Elasticsearch 5.5.1** Elastic...
flume-ng-elasticsearch-sink-1.9.0.jar
09-19
flume1.9 sink支持elasticsearch6.x；采用源码重构的方式；个人测试功能完善，但未投入生产使用，请读者自行考虑慎重使用
flume1.8.0和elasticsearch5.2.6整合
12-28
flume1.8.0和elasticsearch5.2.6整合,flume最新版本目前只支持elasticsearch1.x版本.elasticsearch更新迭代非常快，需要修改适配代码。
Flume 根据字段进行路由插入 ES
04-26
基于Flume-1.8.0，ES-5.6.11 ， 根据某个字段创建 type ,进行路由批量插入 ES，使用redis 缓存已从MySQL 查出的转换数据
Flume1.8+Elasticsearch6.2.4+Kibana6.2.4项目linux搭建
11-17
操作系统：CentOS 7.1 Java版本：1.8 Elasticsearch 6.2.4 Flume 1.8 一步一步的搭建一步一步写的
flume+es+kibana日志系统
m0_54883970的博客
09-02 1491
为了项目做日志监控，日至查询分析目前SIT IP监控ES存储健康（相当于NOSQL数据库）监控ES集群性能日志查看页面。
Flume或Kafka和Elasticsearch整合
小强签名设计 的博客
10-11 3951
前言：我这里是在Centos7.2 64位单机做的测试 安装elasticsearch的时候不应该安装在root用户下，否则启动会报错： Exception in thread "main" java.lang.RuntimeException: don't run elasticsearch as root. at org.elasticsearch.bootstrap.
flume采集数据导入elasticsearch 配置
热门推荐
u012085379的专栏
02-25 1万+
Flume往elasticsearch转发数据的详细配置。
flume之自定义sink组件
赶路人儿
09-29 5121
flume内部提供了很多种sink，如：logger、file_roll、avro、hdfs、kafak、es等，方便直接将event数据对接到本地磁盘、或者其他第三方存储中。有的时候，我们需要自定义source，来完成特殊需求。本文介绍如何开发自定义sink，来实现将event数据存储到Mysql。 1、pom.xml   &lt;?xml version="1.0"?&gt; &lt;...
如何下载flume elasticsearch插件
最新发布
05-12
您可以按照以下步骤下载Flume Elasticsearch插件： 1. 首先，您需要从https://github.com/apache/flume/releases下载Flume的最新版本。 2. 您还需要下载Elasticsearch插件的源代码。您可以从https://github.com/apache/flume/tree/trunk/flume-ng-sinks/flume-ng-elasticsearch-sink下载源代码。 3. 下载源代码后，您需要编译并生成JAR文件。您可以使用以下命令完成此操作： ``` $ cd flume-ng-elasticsearch-sink $ mvn clean package ``` 4. 执行以上命令后，将在target目录中生成一个名为flume-ng-elasticsearch-sink-<version>.jar的JAR文件。将此文件复制到Flume的lib目录中。 5. 最后，在Flume的配置文件中添加Elasticsearch插件的配置。例如： ``` agent.sources = ... agent.sinks = elasticsearch agent.channels = ... # Elasticsearch Sink agent.sinks.elasticsearch.type = org.apache.flume.sink.elasticsearch.ElasticSearchSink agent.sinks.elasticsearch.hostNames = localhost:9300 agent.sinks.elasticsearch.indexName = myindex agent.sinks.elasticsearch.indexType = mytype # Channel agent.channels.memory.type = memory agent.channels.memory.capacity = 10000 ``` 6. 启动Flume代理并验证Elasticsearch插件是否正常工作。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

赶路人儿

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值