Source: 负责产生数据,将数据“传入”到channel,一旦写入channel成功,source就认为数据已经可靠的传输。
Sink: sink不停的从属于自己的channel(此处有点类似于消息队列的消费者了)中获取数据,并且将数据写入到相应的后端中,如果数据写入失败则需要将相应的状况通知channel,让channel知道数据传输失败,下次获取时可以继续获取到这些数据;
Channel:在flume的架构中,channel负责对数据提供可靠性保证。任何一种channel(memory,file)都需要提供相应channel的transaction,保证实现如下几个操作:
doBegin: 事务开始;在source准备写入数据或sink准备读取数据之前调用;
doPut: 数据写入(生产者);由channelprocessor的processEvent或processEventBatch调用(rootcaller实际上是各种各样的source);
doTake: 数据读出(消费者);消费端(sink)调用,然后将数据做后端存储或转发处理;
doCommit: 数据操作(写入或读出消费)成功;
doRollback: 数据操作(写入或读出消费)失败,回滚;
doClose: 关闭事务;
主要类图如下图:
2.初始化
- public MaterializedConfiguration getConfiguration() {
- MaterializedConfiguration conf = new SimpleMaterializedConfiguration();
- FlumeConfiguration fconfig = getFlumeConfiguration();
- AgentConfiguration agentConf = fconfig.getConfigurationFor(getAgentName());
- if (agentConf != null) {
- Map<String, ChannelComponent> channelComponentMap = Maps.newHashMap();
- Map<String, SourceRunner> sourceRunnerMap = Maps.newHashMap();
- Map<String, SinkRunner> sinkRunnerMap = Maps.newHashMap();
- try {
- loadChannels(agentConf, channelComponentMap);
- loadSources(agentConf, channelComponentMap, sourceRunnerMap);
- loadSinks(agentConf, channelComponentMap, sinkRunnerMap);
- Set<String> channelNames = new HashSet<String>(channelComponentMap.keySet());
- for (String channelName : channelNames) {
- ChannelComponent channelComponent = channelComponentMap.get(channelName);
- if (channelComponent.components.isEmpty()) {
- LOGGER.warn(String.format("Channel %s has no components connected" +
- " and has been removed.", channelName));
- channelComponentMap.remove(channelName);
- Map<String, Channel> nameChannelMap =
- channelCache.get(channelComponent.channel.getClass());
- if (nameChannelMap != null) {
- nameChannelMap.remove(channelName);
- }
- } else {
- LOGGER.info(String.format("Channel %s connected to %s",
- channelName, channelComponent.components.toString()));
- conf.addChannel(channelName, channelComponent.channel);
- }
- }
- for (Map.Entry<String, SourceRunner> entry : sourceRunnerMap.entrySet()) {
- conf.addSourceRunner(entry.getKey(), entry.getValue());
- }
- for (Map.Entry<String, SinkRunner> entry : sinkRunnerMap.entrySet()) {
- conf.addSinkRunner(entry.getKey(), entry.getValue());
- }
- } catch (InstantiationException ex) {
- LOGGER.error("Failed to instantiate component", ex);
- } finally {
- channelComponentMap.clear();
- sourceRunnerMap.clear();
- sinkRunnerMap.clear();
- }
- } else {
- LOGGER.warn("No configuration found for this host:{}", getAgentName());
- }
- return conf;
- }
3.source的事务是通过
- public void processEvent(Event event) {
- event = interceptorChain.intercept(event);
- if (event == null) {
- return;
- }
- // Process required channels
- List<Channel> requiredChannels = selector.getRequiredChannels(event);
- for (Channel reqChannel : requiredChannels) {
- Transaction tx = reqChannel.getTransaction();
- Preconditions.checkNotNull(tx, "Transaction object must not be null");
- try {
- tx.begin();
- reqChannel.put(event);
- tx.commit();
- } catch (Throwable t) {
- tx.rollback();
- if (t instanceof Error) {
- LOG.error("Error while writing to required channel: " + reqChannel, t);
- throw (Error) t;
- } else if (t instanceof ChannelException) {
- throw (ChannelException) t;
- } else {
- throw new ChannelException("Unable to put event on required " +
- "channel: " + reqChannel, t);
- }
- } finally {
- if (tx != null) {
- tx.close();
- }
- }
- }
- // Process optional channels
- List<Channel> optionalChannels = selector.getOptionalChannels(event);
- for (Channel optChannel : optionalChannels) {
- Transaction tx = null;
- try {
- tx = optChannel.getTransaction();
- tx.begin();
- optChannel.put(event);
- tx.commit();
- } catch (Throwable t) {
- tx.rollback();
- LOG.error("Unable to put event on optional channel: " + optChannel, t);
- if (t instanceof Error) {
- throw (Error) t;
- }
- } finally {
- if (tx != null) {
- tx.close();
- }
- }
- }
- }
的processEventBatch或processEvent的调用实现的
4.sink事务的控制是在各自的process中实现
- public Status process() throws EventDeliveryException {
- Status result = Status.READY;
- Channel channel = getChannel();
- Transaction transaction = channel.getTransaction();
- Event event = null;
- try {
- transaction.begin();
- event = channel.take();
- if (event != null) {
- if (logger.isInfoEnabled()) {
- logger.info("Event: " + EventHelper.dumpEvent(event, maxBytesToLog));
- }
- } else {
- // No event found, request back-off semantics from the sink runner
- result = Status.BACKOFF;
- }
- transaction.commit();
- } catch (Exception ex) {
- transaction.rollback();
- throw new EventDeliveryException("Failed to log event: " + event, ex);
- } finally {
- transaction.close();
- }
- return result;
- }