Flume中channel选择器(selector.type配置)必须实现ChannelSelector接口,实现了该接口的类主要作用是告诉Source中接收到的Event应该发送到哪些Channel,在Flume中主要由两个实现方式:
1,复用,实现类:MultiplexingChannelSelector
2,复制,实现类:ReplicatingChannelSelector
如下:
ChannelSelector接口两个主要的方法是:
//获取必选的Channel列表
public List<Channel> getRequiredChannels(Event event);
//获取可选的Channel列表
public List<Channel> getOptionalChannels(Event event);
ReplicatingChannelSelector (所有Channel默认的方式)
属性名 | 默认 | 描述 |
---|---|---|
selector.type | replicating | 组件名:replicating |
selector.optional | – | 标记哪些Channels是可选的 |
以下例子将c3标记为可选,写入c3失败的话会被忽略,如果写入c1和c2失败的话,这个事务就会失败:
a1.sources = r1
a1.channels = c1 c2 c3
a1.source.r1.selector.type = replicating
a1.source.r1.channels = c1 c2 c3
a1.source.r1.selector.optional = c3
ReplicatingChannelSelector初始化过程:
public void configure(Context context) {
//获取哪些Channel标记为可选
String optionalList = context.getString(CONFIG_OPTIONAL);
//将所有Channel都方法必须的Channel列表中
requiredChannels = new ArrayList<Channel>(getAllChannels());
Map<String, Channel> channelNameMap = getChannelNameMap();
if(optionalList != null && !optionalList.isEmpty()) {
//下面的操作:如果channel属于可选的,则加入可选的列表中,并从必选的列表中删除
for(String optional : optionalList.split("\\s+")) {
Channel optionalChannel = channelNameMap.get(optional);
requiredChannels.remove(optionalChannel);
if (!optionalChannels.contains(optionalChannel)) {
optionalChannels.add(optionalChannel);
}
}
}
}
MultiplexingChannelSelector
属性名 | 默认 | Description |
---|---|---|
selector.type | replicating | 组件名:multiplexing |
selector.optional | – |
标记哪些Channels是可选的
|
selector.header
|
flume.selector.header
|
|
selector.default | – |
|
selector.mapping.* | – |
|
示例:
a1.sources = r1
a1.channels = c1 c2 c3 c4
a1.sources.r1.selector.type = multiplexing
a1.sources.r1.selector.header = state
a1.sources.r1.selector.mapping.CZ = c1
a1.sources.r1.selector.mapping.US = c2 c3
a1.sources.r1.selector.default = c4
根据header中key为state的值,决定将数据写入那个channel中,如上示例将state=CZ写入到c1中,将state=US写入到c2,c3中,默认情况下写入c4
MultiplexingChannelSelector的初始化过程:
public void configure(Context context) {
//获取Header的值
this.headerName = context.getString(CONFIG_MULTIPLEX_HEADER_NAME,
DEFAULT_MULTIPLEX_HEADER);
Map<String, Channel> channelNameMap = getChannelNameMap();
//获取默认的Channel
defaultChannels = getChannelListFromNames(
context.getString(CONFIG_DEFAULT_CHANNEL), channelNameMap);
//获取Mapping的值
Map<String, String> mapConfig =
context.getSubProperties(CONFIG_PREFIX_MAPPING);
//channelMapping变量存放了header变量中必须的Channel列表
channelMapping = new HashMap<String, List<Channel>>();
//将header对应的Channels存放到channelMapping变量中。
for (String headerValue : mapConfig.keySet()) {
List<Channel> configuredChannels = getChannelListFromNames(
mapConfig.get(headerValue),
channelNameMap);
//This should not go to default channel(s)
//because this seems to be a bad way to configure.
if (configuredChannels.size() == 0) {
throw new FlumeException("No channel configured for when "
+ "header value is: " + headerValue);
}
if (channelMapping.put(headerValue, configuredChannels) != null) {
throw new FlumeException("Selector channel configured twice");
}
}
//If no mapping is configured, it is ok.
//All events will go to the default channel(s).
Map<String, String> optionalChannelsMapping =
context.getSubProperties(CONFIG_PREFIX_OPTIONAL + ".");
//以下这一整段代码中是赛选出Header对应那些可选Channel列表。
optionalChannels = new HashMap<String, List<Channel>>();
for (String hdr : optionalChannelsMapping.keySet()) {
List<Channel> confChannels = getChannelListFromNames(
optionalChannelsMapping.get(hdr), channelNameMap);
if (confChannels.isEmpty()) {
confChannels = EMPTY_LIST;
}
//Remove channels from optional channels, which are already
//configured to be required channels.
List<Channel> reqdChannels = channelMapping.get(hdr);
//Check if there are required channels, else defaults to default channels
if(reqdChannels == null || reqdChannels.isEmpty()) {
//如果header对应的必选Channel列表为空,那么deault就作为它的必选Channel
reqdChannels = defaultChannels;
}
for (Channel c : reqdChannels) {
//如果header对应的Channel是必选的,那么就在可选的列表中删除。
if (confChannels.contains(c)) {
confChannels.remove(c);
}
}
if (optionalChannels.put(hdr, confChannels) != null) {
throw new FlumeException("Selector channel configured twice");
}
}
}
在看看MultiplexingChannelSelector中getRequiredChannels和getOptionalChannels方法,这两个方法也是根据HeaderName来获取Channel列表的:
@Override
public List<Channel> getRequiredChannels(Event event) {
String headerValue = event.getHeaders().get(headerName);
//headerValue不存在,就获取默认
if (headerValue == null || headerValue.trim().length() == 0) {
return defaultChannels;
}
//根据headerName获取必选的Channel列表
List<Channel> channels = channelMapping.get(headerValue);
//This header value does not point to anything
//Return default channel(s) here.
//必选列表为null,则返回默认的Channel列表
if (channels == null) {
channels = defaultChannels;
}
return channels;
}
@Override
public List<Channel> getOptionalChannels(Event event) {
String hdr = event.getHeaders().get(headerName);
//根据HeaderValue获取可选的channel列表
List<Channel> channels = optionalChannels.get(hdr);
//可选列表为null,则返回空列表。
if(channels == null) {
channels = EMPTY_LIST;
}
return channels;
}
自定义 Channel Selector
自定义的Channel Selector必须实现ChannelSelector接口,配置如下(全类名)
a1.sources.r1.selector.type = org.example.MyChannelSelector