一直用flume做数据收集,用了好一阵子都没出现数据误差的问题,今天在导入数据的时候却突然出现了数据暴增的问题,查看原始数据文件,只有几十万条数据,可却收集到了几百万的event,而且还在持续的增加。很奇怪,首先以为是插件更新的问题,后来换成最原始的console作为Collector的sink,问题依旧存在。然后清理配置数据重新启动,问题还是没有解决。
查看原始数据并与原有的数据对比,发现有超大内容的的event,是业务端的错误。所以估计应该是大event的问题,然后查看源码,发现异样
public class SyslogUdpSource extends EventSource.Base {
static final Logger LOG = LoggerFactory.getLogger(SyslogUdpSource.class);
final public static int SYSLOG_UDP_PORT = 514;
int port = SYSLOG_UDP_PORT; // default udp syslog port
int maxsize = 1 << 16; // 64k is max allowable in RFC 5426
long rejects = 0;
DatagramSocket sock;
public SyslogUdpSource() {
}
public SyslogUdpSource(int port) {
this.port = port;
}
@Override
public void close() throws IOException {
LOG.info("closing SyslogUdpSource on port " + port);
if (sock == null) {
LOG.warn("double close of SyslogUdpSocket on udp:" + port
+ " , (this is ok but odd)");
return;
}
sock.close();
}
@Override
public Event next() throws IOException {
byte[] buf = new byte[maxsize];
DatagramPacket pkt = new DatagramPacket(buf, maxsize);
Event e = null;
do { // loop until we get a valid packet, drop bad ones.
sock.receive(pkt);
ByteBuffer bb = ByteBuffer.wrap(buf, 0, pkt.getLength());
ByteBufferInputStream bbis = new ByteBufferInputStream(bb);
DataInputStream in = new DataInputStream(bbis);
try {
e = SyslogWireExtractor.extractEvent(in);
} catch (EventExtractException ex) {
rejects++;
LOG.warn(rejects + " rejected packets. packet: " + pkt, ex);
LOG.debug("raw bytes " + Arrays.toString(pkt.getData()));
// TODO (jon) maybe have a hook here to do something with rejects
}
// need a sane way to fall out of his loop.
} while (e == null);
updateEventProcessingStats(e);
return e;
}
@Override
public void open() throws IOException {
sock = new DatagramSocket(port);
}
public static SourceBuilder builder() {
return new SourceBuilder() {
@Override
public EventSource build(Context ctx, String... argv) {
int port = SYSLOG_UDP_PORT; // default udp port, need root permissions
// for this.
if (argv.length > 1) {
throw new IllegalArgumentException("usage: syslogUdp([port no]) ");
}
if (argv.length == 1) {
port = Integer.parseInt(argv[0]);
}
return new SyslogUdpSource(port);
}
};
}
}
64k is max allowable in RFC 5426(有关RFC 5426 event大小的限制)
然后网上查看各种资料,原来当event大于64K时,flume agent端会重复发送该event,从而导致数据暴增的问题(更详细的原因有待有空的时候考证)。所以应该在业务端控制好event的大小。