Metron pcap backend代码解析

Metron pcap backend代码解析

metron pcap backend 主要用于创建一个storm topology 处理流，将来自于Kafka的原始数据解析到HDFS中。
其数据流如下：

Pcap topology

这个topology 主要是读取数据，并将数据以顺序的方式写入到HDFS中。
主要流程如下：
抓取数据包，并发送到kafka中

数据发送到Kafka以后，pcap_backend提交到storm中的流处理任务将消费该数据，并将数据存储到HDFS中。
最后在运行另外一段解析的程序 pcap_inspector.sh。

可以看到数据包的一些基础信息。那么其具体实现流程如何，我们开始跟踪实现的具体步骤。

所谓的启动，其实主要是看 pcap_backend中的start_pcap_topology脚本。

metron-pcap-backend->src->main->scripts->start_pcap_topology

export METRON_VERSION=${project.version}
export METRON_HOME=/usr/metron/$METRON_VERSION
export TOPOLOGIES_JAR=${project.artifactId}-$METRON_VERSION.jar

storm jar $METRON_HOME/lib/$TOPOLOGIES_JAR org.apache.storm.flux.Flux --remote $METRON_HOME/flux/pcap/remote.yaml --filter $METRON_HOME/config/pcap.properties

由上可以看实际是通过flux的方式加载对应的yaml配置进行的项目提交。
pcap.properites:

spout.kafka.topic.pcap=pcap
topology.worker.childopts=
# kerberos token ticke 值配置
topology.auto-credentials=[]
topology.workers=1
kafka.zk=node1:2181
hdfs.sync.every=1
hdfs.replication.factor=-1
kafka.security.protocol=PLAINTEXT
# One of EARLIEST, LATEST, UNCOMMITTED_EARLIEST, UNCOMMITTED_LATEST
kafka.pcap.start=UNCOMMITTED_EARLIEST
#一个文件里面保留的最大包数量
kafka.pcap.numPackets=1000
#配置这个保存包的时间
kafka.pcap.maxTimeMS=300000
kafka.pcap.ts_scheme=FROM_KEY
#HDFS中存储这些采集包的路径
kafka.pcap.out=/apps/metron/pcap
#时间单位
kafka.pcap.ts_granularity=MICROSECONDS
kafka.spout.parallelism=1

看了配置，在看topology的流配置。metron 中的storm统一采用Flunk进行配置。

name: "pcap"
config:
    topology.workers: ${topology.workers}
    topology.worker.childopts: ${topology.worker.childopts}
    topology.auto-credentials: ${topology.auto-credentials}

components:

  # Any kafka props for the producer go here.
  - id: "kafkaProps"
    className: "java.util.HashMap"
    configMethods:
      -   name: "put"
          args:
            - "value.deserializer"
            - "org.apache.kafka.common.serialization.ByteArrayDeserializer"
      -   name: "put"
          args:
            - "key.deserializer"
            - "org.apache.kafka.common.serialization.ByteArrayDeserializer"
      -   name: "put"
          args:
            - "group.id"
            - "pcap"
      -   name: "put"
          args:
            - "security.protocol"
            - "${kafka.security.protocol}"

  - id: "kafkaConfig"
    className: "org.apache.metron.storm.kafka.flux.SimpleStormKafkaBuilder"
    constructorArgs:
      - ref: "kafkaProps"
      # topic name
      - "${spout.kafka.topic.pcap}"
      - "${kafka.zk}"
    configMethods:
      -   name: "setFirstPollOffsetStrategy"
          args:
            # One of EARLIEST, LATEST, UNCOMMITTED_EARLIEST, UNCOMMITTED_LATEST
            - ${kafka.pcap.start}

  - id: "writerConfig"
    className: "org.apache.metron.spout.pcap.HDFSWriterConfig"
    configMethods:
      -   name: "withOutputPath"
          args:
            - "${kafka.pcap.out}"
      -   name: "withNumPackets"
          args:
            - ${kafka.pcap.numPackets}
      -   name: "withMaxTimeMS"
          args:
            - ${kafka.pcap.maxTimeMS}
      -   name: "withZookeeperQuorum"
          args:
            - "${kafka.zk}"
      -   name: "withSyncEvery"
          args:
            - ${hdfs.sync.every}
      -   name: "withReplicationFactor"
          args:
            - ${hdfs.replication.factor}
      -   name: "withDeserializer"
          args:
            - "${kafka.pcap.ts_scheme}"
            - "${kafka.pcap.ts_granularity}"
spouts:
  - id: "kafkaSpout"
    className: "org.apache.metron.spout.pcap.KafkaToHDFSSpout"
    parallelism: ${kafka.spout.parallelism}
    constructorArgs:
      - ref: "kafkaConfig"
      - ref: "writerConfig"

看了上面的配置，我们再核实一下：

在分析写入HDFSF代码前。我们先查看下，直接消费kafka里面数据的结果：

通过上图可以看出，Kafka中和我们原来想象的不太一样，不是pcap的格式。其具体格式内容的问题：可以参考笔者关于pcapy的博客，这里不再讲述。

根据配置我们可以看到起入口主函数为pcap下的KafkaToHDFSSpout（其关联了2个配置构造函数）
我们先看spout 源

public KafkaToHDFSSpout( SimpleStormKafkaBuilder<byte[], byte[]> spoutConfig
                         , HDFSWriterConfig config
                         )
  {
    super(spoutConfig
         , HDFSWriterCallback.class
         );
    this.config = config;
  }

该函数继承的是kafka
核心其实就是调用了HDFSWriteCallback这个回调函数。在继续跟踪到HDFSWriteCallback下面

import org.apache.storm.kafka.Callback;
import org.apache.storm.kafka.EmitContext;
import org.apache.storm.kafka.spout.KafkaSpoutConfig;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

/**
 * A callback which gets executed as part of the spout to write pcap data to HDFS.
 */
public class HDFSWriterCallback implements Callback {

该函数继承的是org.apache.storm.kafka.Callback下面的函数

public interface Callback extends AutoCloseable, Serializable {
    List<Object> apply(List<Object> tuple, EmitContext context);
    void initialize(EmitContext context);
}

可以看到先提交，再初始化，而在HDFSWrite对这个数据进行了重写。

 @Override
    public List<Object> apply(List<Object> tuple, EmitContext context) {
        #读取流数据
        byte[] key = (byte[]) tuple.get(0);
        byte[] value = (byte[]) tuple.get(1);
        long tsDeserializeStart = System.nanoTime();
        #解析kafka中的Key-value值
        KeyValueDeserializer.Result result = config.getDeserializer().deserializeKeyValue(key, value);
        long tsDeserializeEnd = System.nanoTime();

        if (LOG.isDebugEnabled() && !result.foundTimestamp) {
            List<String> debugStatements = new ArrayList<>();
            if (key != null) {
                debugStatements.add("Key length: " + key.length);
                debugStatements.add("Key: " + DatatypeConverter.printHexBinary(key));
            } else {
                debugStatements.add("Key is null!");
            }

            if (value != null) {
                debugStatements.add("Value length: " + value.length);
                debugStatements.add("Value: " + DatatypeConverter.printHexBinary(value));
            } else {
                debugStatements.add("Value is null!");
            }
            LOG.debug("Dropping malformed packet: {}", Joiner.on(" / ").join(debugStatements));
        }

        long tsWriteStart = System.nanoTime();
        try {
            getWriter(new Partition( topic
                                   , context.get(EmitContext.Type.PARTITION))
                     ).handle(result.key, result.value);
        } catch (IOException e) {
            LOG.error(e.getMessage(), e);
            //drop?  not sure..
        }
        long tsWriteEnd = System.nanoTime();
        if(LOG.isDebugEnabled() && (Math.random() < 0.001 || !inited)) {
            LOG.debug("Deserialize time (ns): {}", (tsDeserializeEnd - tsDeserializeStart));
            LOG.debug("Write time (ns): {}", (tsWriteEnd - tsWriteStart));
        }
        inited = true;
        return tuple;
    }

上面有一段解析Kafka的方法函数deserializeKeyValue，获取其返回值result,（里面就是解析出来的发送的key和value,但需要注意的是这里的Key-value已经和发送的略微有了点变化），具体如下：

metron->spout->pcap->deserializer->FromKeyDeserializer

  @Override
  public Result deserializeKeyValue(byte[] key, byte[] value) {
    if (key == null) {
      throw new IllegalArgumentException("Expected a key but none provided");
    }
    #从前文可以知道Kafa发送的消息中的key也就是时间戳，这部分比较简单
    long ts = converter.toNanoseconds(fromBytes(key));
    #返回的第2部分是一个数据包头（addHeaders
    return new Result(ts, PcapHelper.addHeaders(ts, value, endianness), true);
  }

在上面代码中我们看到解析的值里面有一个addHeaders函数，注意这部分代码已经位于metron-pcap模块中了

  #根据大小端生成不同的pcap 的gloabla header
  public static byte[] getPcapGlobalHeader(Endianness endianness) {
    if(swapBytes(endianness)) {
      //swap
      return new byte[] {
              (byte) 0xd4, (byte) 0xc3, (byte) 0xb2, (byte) 0xa1 //swapped magic number 0xa1b2c3d4
              , 0x02, 0x00 //swapped major version 2
              , 0x04, 0x00 //swapped minor version 4
              , 0x00, 0x00, 0x00, 0x00 //GMT to local tz offset (= 0)
              , 0x00, 0x00, 0x00, 0x00 //sigfigs (= 0)
              , (byte) 0xff, (byte) 0xff, 0x00, 0x00 //snaplen (=65535)
              , 0x01, 0x00, 0x00, 0x00 // swapped link layer header type (1 = ethernet)
                        };
    }
    else {
      //no need to swap
      return new byte[] {
              (byte) 0xa1, (byte) 0xb2, (byte) 0xc3, (byte) 0xd4 //magic number 0xa1b2c3d4
              , 0x00, 0x02 //major version 2
              , 0x00, 0x04 //minor version 4
              , 0x00, 0x00, 0x00, 0x00 //GMT to local tz offset (= 0)
              , 0x00, 0x00, 0x00, 0x00 //sigfigs (= 0)
              , 0x00, 0x00, (byte) 0xff, (byte) 0xff //snaplen (=65535)
              , 0x00, 0x00, 0x00, 0x01 // link layer header type (1 = ethernet)
                        };
    }
  }
  ......
  
  public static byte[] addHeaders(long tsNano, byte[] packet, Endianness endianness) {
    #endianness 是代表大小端属性
    #这里的packet就是原始的数据包了
    byte[] ret = new byte[GLOBAL_HEADER_SIZE + PACKET_HEADER_SIZE + packet.length];
    #根据大小端生成pcap 格式中的globalHeader投
    byte[] globalHeader = getPcapGlobalHeader(endianness);
    int offset = 0;
    System.arraycopy(globalHeader, 0, ret, offset, GLOBAL_HEADER_SIZE);
    offset += globalHeader.length;
    #packet 时间等信息，获取并移位填写
    {
      boolean swapBytes = swapBytes(endianness);
      long micros = Long.divideUnsigned(tsNano, 1000);
      int secs = (int)(micros / 1000000);
      int usec = (int)(micros % 1000000);
      int capLen = packet.length;
      {
        byte[] b = Bytes.toBytes(swapBytes?ByteOrderConverter.swap(secs):secs);
        System.arraycopy(b, 0, ret, offset, Integer.BYTES);
        offset += Integer.BYTES;
      }
      {
        byte[] b = Bytes.toBytes(swapBytes?ByteOrderConverter.swap(usec):usec);
        System.arraycopy(b, 0, ret, offset, Integer.BYTES);
        offset += Integer.BYTES;
      }
      {
        byte[] b = Bytes.toBytes(swapBytes?ByteOrderConverter.swap(capLen):capLen);
        System.arraycopy(b, 0, ret, offset, Integer.BYTES);
        offset += Integer.BYTES;
      }
      {
        byte[] b = Bytes.toBytes(swapBytes?ByteOrderConverter.swap(capLen):capLen);
        System.arraycopy(b, 0, ret, offset, Integer.BYTES);
        offset += Integer.BYTES;
      }
    }
    System.arraycopy(packet, 0, ret, offset, packet.length);
    return ret;
  }

根据代码所示，在写入到HDFS的文件中应该有一部分是pcap Header的头（24byte）,packer_hader(16byte),后面就是原始包才对。我们打开一个存入的HDFS文件进行验证：
从上图可以看到完全和存入到HDFS的文件能对上，但是在gloabal header前面还有115byte的数据，不知道是怎么来的？
在前面List函数中最后可以看到一句：

     try {
            getWriter(new Partition( topic
                                   , context.get(EmitContext.Type.PARTITION))
                     ).handle(result.key, result.value);
        } catch (IOException e) {
            LOG.error(e.getMessage(), e);
            //drop?  not sure..
        }
        ....
        继续跟到hadle代码中
        public void handle(long ts, byte[] value) throws IOException {
    turnoverIfNecessary(ts);
    BytesWritable bw = new BytesWritable(value);
    try {
      writer.append(new LongWritable(ts), bw);
    }
    catch(ArrayIndexOutOfBoundsException aioobe) {
      LOG.warn("This appears to be HDFS-7765 (https://issues.apache.org/jira/browse/HDFS-7765), " +
              "which is an issue with syncing and not problematic: {}", aioobe.getMessage(), aioobe);
    }
    numWritten++;
    if(numWritten % config.getSyncEvery() == 0) {
      syncHandler.sync(outputStream);
    }
  }
  似乎什么也没做，但是有一个很不起眼的函数turnoverIfNecessary
   private void turnoverIfNecessary(long ts, boolean force) throws IOException {
    long duration = ts - batchStartTime;
    boolean initial = outputStream == null;
    boolean overDuration = config.getMaxTimeNS() <= 0 ? false : Long.compareUnsigned(duration, config.getMaxTimeNS()) >= 0;
    boolean tooManyPackets = numWritten >= config.getNumPackets();
    if(force || initial || overDuration || tooManyPackets ) {
      //turnover
      Path path = getPath(ts);
      close();

      if(fs instanceof LocalFileSystem) {
        outputStream = new FSDataOutputStream(new FileOutputStream(new File(path.toString())));
        syncHandler = SyncHandlers.LOCAL.getHandler();
      }
      else {
        outputStream = fs.create(path, true);
        if (outputStream instanceof HdfsDataOutputStream) {
          if (initial) {
            LOG.info("Using the HDFS sync handler.");
          }
          syncHandler = SyncHandlers.HDFS.getHandler();
        } else {
          if (initial) {
            LOG.info("Using the default sync handler, which cannot guarantee atomic appends at the record level, be forewarned!");
          }
          syncHandler = SyncHandlers.DEFAULT.getHandler();
        }

      }
      writer = SequenceFile.createWriter(this.fsConfig
              , SequenceFile.Writer.keyClass(LongWritable.class)
              , SequenceFile.Writer.valueClass(BytesWritable.class)
              , SequenceFile.Writer.stream(outputStream)
              , SequenceFile.Writer.compression(SequenceFile.CompressionType.NONE)
      );
      //reset state
      LOG.info("Turning over and writing to {}: [duration={} NS, force={}, initial={}, overDuration={}, tooManyPackets={}]", path, duration, force, initial, overDuration, tooManyPackets);
      batchStartTime = ts;
      numWritten = 0;
    }
  }

}
看到没，writer 写入了一堆序列化的信息到里面去，但注意根据代码这段只有在创建时才有效，所以一个保存的文件，也只有一个序列化信息，这点和现状也刚好吻合。
  

这是否就是HDFS文件中的内容了，可以看到开始写入的是LongWriteable的类名（org.apache.hadoop.io.LongWriteable）和BytesWritable的类名（org.apache.hadoop.io.ByteWritable）

转载于:https://www.cnblogs.com/angellst/p/8717042.html