JStorm自定义Metric的简单步骤和原理

简单介绍

可以在JStorm的控制台看见。

和常见的java metrics系统一样，支持

• Counter:累计值
• Gauge：瞬时值
• Histogram：瞬时值的分布

使用

public class MyBolt extends BaseBasicBolt {

    private MetricClient metricClient;
    private AsmCounter myCounter;

    @Override
    public void prepare(Map stormConf, TopologyContext context) {
        metricClient = new MetricClient(context);
        myCounter = metricClient.registerCounter("myCounter");
    }

    public void execute(Tuple input, BasicOutputCollector collector) {
        Object value = input.getValueByField("value");
        myCounter.inc();
        if(value instanceof UmetripEvent){
            System.out.println("RECEIVE");
            UmetripEvent event = (UmetripEvent)value;

        }
       //throw new FailedException("manual fail");


    }

    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        declarer.declare(new Fields("EVENT"));
    }


}

原理

首先要介绍Topology Master。

在没有Topology Master的情况下，所有的Task汇报心跳、发送Metrics的时候都需要写ZK，并且Storm是把Metrics直接ZK里面，这样ZK的压力更大。有了Topology Master之后，Topology Master是Task的总节点，Task汇报心跳的时候只需要向Topology Master汇报心跳，Topology Master汇总后再发送给Nimbus或者ZK，这样会使ZK的压力缩到拓扑的量级。

    //JStormMetricsReporter。当工作在Worker中时，发送给topo master。topo master会在聚合后向nimbus发送metrics数据
    //当在Nimbus/Supervisor工作时，直接传给Nimbus。
    public void uploadMetricData(WorkerUploadMetrics metrics) {
        if (inTopology) {
            //in Worker, we upload data via netty transport
            if (boltOutput != null) {
                LOG.debug("emit metrics through bolt collector.");
                ((BoltCollector) boltOutput.getDelegate()).emitCtrl(Common.TOPOLOGY_MASTER_METRICS_STREAM_ID, null,
                        new Values(JStormServerUtils.getName(host, port), metrics));
            } else if (spoutOutput != null) {
                LOG.debug("emit metrics through spout collector.");
                ((SpoutCollector) spoutOutput.getDelegate()).emitCtrl(Common.TOPOLOGY_MASTER_METRICS_STREAM_ID,
                        new Values(JStormServerUtils.getName(host, port), metrics), null);
            } else {
                LOG.warn("topology:{}, both spout/bolt collectors are null, don't know what to do...", topologyId);
            }
        } else {
            // in supervisor or nimbus, we upload metric data via thrift
            LOG.debug("emit metrics through nimbus client.");
            TopologyMetric tpMetric = MetricUtils.mkTopologyMetric();
            tpMetric.set_workerMetric(metrics.get_allMetrics());

            //UpdateEvent.pushEvent(topologyId, tpMetric);
            try {
                // push metrics via nimbus client
                if (client == null) {
                    LOG.warn("nimbus client is null...");
                    client = new NimbusClientWrapper();
                    client.init(conf);
                }
                client.getClient().uploadTopologyMetrics(topologyId, tpMetric);
            } catch (Throwable ex) {
                LOG.error("upload metrics error:", ex);
                if (client != null) {
                    client.cleanup();
                    client = null;
                }
            }
        }
        //MetricUtils.logMetrics(metrics.get_allMetrics());
    }