Flume (七) Monitoring

Flume中的监控仍在进行中。 变化可能经常发生。 几个Flume组件向JMX平台MBean服务器报告度量标准。 可以使用Jconsole查询这些指标。

JMX Reporting

可以通过使用flume-env.sh在JAVA_OPTS环境变量中指定JMX参数来启用JMX报告，如

export JAVA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=5445 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

注意：上面的示例禁用安全性。 要启用安全性，请参阅
http://docs.oracle.com/javase/6/docs/technotes/guides/management/agent.html

Ganglia Reporting

Flume can also report these metrics to Ganglia 3 or Ganglia 3.1 metanodes. To report metrics to Ganglia, a flume agent must be started with this support. The Flume agent has to be started by passing in the following parameters as system properties prefixed by flume.monitoring., and can be specified in the flume-env.sh:

Property Name	Default	Description
type	–	组件类型名称必须是ganglia
hosts	–	以逗号分隔Ganglia服务器的hostname:port
pollFrequency	60	连续向Ganglia服务器报告之间的时间（以秒为单位）
isGanglia3	false	Ganglia服务器版本为3.默认情况下，Flume以Ganglia 3.1格式发送

我们可以使用Ganglia支持启动Flume，如下所示：

$ bin/flume-ng agent --conf-file example.conf --name a1 -Dflume.monitoring.type=ganglia -Dflume.monitoring.hosts=com.example:1234,com.example2:5455

JSON Reporting

Flume还可以以JSON格式报告指标。 为了以JSON格式启用报告，Flume在可配置端口上托管Web服务器。 Flume以以下JSON格式报告指标：

{
"typeName1.componentName1" : {"metric1" : "metricValue1", "metric2" : "metricValue2"},
"typeName2.componentName2" : {"metric3" : "metricValue3", "metric4" : "metricValue4"}
}

Here is an example:

{
"CHANNEL.fileChannel":{"EventPutSuccessCount":"468085",
                      "Type":"CHANNEL",
                      "StopTime":"0",
                      "EventPutAttemptCount":"468086",
                      "ChannelSize":"233428",
                      "StartTime":"1344882233070",
                      "EventTakeSuccessCount":"458200",
                      "ChannelCapacity":"600000",
                      "EventTakeAttemptCount":"458288"},
"CHANNEL.memChannel":{"EventPutSuccessCount":"22948908",
                   "Type":"CHANNEL",
                   "StopTime":"0",
                   "EventPutAttemptCount":"22948908",
                   "ChannelSize":"5",
                   "StartTime":"1344882209413",
                   "EventTakeSuccessCount":"22948900",
                   "ChannelCapacity":"100",
                   "EventTakeAttemptCount":"22948908"}
}

Property Name	Default	Description
type	–	组件类型名称必须是http
port	41414	启动服务器的端口。

We can start Flume with JSON Reporting support as follows:

$ bin/flume-ng agent --conf-file example.conf --name a1 -Dflume.monitoring.type=http -Dflume.monitoring.port=34545

然后，可以在http://<hostname>:<port>/metrics网页上获得度量标准。 自定义组件可以报告上面Ganglia部分中提到的指标。

Custom Reporting

可以通过编写执行报告的服务器向其他系统报告度量标准。 任何报告类都必须实现org.apache.flume.instrumentation.MonitorService接口。 这样的类可以与GangliaServer用于报告的方式相同。 他们可以轮询平台mbean服务器以轮询mbeans以获取指标。 例如，如果可以使用名为HTTPReporting的HTTP监视服务，如下所示：

$ bin/flume-ng agent --conf-file example.conf --name a1 -Dflume.monitoring.type=com.example.reporting.HTTPReporting -Dflume.monitoring.node=com.example:332

Property Name	Default	Description
type	–	组件类型名称必须是FQCN

Reporting metrics from custom components

任何自定义flume组件都应继承自org.apache.flume.instrumentation.MonitoredCounterGroup类。 然后，该类应为其公开的每个度量标准提供getter方法。 请参阅下面的代码。 MonitoredCounterGroup需要一个属性列表，其度量由此类公开。 截至目前，此类仅支持将指标公开为long 。

public class SinkCounter extends MonitoredCounterGroup implements
    SinkCounterMBean {

  private static final String COUNTER_CONNECTION_CREATED =
    "sink.connection.creation.count";

  private static final String COUNTER_CONNECTION_CLOSED =
    "sink.connection.closed.count";

  private static final String COUNTER_CONNECTION_FAILED =
    "sink.connection.failed.count";

  private static final String COUNTER_BATCH_EMPTY =
    "sink.batch.empty";

  private static final String COUNTER_BATCH_UNDERFLOW =
      "sink.batch.underflow";

  private static final String COUNTER_BATCH_COMPLETE =
    "sink.batch.complete";

  private static final String COUNTER_EVENT_DRAIN_ATTEMPT =
    "sink.event.drain.attempt";

  private static final String COUNTER_EVENT_DRAIN_SUCCESS =
    "sink.event.drain.sucess";

  private static final String[] ATTRIBUTES = {
    COUNTER_CONNECTION_CREATED, COUNTER_CONNECTION_CLOSED,
    COUNTER_CONNECTION_FAILED, COUNTER_BATCH_EMPTY,
    COUNTER_BATCH_UNDERFLOW, COUNTER_BATCH_COMPLETE,
    COUNTER_EVENT_DRAIN_ATTEMPT, COUNTER_EVENT_DRAIN_SUCCESS
  };


  public SinkCounter(String name) {
    super(MonitoredCounterGroup.Type.SINK, name, ATTRIBUTES);
  }

  @Override
  public long getConnectionCreatedCount() {
    return get(COUNTER_CONNECTION_CREATED);
  }

  public long incrementConnectionCreatedCount() {
    return increment(COUNTER_CONNECTION_CREATED);
  }

}