linux 源码解读 自定义,nGrinder 对监控机器收集自定义数据及源码分析

0.背景

性能测试工具 nGrinder 支持在无需修改源码的情况下，对目标服务器收集自定义数据，最多支持 5 类；

在性能测试详细报告页，目标服务器->你的机器 ip 便签页下，默认只收集 CPU, Memory, Received Byte/s, Sent Byte Per Secode/s 等 4 类数据；

可能你还需要监控其它的性能统计数据，用于分析 (比如 load, Full Gc)；本文先介绍实现方法；再分析 nGrinder 源码，看它是怎么实现的。

1.实现

1-1. 安装 monitor

在你的 nGrinder 系统下，下载监控

安装到你测试服务所在的机器，解压 tar 包，执行 sh run_monitor_bg.sh;

其实脚本是启了个 java 服务，以 monitor 模式启动；

之前介绍过 Agent 有 2 种模式：

gent mode: 运行进程和线程，压测目标服务；

monitor mode: 监控目标系统性能 (cpu/memory)。

[root@10 ngrinder-monitor]# cat run_monitor.sh

#!/bin/sh

curpath=`dirname $0`

cd ${curpath}

java -server -cp "lib/*" org.ngrinder.NGrinderAgentStarter --mode monitor --command run $@

Agent 的 home 路径为/root/.ngrinder_agent，你在执行 sh run_monitor_bg.sh 默认获取的配置信息为/root/.ngrinder_agent/agent.conf； 如果加上-o，sh run_monitor_bg.sh 读取你安装 monitor 目录下的__agent.conf， 该配置文件定义了 Agent 的模式，ip, 端口。

[root@10 .ngrinder_agent]# cat agent.conf

common.start_mode=monitor

#If you want to monitor bind to the different local ip not automatically selected ip. Specify below field.

#monitor.binding_port=hostname_or_ip

monitor.binding_port=13243

自定义数据需放在/root/.ngrinder_agent/monitor/custom.data 文件里，格式如下：

类型1数据,类型2数据,类型3数据,类型4数据,类型5数据

最多支持 5 类，每类数据用 “,” 分隔，注意的是： 数据是实时的写文件，不是累积数据到文件中 (类似 shell 中的>, 不是>>)，即同一时刻，只有一行数据。

1-2. 定制收集脚本

以收集 load 和 full GC 为例：

[root@10 bin]# cat updateCustomData.sh

#!/bin/sh

#@author hugang

customDataRoot=/root/.ngrinder_agent/monitor/custom.data;

# 获取load信息

load=`/bin/cat /proc/loadavg | awk '{print $1}'`;

# 获取full gc count

if [[ $1 -gt 0 ]]; then

fgc=`jstat -gcutil $1 | tail -1 | awk '{print $8}'`;

echo $load,$fgc > $customDataRoot;

else

echo $load > $customDataRoot;

fi;

开始性能测试时，每秒去执行该脚本，收集数据到 custom.data 中：

watch -n 1 sh updateCustomData.sh 5528

5528 为需监控 java 服务进程 pid;

当你性能测试结束后，monitor 收集的数据会放到/root/.ngrinder/perftest/0_999/${test_id}/report/monitor_system_${ip}.data 文件中:

[root@10 report]# cat monitor_system_10.13.1.139.data

ip,system,collectTime,freeMemory,totalMemory,cpuUsedPercentage,receivedPerSec,sentPerSec,customValues

10.13.1.139,LINUX,20160302151441,97102768,132112072,26.895683,32954,27897,4.93,49

10.13.1.139,LINUX,20160302151443,97075896,132112072,30.513468,45702,32306,4.93,49

10.13.1.139,LINUX,20160302151445,97034772,132112072,30.411074,110306,65391,5.02,49

10.13.1.139,LINUX,20160302151447,96972504,132112072,22.073017,84813,57503,5.02,49

...

1-3.结果展示

2.源码分析

System memory, swap, cpu, load average, uptime, logins

Per-process memory, cpu, credential info, state, arguments, environment, open files

File system detection and metrics

Network interface detection, configuration info and metrics

TCP and UDP connection tables

Network route table

[root@10 testsigar]# ls

libsigar-amd64-linux.so sigar-1.6.4.jar sigar-1.6.4.jar.zip

[root@10 testsigar]#

[root@10 testsigar]# java -jar ./sigar-1.6.4.jar

sigar> free

total used free

Mem: 132112072 96855372 35256700

-/+ buffers/cache: 34855500 97256572

Swap: 8388600 264980 8123620

RAM: 129016MB

sigar>

收集系统数据的 java 文件为：

ngrinder-core/src/main/java/org/ngrinder/monitor/collector/SystemDataCollector.java

继承和实现关系：

SystemDataCollector extends DataCollector

DataCollector implements Runnable

SystemDataCollector 的线程执行体：

public void run() {

// 初始化sigar

initSigar();

SystemMonitoringData systemMonitoringData = (SystemMonitoringData) getMXBean(SYSTEM);

// execute()通过sigar api获取系统信息

systemMonitoringData.setSystemInfo(execute());

}

execute() 获取系统信息 SystemInfo(System info object to save date collected by monitor)：

/**

* Execute the collector to get the system info model.

*

* @return SystemInfo in current time

*/

public synchronized SystemInfo execute() {

SystemInfo systemInfo = new SystemInfo();

systemInfo.setCollectTime(System.currentTimeMillis());

try {

BandWidth networkUsage = getNetworkUsage();

BandWidth bandWidth = networkUsage.adjust(prev.getBandWidth());

systemInfo.setBandWidth(bandWidth);

systemInfo.setCPUUsedPercentage((float) sigar.getCpuPerc().getCombined() * 100);

Cpu cpu = sigar.getCpu();

systemInfo.setTotalCpuValue(cpu.getTotal());

systemInfo.setIdleCpuValue(cpu.getIdle());

Mem mem = sigar.getMem();

systemInfo.setTotalMemory(mem.getTotal() / 1024L);

systemInfo.setFreeMemory(mem.getActualFree() / 1024L);

systemInfo.setSystem(OperatingSystem.IS_WIN32 ? SystemInfo.System.WINDOW : SystemInfo.System.LINUX);

systemInfo.setCustomValues(getCustomMonitorData());

} catch (Throwable e) {

LOGGER.error("Error while getting system perf data:{}", e.getMessage());

LOGGER.debug("Error trace is ", e);

}

prev = systemInfo;

return systemInfo;

}

其中：getCustomMonitorData() 获取自定义数据，读取 custom.data 文件中一行数据

private String getCustomMonitorData() {

if (customDataFile != null && customDataFile.exists()) {

BufferedReader customDataFileReader = null;

try {

customDataFileReader = new BufferedReader(new FileReader(customDataFile));

return customDataFileReader.readLine(); // these data will be parsed at

// monitor client side.

} catch (IOException e) {

// Error here is very natural

LOGGER.debug("Error to read custom monitor data", e);

} finally {

IOUtils.closeQuietly(customDataFileReader);

}

}

return prev.getCustomValues();

}

综上：类 SystemDataCollector 作用就是作为线程执行体，线程每次执行通过 sigar 获取系统信息：SystemInfo，赋值给 SystemMonitoringData 成员变量 SystemInfo。

前面介绍启动 monitor 时，其实是执行了 org.ngrinder.NGrinderAgentStarter 类，我们再分析下该文件，ngrinder-core/src/main/java/org/ngrinder/NGrinderAgentStarter.java

/**

* Agent starter.

*

* @param args arguments

*/

public static void main(String[] args) {

NGrinderAgentStarter starter = new NGrinderAgentStarter();

final NGrinderAgentStarterParam param = new NGrinderAgentStarterParam();

checkJavaVersion();

JCommander commander = new JCommander(param);

commander.setProgramName("ngrinder-agent");

commander.setAcceptUnknownOptions(true);

try {

commander.parse(args);

} catch (Exception e) {

LOG.error(e.getMessage());

return;

}

final List unknownOptions = commander.getUnknownOptions();

modeParam = param.getModeParam();

modeParam.parse(unknownOptions.toArray(new String[unknownOptions.size()]));

if (modeParam.version != null) {

System.out.println("nGrinder v" + getStaticVersion());

return;

}

if (modeParam.help != null) {

modeParam.usage();

return;

}

System.getProperties().putAll(modeParam.params);

starter.init();

final String startMode = modeParam.name();

if ("stop".equalsIgnoreCase(param.command)) {

starter.stopProcess(startMode);

System.out.println("Stop the " + startMode);

return;

}

starter.checkDuplicatedRun(startMode);

if (startMode.equalsIgnoreCase("agent")) {

starter.startAgent();

} else if (startMode.equalsIgnoreCase("monitor")) {

starter.startMonitor();

} else {

staticPrintHelpAndExit("Invalid agent.conf, '--mode' must be set as 'monitor' or 'agent'.");

}

}

monitor 模式执行该方法：starter.startMonitor()

/**

* Start the performance monitor.

*/

public void startMonitor() {

printLog("***************************************************");

printLog("* Start nGrinder Monitor... ");

printLog("***************************************************");

try {

MonitorServer.getInstance().init(agentConfig);

MonitorServer.getInstance().start();

} catch (Exception e) {

LOG.error("ERROR: {}", e.getMessage());

printHelpAndExit("Error while starting Monitor", e);

}

}

MonitorServer.getInstance().start():

/**

* Start monitoring.

*

* @throws IOException exception

*/

public void start() throws IOException {

if (!isRunning()) {

jmxServer.start();

DataCollectManager.getInstance().init(agentConfig);

DataCollectManager.getInstance().start();

isRunning = true;

}

}

DataCollectManager.getInstance().start();

/**

* start a scheduler for the data collector jobs.

*/

public void start() {

int collectorCount = MXBeanStorage.getInstance().getSize();

scheduler = Executors.newScheduledThreadPool(collectorCount);

if (!isRunning()) {

Collection mxBeans = MXBeanStorage.getInstance().getMXBeans();

for (MXBean mxBean : mxBeans) {

DataCollector collector = mxBean.gainDataCollector(agentConfig.getHome().getDirectory());

scheduler.scheduleWithFixedDelay(collector, 0L, getInterval(), TimeUnit.SECONDS);

LOG.info("{} started.", collector.getClass().getSimpleName());

}

LOG.info("Collection interval : {}s).", getInterval());

isRunning = true;

}

}

scheduler.scheduleWithFixedDelay(collector, 0L, getInterval(), TimeUnit.SECONDS);

线程池周期地执行 SystemDataCollector 中 run() 去获取系统数据。

@Override

public void run() {

initSigar();

SystemMonitoringData systemMonitoringData = (SystemMonitoringData) getMXBean(SYSTEM);

systemMonitoringData.setSystemInfo(execute());

}

3.总结：

后台启动的 monitor， 运行的是一个 java 服务:

java -server -cp lib/* org.ngrinder.NGrinderAgentStarter --mode monitor --command run

通过线程池周期获取系统性信息 (sigar 工具获取)，存放在 SystemInfo;

ngrinder-controller/src/main/java/org/ngrinder/perftest/service/samplinglistener/MonitorCollectorPlugin.java 中 startSampling():

@Override

public void startSampling(final ISingleConsole singleConsole, PerfTest perfTest,

IPerfTestService perfTestService) {

final List targetHostIP = perfTest.getTargetHostIP();

final Integer samplingInterval = perfTest.getSamplingInterval();

for (final String target : targetHostIP) {

scheduledTaskService.runAsync(new Runnable() {

@Override

public void run() {

LOGGER.info("Start JVM monitoring for IP:{}", target);

MonitorClientService client = new MonitorClientService(target, MonitorCollectorPlugin.this.port);

client.init();

if (client.isConnected()) {

File testReportDir = singleConsole.getReportPath();

File dataFile = null;

try {

dataFile = new File(testReportDir, MONITOR_FILE_PREFIX + target + ".data");

FileWriter fileWriter = new FileWriter(dataFile, false);

BufferedWriter bw = new BufferedWriter(fileWriter);

// write header info

bw.write(SystemInfo.HEADER);

bw.newLine();

bw.flush();

clientMap.put(client, bw);

} catch (IOException e) {

LOGGER.error("Error to write to file:{}, Error:{}", dataFile.getPath(), e.getMessage());

}

}

}

});

}

assignScheduledTask(samplingInterval);

}

根据 SystemInfo 写到/root/.ngrinder/perftest/0_999/${test_id}/report/monitor_system_${ip}.data 文件中；

ngrinder-controller/src/main/java/org/ngrinder/perftest/PerfTestService.java 中 getMonitorGraph() 根据/root/.ngrinder/perftest/0_999/${test_id}/report/monitor_system_${ip}.data 获取系统信息数据

/**

* Get system monitor data and wrap the data as a string value like "[22,11,12,34,....]", which can be used directly

* in JS as a vector.

*

* @param testId test id

* @param targetIP ip address of the monitor target

* @param dataInterval interval value to get data. Interval value "2" means, get one record for every "2" records.

* @return return the data in map

*/

public Map getMonitorGraph(long testId, String targetIP, int dataInterval) {

Map returnMap = Maps.newHashMap();

File monitorDataFile = new File(config.getHome().getPerfTestReportDirectory(String.valueOf(testId)),

MONITOR_FILE_PREFIX + targetIP + ".data");

BufferedReader br = null;

try {

StringBuilder sbUsedMem = new StringBuilder("[");

StringBuilder sbCPUUsed = new StringBuilder("[");

StringBuilder sbNetReceived = new StringBuilder("[");

StringBuilder sbNetSent = new StringBuilder("[");

StringBuilder customData1 = new StringBuilder("[");

StringBuilder customData2 = new StringBuilder("[");

StringBuilder customData3 = new StringBuilder("[");

StringBuilder customData4 = new StringBuilder("[");

StringBuilder customData5 = new StringBuilder("[");

br = new BufferedReader(new FileReader(monitorDataFile));

br.readLine(); // skip the header.

// "ip,system,collectTime,freeMemory,totalMemory,cpuUsedPercentage,receivedPerSec,sentPerSec"

String line = br.readLine();

int skipCount = dataInterval;

// to be compatible with previous version, check the length before

// adding

while (StringUtils.isNotBlank(line)) {

if (skipCount < dataInterval) {

skipCount++;

} else {

skipCount = 1;

String[] datalist = StringUtils.split(line, ",");

if ("null".equals(datalist[4]) || "undefined".equals(datalist[4])) {

sbUsedMem.append("null").append(",");

} else {

sbUsedMem.append(Long.valueOf(datalist[4]) - Long.valueOf(datalist[3])).append(",");

}

addCustomData(sbCPUUsed, 5, datalist);

addCustomData(sbNetReceived, 6, datalist);

addCustomData(sbNetSent, 7, datalist);

addCustomData(customData1, 8, datalist);

addCustomData(customData2, 9, datalist);

addCustomData(customData3, 10, datalist);

addCustomData(customData4, 11, datalist);

addCustomData(customData5, 12, datalist);

line = br.readLine();

}

}

completeCustomData(returnMap, "cpu", sbCPUUsed);

completeCustomData(returnMap, "memory", sbUsedMem);

completeCustomData(returnMap, "received", sbNetReceived);

completeCustomData(returnMap, "sent", sbNetSent);

completeCustomData(returnMap, "customData1", customData1);

completeCustomData(returnMap, "customData2", customData2);

completeCustomData(returnMap, "customData3", customData3);

completeCustomData(returnMap, "customData4", customData4);

completeCustomData(returnMap, "customData5", customData5);

} catch (IOException e) {

LOGGER.info("Error while getting monitor {} data file at {}", targetIP, monitorDataFile);

} finally {

IOUtils.closeQuietly(br);

}

return returnMap;

}

数据提供给 Controller 端：

ngrinder-controller/src/man/java/org/ngrinder/perftest/controller/PerfTestController.java

private Map getMonitorGraphData(long id, String targetIP, int imgWidth) {

int interval = perfTestService.getMonitorGraphInterval(id, targetIP, imgWidth);

Map sysMonitorMap = perfTestService.getMonitorGraph(id, targetIP, interval);

PerfTest perfTest = perfTestService.getOne(id);

sysMonitorMap.put("interval", String.valueOf(interval * (perfTest != null ? perfTest.getSamplingInterval() : 1)));

return sysMonitorMap;

}

/**

* Get the monitor data of the target having the given IP.

*

* @param id test Id

* @param targetIP targetIP

* @param imgWidth image width

* @return json message

*/

@RestAPI

@RequestMapping("/api/{id}/monitor")

public HttpEntity getMonitorGraph(@PathVariable("id") long id,

@RequestParam("targetIP") String targetIP, @RequestParam int imgWidth) {

return toJsonHttpEntity(getMonitorGraphData(id, targetIP, imgWidth));

}