1.Nightingale系统简介
夜莺是一套分布式高可用的运维监控系统,最大的特点是混合云支持,既可以支持传统物理机虚拟机的场景,也可以支持K8S容器的场景。同时,夜莺也不只是监控,还有一部分CMDB的能力、自动化运维的能力,很多公司都基于夜莺开发自己公司的运维平台。开源的这部分功能模块也是商业版本的一部分,所以可靠性有保障、会持续维护,诸君可放心使用。
2.夜莺部署
本文档着重介绍夜莺监控JVM,部署手册详见:Welcome to tengine!
3.实施步骤
3.1 安装agent服务
3.1.1 安装夜莺
3.1.2 执行 ./control start all
3.1.3 查看代理端口2080使用情况
3.1.4 查看夜莺系统节点情况
3.2 Nightingale对Java程序埋点监控
3.3.1 版本更新
1.6.5更新说明
新增了自动注册节点功能,依赖同步为micrometer-core 1.6.5版本
1.6.4更新说明
新增自动获取endpoint功能,新增指标屏蔽功能
1.6.3更新说明
初始版本,实现数据上报到n9e中
3.3.2 使用方法
- 如果是spring boot项目,引入micrometer-registry-nightingale-boot-starter依赖
- 进行必要的参数配置,如果agent和jvm是在同一台机器的,只需要配置 management.metrics.export.nightingale.enabled=true,即可自动上报,其余配置信息请参考参数配置
management:
metrics:
export:
nightingale:
addr: agent地址
endpoint: http://127.0.0.1:2080/endpoint 端点IP,如未配置则自动获得本机IP,如果配置的是URL则自动获取URL的值,启动时获取,后续不会自动刷新
nid: 如果配置了nid,则视为机器无关指标,endpoint配置不再生效
step: 上报频率,默认配置10S,格式为java.time.Duration
append-tags: 应用附加Tag, 格式"key1=value1,key2=value2"
enabled: 是否启用(true|false)
metric-block-list: #需要屏蔽的metric
auto-registry: true #开启自动注册,开启后应用将会自动注册到nid节点下面,成为一个资源节点。开启此功能时必须要配置nid,否则会有异常
api-addr: "http://n9e.com" #夜莺的服务器地址
user-token: "token" #调用夜莺API的token
3.3.3 maven依赖
项目已经发布到了maven仓库中,在项目中添加依赖可以直接使用
micrometer-registry-nightingale
<dependency>
<groupId>com.github.lynxcat</groupId>
<artifactId>micrometer-registry-nightingale</artifactId>
<version>1.6.5</version>
</dependency>
micrometer-registry-nightingale-boot-starter
<dependency>
<groupId>com.github.lynxcat</groupId>
<artifactId>micrometer-registry-nightingale-boot-starter</artifactId>
<version>1.6.5</version>
</dependency>
3.3.4 依赖说明(重要)
项目底层依赖micrometer-core 1.6.5版本,低版本的spring-boot-starter-actuator中包含的micrometer-core版本比较低,可以在项目依赖中添加高版本micrometer-core来解决,高于1.5.6版本即可。此项目基于spring boot 2.3.x版本开发,高于这个版本不会有依赖问题
3.3 本地试用
3.3.1 添加Maven依赖
3.3.2 修改本地配置
3.3.2.1 重点参数说明
management. metrics.export.nightingale.addr:夜莺agent服务(步骤3.1安装)的ip和prot,默认为2080
management. metrics.export.nightingale.endpoint:夜莺系统中节点IP(详见3.3.2.2)
management. metrics.export.nightingale.api-addr:夜莺系统服务地址
management. metrics.export.nightingale.user-token:夜莺系统用户秘钥(详见3.3.2.3)
3.3.2.2 节点IP查看
3.3.2.3 用户秘钥查看
夜莺系统-个人设置-秘钥管理-Token
3.3.3 本地执行
3.3.4 数据结构
[
{
"timestamp":1627269583,
"metric":"jvm.buffer.memory.used",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"id=direct",
"value":8192.0
},
{
"timestamp":1627269583,
"metric":"jvm.threads.states",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"state=blocked",
"value":0.0
},
{
"timestamp":1627269583,
"metric":"jvm.memory.committed",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"area=heap,id=PS-Old-Gen",
"value":1.22159104E8
},
{
"timestamp":1627269583,
"metric":"process.uptime",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"value":13514.0
},
{
"timestamp":1627269583,
"metric":"jvm.memory.used",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"area=nonheap,id=Compressed-Class-Space",
"value":4627304.0
},
{
"timestamp":1627269583,
"metric":"jvm.threads.states",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"state=runnable",
"value":11.0
},
{
"timestamp":1627269583,
"metric":"jvm.gc.pause",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"action=end-of-major-GC,cause=Metadata-GC-Threshold",
"value":1
},
{
"timestamp":1627269583,
"metric":"jvm.gc.pause.sum",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"action=end-of-major-GC,cause=Metadata-GC-Threshold",
"value":58.0
},
{
"timestamp":1627269583,
"metric":"jvm.gc.pause.mean",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"action=end-of-major-GC,cause=Metadata-GC-Threshold",
"value":58.0
},
{
"timestamp":1627269583,
"metric":"jvm.gc.pause.max",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"action=end-of-major-GC,cause=Metadata-GC-Threshold",
"value":58.0
},
{
"timestamp":1627269583,
"metric":"jvm.memory.committed",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"area=heap,id=PS-Survivor-Space",
"value":1.7825792E7
},
{
"timestamp":1627269583,
"metric":"jvm.memory.used",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"area=nonheap,id=Metaspace",
"value":3.3840184E7
},
{
"timestamp":1627269583,
"metric":"jvm.memory.committed",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"area=heap,id=PS-Eden-Space",
"value":2.6476544E8
},
{
"timestamp":1627269583,
"metric":"logback.events",
"counterType":"COUNTER",
"step":10,
"endpoint":"10.0.53.125",
"tags":"level=trace",
"value":0.0
},
{
"timestamp":1627269583,
"metric":"jvm.memory.max",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"area=nonheap,id=Code-Cache",
"value":2.5165824E8
},
{
"timestamp":1627269583,
"metric":"tomcat.sessions.active.current",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"value":0.0
},
{
"timestamp":1627269583,
"metric":"jvm.buffer.total.capacity",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"id=direct",
"value":8192.0
},
{
"timestamp":1627269583,
"metric":"jvm.gc.pause",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"action=end-of-minor-GC,cause=Metadata-GC-Threshold",
"value":1
},
{
"timestamp":1627269583,
"metric":"jvm.gc.pause.sum",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"action=end-of-minor-GC,cause=Metadata-GC-Threshold",
"value":8.0
},
{
"timestamp":1627269583,
"metric":"jvm.gc.pause.mean",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"action=end-of-minor-GC,cause=Metadata-GC-Threshold",
"value":8.0
},
{
"timestamp":1627269583,
"metric":"jvm.gc.pause.max",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"action=end-of-minor-GC,cause=Metadata-GC-Threshold",
"value":8.0
},
{
"timestamp":1627269584,
"metric":"system.cpu.usage",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"value":0.0
},
{
"timestamp":1627269584,
"metric":"jvm.threads.live",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"value":27.0
},
{
"timestamp":1627269584,
"metric":"jvm.classes.unloaded",
"counterType":"COUNTER",
"step":10,
"endpoint":"10.0.53.125",
"value":0.0
},
{
"timestamp":1627269584,
"metric":"jvm.memory.committed",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"area=nonheap,id=Compressed-Class-Space",
"value":5111808.0
},
{
"timestamp":1627269584,
"metric":"jvm.memory.used",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"area=heap,id=PS-Survivor-Space",
"value":0.0
},
{
"timestamp":1627269584,
"metric":"jvm.classes.loaded",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"value":6943.0
},
{
"timestamp":1627269584,
"metric":"jvm.threads.peak",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"value":30.0
},
{
"timestamp":1627269584,
"metric":"jvm.gc.pause",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"action=end-of-minor-GC,cause=Allocation-Failure",
"value":1
},
{
"timestamp":1627269584,
"metric":"jvm.gc.pause.sum",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"action=end-of-minor-GC,cause=Allocation-Failure",
"value":12.0
},
{
"timestamp":1627269584,
"metric":"jvm.gc.pause.mean",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"action=end-of-minor-GC,cause=Allocation-Failure",
"value":12.0
},
{
"timestamp":1627269584,
"metric":"jvm.gc.pause.max",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"action=end-of-minor-GC,cause=Allocation-Failure",
"value":12.0
},
{
"timestamp":1627269584,
"metric":"logback.events",
"counterType":"COUNTER",
"step":10,
"endpoint":"10.0.53.125",
"tags":"level=warn",
"value":0.0
},
{
"timestamp":1627269584,
"metric":"jvm.memory.used",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"area=heap,id=PS-Eden-Space",
"value":2.0772184E7
},
{
"timestamp":1627269584,
"metric":"tomcat.sessions.rejected",
"counterType":"COUNTER",
"step":10,
"endpoint":"10.0.53.125",
"value":0.0
},
{
"timestamp":1627269584,
"metric":"tomcat.sessions.alive.max",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"value":0.0
},
{
"timestamp":1627269584,
"metric":"logback.events",
"counterType":"COUNTER",
"step":10,
"endpoint":"10.0.53.125",
"tags":"level=error",
"value":0.0
},
{
"timestamp":1627269584,
"metric":"jvm.gc.memory.promoted",
"counterType":"COUNTER",
"step":10,
"endpoint":"10.0.53.125",
"value":4211176.0
},
{
"timestamp":1627269584,
"metric":"jvm.memory.max",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"area=nonheap,id=Compressed-Class-Space",
"value":1.073741824E9
},
{
"timestamp":1627269584,
"metric":"lynxcat.metrics.test",
"counterType":"COUNTER",
"step":10,
"endpoint":"10.0.53.125",
"tags":"country=china",
"value":0.0
},
{
"timestamp":1627269584,
"metric":"jvm.threads.states",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"state=waiting",
"value":13.0
},
{
"timestamp":1627269584,
"metric":"jvm.memory.max",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"area=heap,id=PS-Old-Gen",
"value":2.845310976E9
},
{
"timestamp":1627269584,
"metric":"jvm.buffer.count",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"id=mapped",
"value":0.0
},
{
"timestamp":1627269584,
"metric":"http.server.requests",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"exception=None,method=GET,outcome=SUCCESS,status=200,uri=/home",
"value":1
},
{
"timestamp":1627269584,
"metric":"http.server.requests.sum",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"exception=None,method=GET,outcome=SUCCESS,status=200,uri=/home",
"value":47.6738
},
{
"timestamp":1627269584,
"metric":"http.server.requests.mean",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"exception=None,method=GET,outcome=SUCCESS,status=200,uri=/home",
"value":47.6738
},
{
"timestamp":1627269584,
"metric":"http.server.requests.max",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"exception=None,method=GET,outcome=SUCCESS,status=200,uri=/home",
"value":47.6738
},
{
"timestamp":1627269584,
"metric":"jvm.memory.max",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"area=heap,id=PS-Survivor-Space",
"value":1.7825792E7
},
{
"timestamp":1627269584,
"metric":"jvm.memory.used",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"area=nonheap,id=Code-Cache",
"value":7710208.0
},
{
"timestamp":1627269584,
"metric":"jvm.gc.memory.allocated",
"counterType":"COUNTER",
"step":10,
"endpoint":"10.0.53.125",
"value":2.00602024E8
},
{
"timestamp":1627269584,
"metric":"logback.events",
"counterType":"COUNTER",
"step":10,
"endpoint":"10.0.53.125",
"tags":"level=info",
"value":7.0
},
{
"timestamp":1627269584,
"metric":"tomcat.sessions.expired",
"counterType":"COUNTER",
"step":10,
"endpoint":"10.0.53.125",
"value":0.0
},
{
"timestamp":1627269584,
"metric":"tomcat.sessions.created",
"counterType":"COUNTER",
"step":10,
"endpoint":"10.0.53.125",
"value":0.0
},
{
"timestamp":1627269584,
"metric":"jvm.buffer.total.capacity",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"id=mapped",
"value":0.0
},
{
"timestamp":1627269584,
"metric":"jvm.gc.max.data.size",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"value":2.845310976E9
},
{
"timestamp":1627269584,
"metric":"system.cpu.count",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"value":4.0
},
{
"timestamp":1627269584,
"metric":"process.start.time",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"value":1.627269570237E12
},
{
"timestamp":1627269584,
"metric":"jvm.buffer.count",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"id=direct",
"value":1.0
},
{
"timestamp":1627269584,
"metric":"jvm.buffer.memory.used",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"id=mapped",
"value":0.0
},
{
"timestamp":1627269584,
"metric":"jvm.memory.used",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"area=heap,id=PS-Old-Gen",
"value":1.4539256E7
},
{
"timestamp":1627269584,
"metric":"jvm.memory.committed",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"area=nonheap,id=Code-Cache",
"value":7733248.0
},
{
"timestamp":1627269584,
"metric":"jvm.threads.daemon",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"value":23.0
},
{
"timestamp":1627269584,
"metric":"jvm.threads.states",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"state=new",
"value":0.0
},
{
"timestamp":1627269584,
"metric":"tomcat.sessions.active.max",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"value":0.0
},
{
"timestamp":1627269584,
"metric":"jvm.threads.states",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"state=timed-waiting",
"value":3.0
},
{
"timestamp":1627269584,
"metric":"jvm.memory.committed",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"area=nonheap,id=Metaspace",
"value":3.6569088E7
},
{
"timestamp":1627269584,
"metric":"jvm.memory.max",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"area=nonheap,id=Metaspace",
"value":-1.0
},
{
"timestamp":1627269584,
"metric":"jvm.threads.states",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"state=terminated",
"value":0.0
},
{
"timestamp":1627269584,
"metric":"jvm.memory.max",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"tags":"area=heap,id=PS-Eden-Space",
"value":1.38674176E9
},
{
"timestamp":1627269584,
"metric":"jvm.gc.live.data.size",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"value":1.4539256E7
},
{
"timestamp":1627269584,
"metric":"logback.events",
"counterType":"COUNTER",
"step":10,
"endpoint":"10.0.53.125",
"tags":"level=debug",
"value":0.0
},
{
"timestamp":1627269584,
"metric":"process.cpu.usage",
"counterType":"GAUGE",
"step":10,
"endpoint":"10.0.53.125",
"value":0.2367567731377659
}
]