[运维]cAdvisor 使用教程

cAdvisor 使用教程

• cAdivisor 作用:分析docker中正在运行的容器的资源使用情况和性能特征。

简单使用

• docker运行命令

sudo docker run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:rw \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --volume=/dev/disk/:/dev/disk:ro \
  --publish=8081:8080 \
  --detach=true \
  --name=cadvisor \
  google/cadvisor:latest

• 浏览器浏览http://localhost:8080/即可看到docker中正在运行的容器的资源使用情况和性能特征。
• 完毕

存储插件(持久化监控到的信息)

• BigQuery. See the documentation for usage.
• ElasticSearch. See the documentation for usage and examples.
• InfluxDB. See the documentation for usage and examples.
• Kafka. See the documentation for usage.
• Prometheus. See the documentation for usage and examples.
• Redis
• StatsD
• stdout - write stats to standard output.

集成grafana(用于UI展示),InfluxDB(用于存储)

• docker-compose一键安装cAdivsor+grafana+InfluxDB

version: '3.2'
services:
  influxsrv:
    image: tutum/influxdb:latest
    ports:
      - "8083:8083"
      - "8086:8086"
    expose:
      - "8090"
      - "8099"
    environment:
      - PRE_CREATE_DB=cadvisor
  cadvisor:
    image: google/cadvisor
    command: -storage_driver=influxdb -storage_driver_db=cadvisor -storage_driver_host=influxsrv:8086
    ports:
      - "8081:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    links:
      - influxsrv:influxsrv
  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    links:
      - influxsrv:influxsrv
    environment:
      - INFLUXDB_HOST=localhost
      - INFLUXDB_PORT=8086
      - INFLUXDB_NAME=cadvisor
      - INFLUXDB_USER=root
      - INFLUXDB_PASS=root

• 分别安装

  • grafana安装

    wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana_5.0.3_amd64.deb
    sudo apt-get install -y adduser libfontconfig
    sudo dpkg -i grafana_5.0.3_amd64.deb

  • InfluxDB安装

  • docker方式

    docker run  -p 8084:8083 -p 8086:8086 -p 8090:8090 -p 8099:8099 influxdb:latest

  • 本地方式

    wget https://dl.influxdata.com/influxdb/releases/influxdb_1.5.1_amd64.deb
    sudo dpkg -i influxdb_1.5.1_amd64.deb

  • 创建测试数据库cadvisor

    curl -XPOST "http://localhost:8086/query" --data-urlencode "q=CREATE DATABASE cadvisor"

  • 浏览数据语句(全部集成好时可用于测试)

    • 查询数据库中的表:SHOW MEASUREMENTS
    • 查询cpu_usage_system表中的键key:SHOW TAG KEYS FROM "cpu_usage_system"
    • 查询cpu使用:SHOW TAG VALUES FROM "cpu_usage_system" WITH KEY = "container_name"
    • SHOW TAG VALUES FROM "load_average" WITH KEY = "machine"
    • SHOW TAG VALUES FROM “load_average” WITH KEY = “com.docker.compose.service”
    • SHOW TAG VALUES FROM “cpu_usage_system” WITH KEY = “machine”
    • SHOW TAG VALUES FROM “cpu_usage_system” WITH KEY = “container_name”
  • 安装完成后使用shell命令influx执行CREATE USER root WITH PASSWORD 'root'添加用户root;开启身份认证（可选）

• cAdivisor安装（设置持久化到influxDB的cadivisor表）,详细参考

  sudo docker run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:rw \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --volume=/dev/disk/:/dev/disk:ro \
  --publish=8081:8080 \
  --detach=true \
  google/cadvisor:latest \
  -storage_driver=influxdb \
  -storage_driver_db ="cadvisor" \
  -storage_driver_host="127.0.0.1:8086" 

使用教程

• 首先在浏览器打开Grafana:http://localhost:3000

• 添加influxDB数据源

  • 在Grafana主页点击Create your first data source,进入添加数据源界面添加influxDB数据源
  • 
  • 数据源配置中的Http settings中的Url改成localhost:8086(也可以自己加host映射),Access改成direct(改成直连);用户名和密码不填

• 添加仪表板

  • 一键添加:点击左侧”+”加号图片,选择import导入damo配置
    json
{
"__inputs": [
{
"name": "DS_INFLUXDB",
"label": "influxDB",
"description": "",
"type": "datasource",
"pluginId": "influxdb",
"pluginName": "InfluxDB"
}
],
"__requires": [
{
"type": "grafana",
"id": "grafana",
"name": "Grafana",
"version": "5.0.3"
},
{
"type": "panel",
"id": "graph",
"name": "Graph",
"version": "5.0.0"
},
{
"type": "datasource",
"id": "influxdb",
"name": "InfluxDB",
"version": "5.0.0"
},
{
"type": "panel",
"id": "table",
"name": "Table",
"version": "5.0.0"
}
],
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"gnetId": null,
"graphTooltip": 0,
"id": null,
"iteration": 1522292599072,
"links": [],
"panels": [
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "${DS_INFLUXDB}",
"fill": 1,
"gridPos": {
"h": 9,
"w": 12,
"x": 0,
"y": 0
},
"id": 4,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"groupBy": [
{
"params": [
"1m"
],
"type": "time"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "memory_usage",
"orderByTime": "ASC",
"policy": "default",
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"value"
],
"type": "field"
},
{
"params": [],
"type": "mean"
}
]
],
"tags": [
{
"key": "machine",
"operator": "=~",
"value": "/^$mathine$/"
},
{
"condition": "AND",
"key": "container_name",
"operator": "=~",
"value": "/^$container$/"
}
]
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "数据显示",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "bytes",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
]
},
{
"columns": [],
"datasource": "${DS_INFLUXDB}",
"fontSize": "100%",
"gridPos": {
"h": 9,
"w": 12,
"x": 0,
"y": 9
},
"id": 2,
"links": [],
"pageSize": null,
"scroll": true,
"showHeader": true,
"sort": {
"col": 1,
"desc": true
},
"styles": [
{
"alias": "Time",
"dateFormat": "YYYY-MM-DD HH:mm:ss",
"pattern": "Time",
"type": "date"
},
{
"alias": "",
"colorMode": null,
"colors": [
"rgba(245, 54, 54, 0.9)",
"rgba(237, 129, 40, 0.89)",
"rgba(50, 172, 45, 0.97)"
],
"decimals": 2,
"pattern": "/.*/",
"thresholds": [],
"type": "number",
"unit": "short"
}
],
"targets": [
{
"groupBy": [
{
"params": [
"$__interval"
],
"type": "time"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "memory_usage",
"orderByTime": "ASC",
"policy": "default",
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"value"
],
"type": "field"
},
{
"params": [],
"type": "mean"
}
]
],
"tags": [
{
"key": "container_name",
"operator": "=~",
"value": "/^$container$/"
}
]
},
{
"groupBy": [
{
"params": [
"$__interval"
],
"type": "time"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"orderByTime": "ASC",
"policy": "default",
"refId": "B",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"value"
],
"type": "field"
},
{
"params": [],
"type": "mean"
}
]
],
"tags": []
}
],
"title": "表格展示方式",
"transform": "timeseries_to_columns",
"type": "table"
}
],
"schemaVersion": 16,
"style": "dark",
"tags": [],
"templating": {
"list": [
{
"allValue": null,
"current": {},
"datasource": "${DS_INFLUXDB}",
"hide": 0,
"includeAll": false,
"label": "选择要查询的机子",
"multi": false,
"name": "mathine",
"options": [],
"query": "SHOW TAG VALUES FROM \"cpu_usage_system\" WITH KEY = \"machine\"",
"refresh": 1,
"regex": "",
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allValue": null,
"current": {},
"datasource": "${DS_INFLUXDB}",
"hide": 0,
"includeAll": false,
"label": "要查询的docker容器",
"multi": false,
"name": "container",
"options": [],
"query": "SHOW TAG VALUES FROM \"cpu_usage_system\" WITH KEY = \"container_name\" ",
"refresh": 1,
"regex": "",
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
}
]
},
"time": {
"from": "now-6h",
"to": "now"
},
"timepicker": {
"refresh_intervals": [
"5s",
"10s",
"30s",
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
],
"time_options": [
"5m",
"15m",
"1h",
"6h",
"12h",
"24h",
"2d",
"7d",
"30d"
]
},
"timezone": "",
"title": "ShowData",
"uid": "6Z1nJ_zmk",
"version": 5
}


  • 手动添加:参考教程的添加仪表盘部分

  • 添加完成后可以在主要的左下角的仪表盘列表中看到名称为ShowData的仪表盘，点进去即可看到cAdvisor提供的信息

• influxDB了解

  • 时间序列数据库了解
  • influxDB教程
  • influxDB常用聚集函数
  • 查看docker容器cadvisorjenkins_influxsrv_1的所有内存使用信息
    

    SELECT * FROM "memory_usage" WHERE container_name = "cadvisorjenkins_influxsrv_1"

集成elasticsearch

• 使用docker-compose运行下面yml文件（一键集成）

elasticsearch:
  image: elasticsearch:2.4.4
  volumes:
   - ~/esdata:/usr/share/elasticsearch/data
  ports:
    - "9200:9200"
    - "9300:9300"
cadvisor:
  image: google/cadvisor:latest
  restart: always
  ports:
    - "8888:8080"
  volumes:
    - /:/rootfs:ro
    - /var/run:/var/run:rw
    - /sys:/sys:ro
    - /var/lib/docker/:/var/lib/docker:ro
  links: 
    - elasticsearch
  command: -storage_driver="elasticsearch" -alsologtostderr=true -storage_driver_es_host="http://elasticsearch:9200"