使用Datadog在docker环境下监控Java, Tomcat, Nginx, Kfaka, ZooKeeper

最近需要使用datadog来监控系统运行情况，由于中间遇到了一些困难，不断的向datadog官方发邮件，幸运的是问题都一一解决了，这里记录一下重要配置方便以后查看：

操作平台：linux ubuntu
docker下面的container有：
server
tomcat
nginx
kafaka

[b]如何只安装一个dd-agent却监测所有的container？[/b]
1. 安装dd-agent container
docker run -d --name dd-agent -h `hostname` -v /var/run/docker.sock:/var/run/docker.sock -v /proc/:/host/proc/:ro -v /cgroup/:/host/sys/fs/cgroup:ro -e API_KEY={your_api_key_here}
datadog/docker-dd-agent

2. 从dd-agent contianer里把cofig里的文件全部拷贝到docker的主机上
docker cp dd-agent:/etc/dd-agent /goeasy/

3. 在docker的主机上 修改conf.d里的配置文件

4. 移除之前创建的dd-agent container
docker rm -f dd-agent

5. 重新启动一个dd-agent且映射conf.d的路径确保每次都从docker主机上读取配置：
sudo weave run 192.168.31.110/24 -d --name dd-agent -h `hostname` -v /goeasy:/goeasy -v /goeasy/dd-agent:/etc/dd-agent -v /var/run/docker.sock:/var/run/docker.sock -v /proc/:/host/proc/:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e API_KEY={your app key} datadog/docker-dd-agent

其中192.168.31.110是我手动设置的dd-agent的IP

[b]如何监测JAVA[/b]
1. 在启动java时加入下列启动参数用来打开JMX remote连接
-Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

2. 在docker host里配置jmx.yaml文件
 

init_config:

instances:
    -   host: {server服务器的IP的地址}
        port: 7199
        name: jmx_instance
        java_bin_path: /goeasy/jdk1.8.0_65/bin/java
        conf:
          - include:
              domain: java.lang
              bean: my_first_bean
              attribute:
                 attributeName:
                    metric_type: counter
                    alias: jmx.my_first_metric

3. 移除之前创建的dd-agent container
docker rm -f dd-agent

4. 重新启动一个dd-agent且映射conf.d的路径确保每次都从docker主机上读取配置：
sudo weave run 192.168.31.110/24 -d --name dd-agent -h `hostname` -v /goeasy:/goeasy -v /goeasy/dd-agent:/etc/dd-agent -v /var/run/docker.sock:/var/run/docker.sock -v /proc/:/host/proc/:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e API_KEY={your app key} datadog/docker-dd-agent

5. 进入到dd-agent container,且用下面的command来看是否监测成功：
/etc/init.d/datadog/ info


[b]如何监测Tomcat[/b]
1. 在tomcat里配置JMX remote connection (catalina.sh的JAVA_OPTS)
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false

2. 在docker host里配置tomcat.yaml文件 (init_config里的配置内容请参考datadog官网)

instances:
  - host: {tomcat的IP}
    port: 7199
    name: tomcat_instance
    java_bin_path: /uwantsoft/public/jre1.8.0_60/bin/java
init_config:
  conf:
    - include:
        type: ThreadPool
        attribute:
          maxThreads:
            alias: tomcat.threads.max
            metric_type: gauge
          currentThreadCount:
            alias: tomcat.threads.count
            metric_type: gauge
          currentThreadsBusy:
            alias: tomcat.threads.busy
            metric_type: gauge

 

3. 移除之前创建的dd-agent container
docker rm -f dd-agent

4. 重新启动一个dd-agent且映射conf.d的路径确保每次都从docker主机上读取配置：
sudo weave run 192.168.31.110/24 -d --name dd-agent -h `hostname` -v /goeasy:/goeasy -v /goeasy/dd-agent:/etc/dd-agent -v /var/run/docker.sock:/var/run/docker.sock -v /proc/:/host/proc/:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e API_KEY={your app key} datadog/docker-dd-agent

5. 进入到dd-agent container,且用下面的command来看是否监测成功：
/etc/init.d/datadog/ info

[b] 如何监测Nginx[/b]
1. 确保在安装nginx的同时也安装了http_stub_status_module的
安装方法参考：http://xiaoxiaoxiqincai.iteye.com/blog/2270140
2. 修改nginx.yaml

init_config:

instances:
    # For every instance, you have an `nginx_status_url` and (optionally)
    # a list of tags.

    -   nginx_status_url: http://172.17.0.1/nginx_status/
        tags:
            -   instance:foo

 

3. 移除之前创建的dd-agent container
docker rm -f dd-agent

4. 重新启动一个dd-agent且映射conf.d的路径确保每次都从docker主机上读取配置：
sudo weave run 192.168.31.110/24 -d --name dd-agent -h `hostname` -v /goeasy:/goeasy -v /goeasy/dd-agent:/etc/dd-agent -v /var/run/docker.sock:/var/run/docker.sock -v /proc/:/host/proc/:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e API_KEY={your app key} datadog/docker-dd-agent

5. 进入到dd-agent container,且用下面的command来看是否监测成功：
/etc/init.d/datadog/ info


2. 在docker host里配置nginx.yaml

init_config:

instances:
    # For every instance, you have an `nginx_status_url` and (optionally)
    # a list of tags.

    -   nginx_status_url: http://{nginx的IP}/nginx_status/
        tags:
            -   instance:foo

 

 

[b]如何监测Kafka[/b]
1. 配置kafka,确保JMX remote connection是打开的 （Kafka根目录/bin/kafka-server-start.sh）
修改下面配置：
export KAFKA_HEAP_OPTS="-Xmx512m -Xms512m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"

2. 配置kafka.yaml文件 (init_config里的配置内容请参考datadog官网)

##########
# WARNING
##########
# This sample works only for Kafka >= 0.8.2.
# If you are running a version older than that, you can refer to agent 5.2.x released
# sample files, https://raw.githubusercontent.com/DataDog/dd-agent/5.2.1/conf.d/kafka.yaml.example

instances:
  - host: {kafka的IP}
    port: 9999
    name: kafka_instance
    java_bin_path: /uwantsoft/public/jre1.8.0_60/bin/java

init_config:
  is_jmx: true

  # Metrics collected by this check. You should not have to modify this.
  conf:
    #
    # Aggregate cluster stats
    #
    - include:
        domain: 'kafka.server'
        bean: 'kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec'
        attribute:
          MeanRate:
            metric_type: gauge
            alias: kafka.net.bytes_out

3. 移除之前创建的dd-agent container
docker rm -f dd-agent

4. 重新启动一个dd-agent且映射conf.d的路径确保每次都从docker主机上读取配置：
sudo weave run 192.168.31.110/24 -d --name dd-agent -h `hostname` -v /goeasy:/goeasy -v /goeasy/dd-agent:/etc/dd-agent -v /var/run/docker.sock:/var/run/docker.sock -v /proc/:/host/proc/:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e API_KEY={your app key} datadog/docker-dd-agent

5. 进入到dd-agent container,且用下面的command来看是否监测成功：
/etc/init.d/datadog/ info

[b]如何监测ZooKeeper[/b]
1. 配置zk.yaml

init_config:

instances:
  - host: {ZooKeeper的IP}
    port: 2181
    # timeout: 3
    # tags:
    #   - optional_tag1
    #   - optional_tag2

 

2. 移除之前创建的dd-agent container
docker rm -f dd-agent

3. 重新启动一个dd-agent且映射conf.d的路径确保每次都从docker主机上读取配置：
sudo weave run 192.168.31.110/24 -d --name dd-agent -h `hostname` -v /goeasy:/goeasy -v /goeasy/dd-agent:/etc/dd-agent -v /var/run/docker.sock:/var/run/docker.sock -v /proc/:/host/proc/:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e API_KEY={your app key} datadog/docker-dd-agent

4. 进入到dd-agent container,且用下面的command来看是否监测成功：
/etc/init.d/datadog/ info

[b]如何监测docker[/b]

1. 配置docker.yaml

init_config:

instances:
    - url: "unix://var/run/docker.sock"
      new_tag_names: true

 

2. 移除之前创建的dd-agent container
docker rm -f dd-agent

3. 重新启动一个dd-agent且映射conf.d的路径确保每次都从docker主机上读取配置：
sudo weave run 192.168.31.110/24 -d --name dd-agent -h `hostname` -v /goeasy:/goeasy -v /goeasy/dd-agent:/etc/dd-agent -v /var/run/docker.sock:/var/run/docker.sock -v /proc/:/host/proc/:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e API_KEY={your app key} datadog/docker-dd-agent

4. 进入到dd-agent container,且用下面的command来看是否监测成功：
/etc/init.d/datadog/ info