saltstack结合Elasticsearch来做salt执行结果展现

salt虽然好用但是机器管理的越来越多，通过cli的结果输出方式查看执行结果越来越多不能满足我的需求，而且作为一个推动运维自动化的攻城狮，使用这种人眼查看执行结果的方式简直土到掉渣！虽然别人看起来逼格很高，但谁累谁知道。。。由于以上原因，给各位推荐一种逼格更高的结果查看方式：

salt returners

先来看一下官方结构图：

Send data returned by Salt Minions to another system, such as a database. Returners can run on the Salt Minion or on the Salt Master.

这个图上画的是redis，其实官方给出的returners有很多很多，这次为了装逼！哦不对，为了更好地筛选和过滤salt的执行结果，我选用了elasticsearch+kibana这种展现方式，好处是es提供一个准实时的全文检索引擎，这样我们可以实时的想搜什么搜什么，多维度过滤，而kibana提供的是便捷的用户界面和帅的一逼的画图界面！

但是，这两者是怎么结合起来的呢，很简单，用一个es支持salt也支持的数据中转站就行了，我选择的是kafka，当然你也可以用rsyslog，redis，等等，任君选择

下面就以 salt-minion -> kafka -> logstash -> elasticseach <- kibana <- 你的浏览器 这条主线进行说明，有纰漏的地方欢迎在评论里指正，谢谢！

现在张成果图镇楼：

下面正时开始：

1. salt-minion端的配置：

1.1 修改minion配置文件，这对于会用salt的我们来说改1w台机器就是分分钟的事儿嘛，但注意，改完要重启minion进程。

/etc/salt/minion：

#kafka broker主机，可以配置多个
returner.kafka.hostnames:
  - "10.64.0.1"
  - "10.64.0.2"
  - "10.64.0.3"

#kafka topic名称
returner.kafka.topic: 'saltstack-topic'

1.2 修改官方returner文件源代码，此处高深莫测，先按下不表

2.kafka 几乎不用配置，正常能用即可，顺便推荐一下kafka manager真好用，从此再也不用记命令了。

https://github.com/yahoo/kafka-manager

3.logstash 作为刚才minion发送到kafka数据的消费者，拿到kafka的数据后按json格式存到es中

3.1 logstash.conf：

input {
  kafka {
    #zk 连接串
    zk_connect => '10.64.0.1:2181,10.64.0.2:2181,10.64.0.3:2181,10.64.1.174:2181,10.64.1.175:2181'
    #与刚才配置的topic一致
    group_id => 'logstash-saltstack'
    topic_id => 'saltstack-topic'
    consumer_id => 'logstash-saltstack-consumer-{{ grains["id"] }}'
    consumer_threads => 1
    queue_size => 200
    codec => plain
  }
}

filter {
  mutate {
    add_field => { "fromAgent" => "logstash-saltstack-{{ grains["id"] }}" }
  }
  json {
    source => "message"
  }
  mutate {
    remove_field => [ "message" ]
    convert => { "fun_args" => "string" }
    rename => { "__run_num__" => "run_num" }
  }
}

output {
  elasticsearch {
    #输出到es
    hosts => ["10.64.0.1:9200","10.64.0.2:9200","10.64.0.3:9200"]
    index => "saltstack-%{+YYYY.MM}"
    flush_size => 3000
    idle_flush_time => 2
    workers => 2
  }
  #stdout { codec => rubydebug }
}

这里有个小坑，顺便解释一下刚才按下不表的为啥非要改源码：es收数据的时候会把json全都拆成嵌套的kv格式存储，啥意思呢，请看图：

这是两个命令执行的不同返回，问题在哪儿呢？问题就在10.64.0.1这个key的value，一会儿是boolen型一会儿是nested型，这样es就没法办了，一个字段的类型实在mapping或者第一次创建数据时确定的，不能随便更改，这样就造成了es不能接收数据的问题。

怎么解决？开始我尝试了修改logstash的配置，不过那个实在不灵活。而且你注意看第二个salt执行的结果，是在一个大json里面的，这样我搜索起来匹配上的话也是一整条数据，还是不方便我看结果。

针对以上两点我还是觉得小改一下salt的returners，修改的地方如下（每台minion都要改，不过这次不用重启了）：

/usr/lib/python2.6/site-packages/salt/returners/kafka_return.py：

def returner(ret):
    '''
    Return information to a Kafka server
    '''
    if __salt__['config.option']('returner.kafka.topic'):
        topic = __salt__['config.option']('returner.kafka.topic')

        conn = _get_conn(ret)
        producer = SimpleProducer(conn)
        if ret["return"] == True or ret["return"] == False:
            producer.send_messages(topic, json.dumps(ret))
        else:
            for retKey in ret["return"]:
                myRet = {}
                myRet["id"] = ret["id"]
                myRet["fun"] = ret["fun"]
                myRet["fun_args"] = ret["fun_args"]
                myRet["jid"] = ret["jid"]
                myRet["retcode"] = ret["retcode"]
                for key in ret["return"][retKey]:
                    myRet[key] = ret["return"][retKey][key]
                retKeys = retKey.split("_|-")
                for i in range(0,len(retKeys)):
                    myRet["key"+str(i)] = retKeys[i]
                producer.send_messages(topic, json.dumps(myRet))

        _close_conn(conn)
    else:
        log.error('Unable to find kafka returner config option: topic')

至此我们就应该能在es里看到每个动作一条记录的salt结果了。

4. es 还是没啥可配置的，正常运转就可以

5. kibana

首先，我用的是kibana 3，这个需要es <2.0版本的配合，不要赶新潮装es2，这样就用不了我的模板啦

5.1 载入kibana模板

把下面的内容存成一个文件，然后在kibana中选择载入，然后赶紧跑个salt任务试试

神马？没有，是这样，我忘了跟你说以后跑任务的时候要加个参数才能在es里看到哦：

salt 你要执行的命令 --return kafka

这样你就得到了开始看到的那个激动人心的逼格界面。 我有啥没说清楚的欢迎在下面评论里写明，我会尽力解答，谢谢！

{
  "title": "Saltstack returns",
  "services": {
    "query": {
      "idQueue": [
        1
      ],
      "list": {
        "0": {
          "query": "result:\"true\"",
          "alias": "成功",
          "color": "#7EB26D",
          "id": 0,
          "pin": false,
          "type": "lucene",
          "enable": true
        },
        "4": {
          "id": 4,
          "color": "#E24D42",
          "alias": "失败",
          "pin": false,
          "type": "lucene",
          "enable": true,
          "query": "result:\"false\""
        }
      },
      "ids": [
        0,
        4
      ]
    },
    "filter": {
      "idQueue": [
        1
      ],
      "list": {
        "0": {
          "type": "time",
          "field": "@timestamp",
          "from": "now-1h",
          "to": "now",
          "mandate": "must",
          "active": true,
          "alias": "",
          "id": 0
        }
      },
      "ids": [
        0
      ]
    }
  },
  "rows": [
    {
      "title": "分类统计",
      "height": "367px",
      "editable": true,
      "collapse": true,
      "collapsable": true,
      "panels": [
        {
          "error": false,
          "span": 4,
          "editable": true,
          "type": "terms",
          "loadingEditor": false,
          "field": "type",
          "exclude": [],
          "missing": false,
          "other": false,
          "size": 50,
          "order": "count",
          "style": {
            "font-size": "10pt"
          },
          "donut": false,
          "tilt": false,
          "labels": true,
          "arrangement": "horizontal",
          "chart": "table",
          "counter_pos": "below",
          "spyable": true,
          "queries": {
            "mode": "all",
            "ids": [
              0,
              1,
              2
            ]
          },
          "tmode": "terms",
          "tstat": "total",
          "valuefield": "",
          "title": "type"
        },
        {
          "error": false,
          "span": 4,
          "editable": true,
          "type": "terms",
          "loadingEditor": false,
          "field": "host.raw",
          "exclude": [],
          "missing": false,
          "other": false,
          "size": 11,
          "order": "count",
          "style": {
            "font-size": "10pt"
          },
          "donut": false,
          "tilt": false,
          "labels": true,
          "arrangement": "horizontal",
          "chart": "table",
          "counter_pos": "none",
          "spyable": true,
          "queries": {
            "mode": "all",
            "ids": [
              0,
              1,
              2
            ]
          },
          "tmode": "terms",
          "tstat": "total",
          "valuefield": "",
          "title": "host"
        },
        {
          "error": false,
          "span": 4,
          "editable": true,
          "type": "terms",
          "loadingEditor": false,
          "field": "tags.raw",
          "exclude": [],
          "missing": false,
          "other": false,
          "size": 50,
          "order": "count",
          "style": {
            "font-size": "10pt"
          },
          "donut": false,
          "tilt": false,
          "labels": true,
          "arrangement": "horizontal",
          "chart": "pie",
          "counter_pos": "none",
          "spyable": true,
          "queries": {
            "mode": "all",
            "ids": [
              0,
              1,
              2
            ]
          },
          "tmode": "terms",
          "tstat": "total",
          "valuefield": "",
          "title": "tags"
        }
      ],
      "notice": false
    },
    {
      "title": "Graph",
      "height": "150px",
      "editable": true,
      "collapse": false,
      "collapsable": true,
      "panels": [
        {
          "span": 12,
          "editable": true,
          "group": [
            "default"
          ],
          "type": "histogram",
          "mode": "count",
          "time_field": "@timestamp",
          "value_field": null,
          "auto_int": true,
          "resolution": 200,
          "interval": "30s",
          "fill": 3,
          "linewidth": 3,
          "timezone": "browser",
          "spyable": true,
          "zoomlinks": true,
          "bars": true,
          "stack": true,
          "points": false,
          "lines": false,
          "legend": true,
          "x-axis": true,
          "y-axis": true,
          "percentage": false,
          "interactive": true,
          "queries": {
            "mode": "all",
            "ids": [
              0,
              4
            ]
          },
          "title": "Events over time",
          "intervals": [
            "auto",
            "1s",
            "1m",
            "5m",
            "10m",
            "30m",
            "1h",
            "3h",
            "12h",
            "1d",
            "1w",
            "1M",
            "1y"
          ],
          "options": true,
          "tooltip": {
            "value_type": "cumulative",
            "query_as_alias": true
          },
          "annotate": {
            "enable": false,
            "query": "*",
            "size": 20,
            "field": "_type",
            "sort": [
              "_score",
              "desc"
            ]
          },
          "pointradius": 5,
          "show_query": true,
          "legend_counts": true,
          "zerofill": true,
          "derivative": false,
          "scale": 1,
          "grid": {
            "max": null,
            "min": 0
          },
          "y_format": "none"
        }
      ],
      "notice": false
    },
    {
      "title": "Events",
      "height": "350px",
      "editable": true,
      "collapse": false,
      "collapsable": true,
      "panels": [
        {
          "title": "All events",
          "error": false,
          "span": 12,
          "editable": true,
          "group": [
            "default"
          ],
          "type": "table",
          "size": 100,
          "pages": 5,
          "offset": 0,
          "sort": [
            "@timestamp",
            "desc"
          ],
          "style": {
            "font-size": "9pt"
          },
          "overflow": "min-height",
          "fields": [
            "@timestamp",
            "id",
            "fun",
            "fun_args",
            "key0",
            "key1",
            "key2",
            "key3",
            "result"
          ],
          "highlight": [],
          "sortable": true,
          "header": true,
          "paging": true,
          "spyable": true,
          "queries": {
            "mode": "all",
            "ids": [
              0,
              4
            ]
          },
          "field_list": true,
          "status": "Stable",
          "trimFactor": 300,
          "normTimes": true,
          "all_fields": false,
          "localTime": true,
          "timeField": "@timestamp"
        }
      ],
      "notice": false
    }
  ],
  "editable": true,
  "failover": false,
  "index": {
    "interval": "month",
    "pattern": "[saltstack-]YYYY.MM",
    "default": "NO_TIME_FILTER_OR_INDEX_PATTERN_NOT_MATCHED",
    "warm_fields": true
  },
  "style": "dark",
  "panel_hints": true,
  "pulldowns": [
    {
      "type": "query",
      "collapse": true,
      "notice": false,
      "query": "*",
      "pinned": true,
      "remember": 10,
      "enable": true
    },
    {
      "type": "filtering",
      "collapse": true,
      "notice": false,
      "enable": true
    }
  ],
  "nav": [
    {
      "type": "timepicker",
      "collapse": false,
      "notice": false,
      "status": "Stable",
      "time_options": [
        "5m",
        "15m",
        "1h",
        "6h",
        "12h",
        "24h",
        "2d",
        "7d",
        "30d"
      ],
      "refresh_intervals": [
        "5s",
        "10s",
        "30s",
        "1m",
        "5m",
        "15m",
        "30m",
        "1h",
        "2h",
        "1d"
      ],
      "timefield": "@timestamp",
      "now": true,
      "filter_id": 0,
      "enable": true
    }
  ],
  "loader": {
    "save_gist": false,
    "save_elasticsearch": true,
    "save_local": true,
    "save_default": true,
    "save_temp": true,
    "save_temp_ttl_enable": true,
    "save_temp_ttl": "30d",
    "load_gist": true,
    "load_elasticsearch": true,
    "load_elasticsearch_size": 20,
    "load_local": true,
    "hide": false
  },
  "refresh": false
}