ansible k8s inventory使用说明及示例

kubernetes.core.k8s inventory插件

使用如下命令可以查看插件的使用方法。

# ansible-doc -t inventory kubernetes.core.k8s

示例如下, 文档中说：“File must be named k8s.yaml or k8s.yml”。插件的配置名，必须是k8s.yaml或k8s.yml。但是实际好像并不是这样。

# cat k8s.yaml 
plugin: k8s
connections:
  - kubeconfig: /data/apps/admin.kubeconfig

然后使用ansible-inventory命令可以获取到集群内的信息。

# ansible-inventory -i k8s.yaml --list > /tmp/k8s_inventory.out

inventory的组织层次

• 10-6-56-10_7443

  • 包含了所有以namespace命名的子组,如namespace_default, namespace_devops

• _meta

  • 主要是hostvars

• all

  • 包含10-6-56-10_7443这个组
  • 包含以lable_开头的各个组
    • 如label_app.kubernetes.io/name_www
    • label_app_contour
    • 再如label_pod-template-hash_54899cb8f9
  • 不包含以namespace_开头的各个组

• 以namespace开头命名的组名

  • 如namespace_default(包含下面两个)
    • namespace_default_pods
      • 所有的pod名（实际上是${pod}_${container}这样的名称）
    • namespace_default_services
      • 就是其名称空间的svc名称

• 以label开头命名的组名, 把各个label都实现了分组

  • 如label_pod-template-hash_7cf4cfb78f
    • 此组内，只有pod名称
  • label_app.kubernetes.io/name_app1

kubectl connection连接插件

由于在_meta中，已经为各主机配置了"ansible_connection": "kubernetes.core.kubectl"变量。

# cat /tmp/k8s_inventory.out  |jq '."_meta"."hostvars"."app1-6bc54678fd-btsfp_app1"'

{
  "annotations": {
    "kubectl.kubernetes.io/restartedAt": "2021-08-12T16:27:37+08:00",
    "prometheus_io_path": "/metrics",
    "prometheus_io_port": "18080",
    "prometheus_io_scrape": "true"
  },
  "ansible_connection": "kubernetes.core.kubectl",
  "ansible_kubectl_container": "app1",
  "ansible_kubectl_namespace": "test1",
  "ansible_kubectl_pod": "app1-6bc54678fd-btsfp",
  "ansible_remote_tmp": "/tmp/",
  "cluster_name": null,
  "container_image": "10.6.63.12:5000/apps/app1:dev_20210423k8s-leo_2021042312094701",
  "container_name": "10.6.63.12:5000/apps/app1:dev_20210423k8s-leo_2021042312094701",
  "container_ready": true,
  "container_state": "Running",
  "docker_cgroupdriver": "systemd",
  "docker_insecure_registry": [],
  "labels": {
    "app.kubernetes.io/instance": "app1",
    "app.kubernetes.io/name": "app1",
    "pod-template-hash": "6bc54678fd"
  },
  "object_type": "pod",
  "pod_host_ip": "10.6.56.163",
  "pod_ip": "10.20.65.69",
  "pod_name": null,
  "pod_node_name": "k8s-node-56-163",
  "pod_phase": "Running",
  "pod_resource_version": "77030206",
  "pod_self_link": null,
  "pod_uid": "eb95536c-04ce-4d40-8abe-242cecabc686"
}
                

所以可以直接以inventory中的组名或${pod}_${container}名（不能直接使用pod的名称）来进行连接，而不需要再为其指定使用kubectl这个connection的插件。

# k get pod
NAME                                 READY   STATUS    RESTARTS   AGE
app1                                 1/1     Running   1          207d
mynginxdeployment-6c9b57b5b8-q447q   1/1     Running   0          65d
mynginxdeployment-6c9b57b5b8-v9plj   1/1     Running   0          65d
mynginxdeployment-6c9b57b5b8-x6hwn   1/1     Running   0          65d
mynginxdeployment-74f48d646b-jxph7   1/1     Running   0          65d
nginx-deploy-2-7b7b5f64cf-qq5sl      1/1     Running   2030       173d
nginx-deploy-2-7b7b5f64cf-zj5fx      1/1     Running   2030       173d
nginx-deploy-7b7b5f64cf-h4dt4        1/1     Running   2444       207d
nginx-deploy-7b7b5f64cf-j42vw        1/1     Running   2444       207d

连接单个pod

# ansible -i k8s.yaml nginx-deploy-7b7b5f64cf-j42vw_nginx -m shell -a "pwd; date"
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
nginx-deploy-7b7b5f64cf-j42vw_nginx | CHANGED | rc=0 >>
/
Mon Feb 28 06:23:09 UTC 2022

连接deploy

使用app.kubernetes.io/name的label

使用label来操作。这个会因为其中包含了svc的名称，而报一个错误，不过其他的存在的pod还是会有结果的。

# ansible  -i k8s.yaml label_app.kubernetes.io/name_app1 -m shell -a "date; hostname"

使用名为app的label

这种app: app1的label是仅包含pod名称列表的组名。

# ansible  -i k8s.yaml label_app_app1 -m shell -a "date; hostname"

问题

service名称的用处

以label开头的组内，总是包含一个以svc名称命名的主机，如label_app.kubernetes.io/instance_app1的hosts中有一个app1, 这个app1不是一个真正的pod, 所以在操作时，会报错。

# ansible -i k8s.yaml label_app.kubernetes.io/name_app1 -m shell -a "hostname"
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
app1 | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname app1: Name or service not known",
    "unreachable": true
}
app1-558cdfc558-72flq_app1 | CHANGED | rc=0 >>
app1-558cdfc558-72flq

后发现，这个app1，可能指代的是namespace_ox_services组中hosts中指代的app1。

但是这个app1的host，到底是有什么作用呢？

alermanager的组中没有额外的host:

"label_app.kubernetes.io/instance_main": {
        "hosts": [
            "alertmanager-main-0_alertmanager",
            "alertmanager-main-0_config-reloader",
            "alertmanager-main-1_alertmanager",
            "alertmanager-main-1_config-reloader",
            "alertmanager-main-2_alertmanager",
            "alertmanager-main-2_config-reloader"
        ]
    }

其namespace_monitoring_services中确实没有main这个svc。

"namespace_monitoring_services": {
        "hosts": [
            "alertmanager-main",
            "alertmanager-operated",
            "blackbox-exporter",
            "grafana",
            "kube-state-metrics",
            "node-exporter",
            "outer-node-exporter",
            "prometheus-adapter",
            "prometheus-k8s",
            "prometheus-operated",
            "prometheus-operator"
        ]
    },

有可能是app.kubernetes.io/name和app.kubernetes.io/instance这两个label比较特殊的原因。

无法操作没有python环境的pod

# ansible -i k8s.yaml grafana-5d5658d6d9-2sp2z_grafana -m shell -a "date"
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
[WARNING]: No python interpreters found for host grafana-5d5658d6d9-2sp2z_grafana (tried ['python3.10', 'python3.9', 'python3.8', 'python3.7', 'python3.6', 'python3.5',
'/usr/bin/python3', '/usr/libexec/platform-python', 'python2.7', '/usr/bin/python', 'python'])
grafana-5d5658d6d9-2sp2z_grafana | FAILED! => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": false,
    "module_stderr": "/bin/sh: /usr/bin/python: not found\ncommand terminated with exit code 127\n",
    "module_stdout": "",
    "msg": "The module failed to execute correctly, you probably need to set the interpreter.\nSee stdout/stderr for the exact error",
    "rc": 127
}

kubernetes.core.k8s_exec模块

这个模块相比kubectl connection插件的好处是，pod只要可以进行kubectl exec就可以执行命令，而不一定需要有py环境。

这个模块主要适合单个pod的情况，如果需要在多个pod中运行操作，那么可以自行获得deployment中的pod, 对pod列表进行轮询。

# ansible   localhost -m kubernetes.core.k8s_exec -a "kubeconfig={{lookup('env','KUBECONFIG')}} namespace=monitoring pod=grafana-7c4bdc5894-pfqt5 command=pwd"
[DEPRECATION WARNING]: The 'return_code' return key is deprecated. Please use 'rc' instead. This feature will be removed from kubernetes.core in version 4.0.0. Deprecation
 warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
localhost | CHANGED => {
    "changed": true,
    "rc": 0,
    "return_code": 0,
    "stderr": "",
    "stderr_lines": [],
    "stdout": "/usr/share/grafana\n",
    "stdout_lines": [
        "/usr/share/grafana"
    ]
}