kubenetes的pod和namespace处于Terminating状态的删除办法

 

1. POD处于Terminating状态的删除办法

情况如下：

[Tilyp@master ~]$ kubectl get pods --namespace kubeflow
NAME                          READY   STATUS        RESTARTS   AGE
ambassador-75b956fb48-vthq7   1/1     Terminating   11         13h
[Tilyp@master ~]$ 

通过查看日志没有发现有用的信息，尝试删除时却处于一直在等待中，强制删除时报错

[Tilyp@master AIKubeflow]$ kubectl delete pods --all --grace-period=0 --force
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "ambassador-75b956fb48-nfb94" force deleted
pod "centraldashboard-7d69c8b4b7-fq69d" force deleted
pod "jupyter-0" force deleted
pod "metacontroller-0" force deleted
pod "minio-d79b65988-2bmxr" force deleted
pod "mysql-ffc889689-96z5x" force deleted
pod "pytorch-operator-7989865774-dnbqg" force deleted
pod "studyjob-controller-654759f8b7-prfwj" force deleted
pod "tf-job-dashboard-85f7b946df-8z4mw" force deleted
pod "tf-job-operator-v1beta1-59b7dbd899-p9cbx" force deleted
pod "vizier-core-rest-7c77f4dc69-mqdw4" force deleted
pod "vizier-db-86dc7d89c5-c7mhz" force deleted
pod "vizier-suggestion-bayesianoptimization-847c8477dc-86jw8" force deleted
pod "workflow-controller-cf79dfbff-rd222" force deleted
[Tilyp@master AIKubeflow]

强制删除指定pod 

kubectl delete pod PODNAME  -n NAMESPACE --grace-period=0 --force

PODNAME:pod名称
NAMESPACE:命名空间

也可以利用创建该pod的文件来强制删除

kubectl delete -f your_file.yaml -n NAMESPACE --grace-period=0 --force

your_file：创建pod的yaml文件
NAMESPACE: 命名空间

 也可以从etcd的存储资源入手来删除该pod，查找该pod的etcd资源：

[Tilyp@master bin]$ ETCDCTL_API=3 /k8s/etcd/bin/etcdctl get "" --from-key > keys.yaml
[Tilyp@master bin]$ ls
etcd  etcdctl  keys.yaml
[Tilyp@master bin]$ ls

由于该资源中数据量很大，要查找很慢，所以就在vim中先输入该pod名称，然后通过shift+#查找该资源，

找到该资源后删除即可

[Tilyp@master bin]$ ETCDCTL_API=3 /k8s/etcd/bin/etcdctl del /registry/pods/kubeflow/ambassador-75b956fb48-vthq7
1
[Tilyp@master bin]$ 

查看pod状态发现该pod已经删除了

[Tilyp@master ~]$ kubectl get pods --namespace kubeflow
No resources found.
[Tilyp@master ~]$

2. NAMESPACE处于Terminating状态的删除办法

        情况如下：

[Tilyp@master bin]$ kubectl get ns
NAME                 STATUS        AGE
angel-group          Active        52d
default              Active        59d
ingress-nginx        Terminating   34d
istio-system         Terminating   2d17h
knative-serving      Active        2d17h
kube-node-lease      Active        59d
kube-public          Active        59d
kube-system          Active        59d
kubeflow             Terminating   2d17h
kubeflow-anonymous   Active        59d
notebook             Terminating   39d

        解决办法是重新打开一个终端执行kubectl proxy跑一个API代理在本地的8081端口

[Tilyp@master bin]$ kubectl proxy --port=8081
Starting to serve on 127.0.0.1:8081

       将namespace信息以json格式输出到文件中

[Tilyp@master bin]$ kubectl get namespace ingress-nginx  -o json > temp.json
{
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "annotations": {
            "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/name\":\"ingress-nginx\",\"app.kubernetes.io/part-of\":\"ingress-nginx\"},\"name\":\"ingress-nginx\"}}\n"
        },
        "creationTimestamp": "2019-11-11T02:25:34Z",
        "deletionTimestamp": "2019-12-13T08:49:55Z",
        "labels": {
            "app.kubernetes.io/name": "ingress-nginx",
            "app.kubernetes.io/part-of": "ingress-nginx"
        },
        "name": "ingress-nginx",
        "resourceVersion": "8112132",
        "selfLink": "/api/v1/namespaces/ingress-nginx",
        "uid": "8a61ccbc-042a-11ea-a893-1866dae6f3a4"
    },
    "spec": {
        "finalizers": [
            "kubernetes"
        ]
    },
    "status": {
        "phase": "Terminating"
    }
}

        打开temp.json，修改以下内容：

修改前====>>
"spec" {
    "finalizers": [
        "kubernetes"
    ]
},

修改后=====>>
"spec" {
    "finalizers": [
    ]
},

       执行以下命令：

[Tilyp@master bin]$ curl -k -H "Content-Type: application/json" -X PUT --data-binary @temp.json 127.0.0.1:8081/api/v1/namespaces/ingress-nginx/finalize
{
  "kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
    "name": "ingress-nginx",
    "selfLink": "/api/v1/namespaces/ingress-nginx/finalize",
    "uid": "8a61ccbc-042a-11ea-a893-1866dae6f3a4",
    "resourceVersion": "8476774",
    "creationTimestamp": "2019-11-11T02:25:34Z",
    "deletionTimestamp": "2019-12-13T08:49:55Z",
    "labels": {
      "app.kubernetes.io/name": "ingress-nginx",
      "app.kubernetes.io/part-of": "ingress-nginx"
    },
    "annotations": {
      "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/name\":\"ingress-nginx\",\"app.kubernetes.io/part-of\":\"ingress-nginx\"},\"name\":\"ingress-nginx\"}}\n"
    }
  },
  "spec": {
    
  },
  "status": {
    "phase": "Terminating"
  }
}

       查看结果：

[Tilyp@master bin]$ kubectl get ns
NAME                 STATUS        AGE
angel-group          Active        52d
default              Active        59d
istio-system         Terminating   2d17h
knative-serving      Active        2d17h
kube-node-lease      Active        59d
kube-public          Active        59d
kube-system          Active        59d
kubeflow             Terminating   2d17h
kubeflow-anonymous   Active        59d
notebook             Terminating   39d

    发现istio-system已经删除，当所有的都已删除完成后，关闭proxy进程

问题以解决，如有问题请加技术交流群：526855734

参考：https://blog.csdn.net/weixin_34082789/article/details/85969414

https://blog.csdn.net/yangqinjiang/article/details/81389201