k8s强制删除处于Terminating状态的namespace

 一、问题背景

        按照kubord官方文档安装删除Kuboard之后，再重新执行kubectlapply时，出现Error from server (Forbidden): error when creating "https://addons.kuboard.cn/kuboard/kuboard-v3-swr.yaml": configmaps "kuboard-v3-config" is forbidden: unable to create new content in namespace kuboard because it is being terminated报错。

 二、排查过程

       根据报错显示unable to create new content in namespace kuboard because it is being terminated：大致意思时kuboard命名空间正在销毁，所以无法创建。

        手动排查使用kubectl get ns，发现果然有一个状态为Terminating的kuboard。

使用delete删除这个命名空间，发现强制也无法删除卡在这个状态，无奈Ctrl+C停止。

[root@master ~]# kubectl delete ns kuboard
namespace "kuboard" deleted
^C
[root@master ~]# kubectl delete ns kuboard --force --grace-period=0
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
namespace "kuboard" force deleted
^C

三、解决方法

查看namespace

运行以下命令以查看处于“Terminating”状态的namespace：

kubectl get namespaces

选择Terminating namespace

选择一个Terminating namespace，并查看namespace 中的finalizer。运行以下命令：

kubectl get namespace <terminating-namespace> -o yaml

输出信息如下：

apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: "2019-11-20T15:18:06Z"
  deletionTimestamp: "2020-01-16T02:50:02Z"
  name: <terminating-namespace>
  resourceVersion: "3249493"
  selfLink: /api/v1/namespaces/knative-eventing
  uid: f300ea38-c8c2-4653-b432-b66103e412db
spec:
  finalizers:
  - kubernetes
status:

导出json格式到文件

kubectl get namespace <terminating-namespace> -o json >tmp.json

编辑tmp.josn

编辑tmp.josn，删除finalizers 字段的值

{
  "apiVersion": "v1",
  "kind": "Namespace",
  "metadata": {
    "creationTimestamp": "2019-11-20T15:18:06Z",
    "deletionTimestamp": "2020-01-16T02:50:02Z",
    "name": "<terminating-namespace>",
    "resourceVersion": "3249493",
    "selfLink": "/api/v1/namespaces/knative-eventing",
    "uid": "f300ea38-c8c2-4653-b432-b66103e412db"
  },
  "spec": {    #从此行开始删除
    "finalizers": []
  },   # 删到此行
  "status": {
    "phase": "Terminating"
  }
}

开启proxy

kubectl proxy

执行该命令后，当前终端会被卡住

打开新的一个窗口，执行以下命令

curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/<terminating-namespace>/finalize

输出信息如下：

{
  "kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
    "name": "istio-system",
    "selfLink": "/api/v1/namespaces/istio-system/finalize",
    "uid": "2e274537-727f-4a8f-ae8c-397473ed619a",
    "resourceVersion": "3249492",
    "creationTimestamp": "2019-11-20T15:18:06Z",
    "deletionTimestamp": "2020-01-16T02:50:02Z"
  },
  "spec": {
    
  },
  "status": {
    "phase": "Terminating"
  }
}

确认结果

确认处于Terminating 状态的namespace已经被删除

kubectl get namespaces

如果还有处于Terminating 状态的namespace，重复以上操作，删除即可！