请启用 Javascript 以查看内容

如何彻底删除EKS中一直卡在Terminating的Namespace

 ·  ☕ 4 分钟  ·  ✍ CNSRE

作者:SRE运维博客
博客地址:https://www.cnsre.cn/
文章地址:https://www.cnsre.cn/posts/220107906441/
相关话题:https://www.cnsre.cn/tags/eks/


介绍

最近一直在玩EKS(Elastic Kubernetes Service – Amazon EKS) 和 kubesphere。 因为之前没有使用过EKS 和 kubesphere 所以这个过程也是一个试错的过程,在我使用 kubesphere 的时候发现有一个日志服务,在好奇心的驱使下,我创建了它。

在我创建了日志服务(KubeSphere Logging System)以后,我发现我并不想使用它。(可能我只是想看看它到底是什么吧。)强迫症我的,就想把它给删除掉。于是我在我的 EKS 中对他进行了强制删除 kubectl delete ns kubesphere-logging-system --force --grace-period=0

让人尴尬的是,这个 Namespace 并没有立马删除,我自我安慰到,可能 Namespace 下边有其他没有删除的资源在等待删除,我在等等。。。

过了半个小时,我去查看的时候

[root@ip-10-0-100-206 ~]# kubectl get  ns  kubesphere-logging-system
NAME                              STATUS        AGE
kubesphere-logging-system         Terminating   6d19h

它好像这地卡在了 Terminating 的状态。
我试着寻找解决方法,http://github.com/kubernetes/kubernetes/issues/60807 但是这中方法要通过 API 才可以实现。EKS 是托管在 AWS 中的,我根本没有办法去操作eks的后台。

我开始后悔我为什么要去安装它,我本可以不折腾的。但是回想如果我不去安装它的话,我肯定会像想在这样难过,因为我无法了解它(KubeSphere Logging System)到底是什么样的。

终于我找到了 commented https://github.com/kubernetes/kubernetes/issues/60807#issuecomment-663853215

如何彻底删除 namespace

获取namespace的详情信息并转为json

1
kubectl get namespace kubesphere-logging-system -o json > kubesphere-logging-system.json

打开 json 文件编辑

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
{
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "creationTimestamp": "2021-12-31T05:03:58Z",
        "deletionTimestamp": "2022-01-05T08:05:40Z",
        "labels": {
            "kubesphere.io/namespace": "kubesphere-logging-system",
            "kubesphere.io/workspace": "system-workspace"
        },
        "managedFields": [
            {
                "apiVersion": "v1",
                "fieldsType": "FieldsV1",
                "fieldsV1": {
                    "f:metadata": {
                        "f:labels": {
                            ".": {},
                            "f:kubesphere.io/namespace": {}
                        },
                        "f:ownerReferences": {
                            ".": {},
                            "k:{\"uid\":\"6d535470-2592-4f3c-a155-eabc362c339d\"}": {
                                ".": {},
                                "f:apiVersion": {},
                                "f:blockOwnerDeletion": {},
                                "f:controller": {},
                                "f:kind": {},
                                "f:name": {},
                                "f:uid": {}
                            }
                        }
                    }
                },
                "manager": "controller-manager",
                "operation": "Update",
                "time": "2021-12-31T05:04:01Z"
            },
            {
                "apiVersion": "v1",
                "fieldsType": "FieldsV1",
                "fieldsV1": {
                    "f:metadata": {
                        "f:labels": {
                            "f:kubesphere.io/workspace": {}
                        }
                    },
                    "f:status": {
                        "f:phase": {}
                    }
                },
                "manager": "kubectl",
                "operation": "Update",
                "time": "2021-12-31T05:04:01Z"
            },
            {
                "apiVersion": "v1",
                "fieldsType": "FieldsV1",
                "fieldsV1": {
                    "f:status": {
                        "f:conditions": {
                            ".": {},
                            "k:{\"type\":\"NamespaceContentRemaining\"}": {
                                ".": {},
                                "f:lastTransitionTime": {},
                                "f:message": {},
                                "f:reason": {},
                                "f:status": {},
                                "f:type": {}
                            },
                            "k:{\"type\":\"NamespaceDeletionContentFailure\"}": {
                                ".": {},
                                "f:lastTransitionTime": {},
                                "f:message": {},
                                "f:reason": {},
                                "f:status": {},
                                "f:type": {}
                            },
                            "k:{\"type\":\"NamespaceDeletionDiscoveryFailure\"}": {
                                ".": {},
                                "f:lastTransitionTime": {},
                                "f:message": {},
                                "f:reason": {},
                                "f:status": {},
                                "f:type": {}
                            },
                            "k:{\"type\":\"NamespaceDeletionGroupVersionParsingFailure\"}": {
                                ".": {},
                                "f:lastTransitionTime": {},
                                "f:message": {},
                                "f:reason": {},
                                "f:status": {},
                                "f:type": {}
                            },
                            "k:{\"type\":\"NamespaceFinalizersRemaining\"}": {
                                ".": {},
                                "f:lastTransitionTime": {},
                                "f:message": {},
                                "f:reason": {},
                                "f:status": {},
                                "f:type": {}
                            }
                        }
                    }
                },
                "manager": "kube-controller-manager",
                "operation": "Update",
                "time": "2022-01-05T08:05:47Z"
            }
        ],
        "name": "kubesphere-logging-system",
        "ownerReferences": [
            {
                "apiVersion": "tenant.kubesphere.io/v1alpha1",
                "blockOwnerDeletion": true,
                "controller": true,
                "kind": "Workspace",
                "name": "system-workspace",
                "uid": "6d535470-2592-4f3c-a155-eabc362c339d"
            }
        ],
        "resourceVersion": "7376520",
        "uid": "2b76e9b1-75f2-4a2e-a819-73b36aea188e"
    },
    "spec": {
        "finalizers": [
            "kubernetes" # 将此行删除
        ]
    },
    "status": {
        "conditions": [
            {
                "lastTransitionTime": "2022-01-05T08:05:47Z",
                "message": "All resources successfully discovered",
                "reason": "ResourcesDiscovered",
                "status": "False",
                "type": "NamespaceDeletionDiscoveryFailure"
            },
            {
                "lastTransitionTime": "2022-01-05T08:05:47Z",
                "message": "All legacy kube types successfully parsed",
                "reason": "ParsedGroupVersions",
                "status": "False",
                "type": "NamespaceDeletionGroupVersionParsingFailure"
            },
            {
                "lastTransitionTime": "2022-01-05T08:05:47Z",
                "message": "All content successfully deleted, may be waiting on finalization",
                "reason": "ContentDeleted",
                "status": "False",
                "type": "NamespaceDeletionContentFailure"
            },
            {
                "lastTransitionTime": "2022-01-05T08:05:47Z",
                "message": "Some resources are remaining: fluentbits.logging.kubesphere.io has 1 resource instances",
                "reason": "SomeResourcesRemain",
                "status": "True",
                "type": "NamespaceContentRemaining"
            },
            {
                "lastTransitionTime": "2022-01-05T08:05:47Z",
                "message": "Some content in the namespace has finalizers remaining: fluentbit.logging.kubesphere.io in 1 resource instances",
                "reason": "SomeFinalizersRemain",
                "status": "True",
                "type": "NamespaceFinalizersRemaining"
            }
        ],
        "phase": "Terminating"
    }
}

找到 spec 将 finalizers 下的 kubernetes 删除
具体如下

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
{
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "creationTimestamp": "2021-12-31T05:03:58Z",
        "deletionTimestamp": "2022-01-05T08:05:40Z",
        "labels": {
            "kubesphere.io/namespace": "kubesphere-logging-system",
            "kubesphere.io/workspace": "system-workspace"
        },
        "managedFields": [
            {
                "apiVersion": "v1",
                "fieldsType": "FieldsV1",
                "fieldsV1": {
                    "f:metadata": {
                        "f:labels": {
                            ".": {},
                            "f:kubesphere.io/namespace": {}
                        },
                        "f:ownerReferences": {
                            ".": {},
                            "k:{\"uid\":\"6d535470-2592-4f3c-a155-eabc362c339d\"}": {
                                ".": {},
                                "f:apiVersion": {},
                                "f:blockOwnerDeletion": {},
                                "f:controller": {},
                                "f:kind": {},
                                "f:name": {},
                                "f:uid": {}
                            }
                        }
                    }
                },
                "manager": "controller-manager",
                "operation": "Update",
                "time": "2021-12-31T05:04:01Z"
            },
            {
                "apiVersion": "v1",
                "fieldsType": "FieldsV1",
                "fieldsV1": {
                    "f:metadata": {
                        "f:labels": {
                            "f:kubesphere.io/workspace": {}
                        }
                    },
                    "f:status": {
                        "f:phase": {}
                    }
                },
                "manager": "kubectl",
                "operation": "Update",
                "time": "2021-12-31T05:04:01Z"
            },
            {
                "apiVersion": "v1",
                "fieldsType": "FieldsV1",
                "fieldsV1": {
                    "f:status": {
                        "f:conditions": {
                            ".": {},
                            "k:{\"type\":\"NamespaceContentRemaining\"}": {
                                ".": {},
                                "f:lastTransitionTime": {},
                                "f:message": {},
                                "f:reason": {},
                                "f:status": {},
                                "f:type": {}
                            },
                            "k:{\"type\":\"NamespaceDeletionContentFailure\"}": {
                                ".": {},
                                "f:lastTransitionTime": {},
                                "f:message": {},
                                "f:reason": {},
                                "f:status": {},
                                "f:type": {}
                            },
                            "k:{\"type\":\"NamespaceDeletionDiscoveryFailure\"}": {
                                ".": {},
                                "f:lastTransitionTime": {},
                                "f:message": {},
                                "f:reason": {},
                                "f:status": {},
                                "f:type": {}
                            },
                            "k:{\"type\":\"NamespaceDeletionGroupVersionParsingFailure\"}": {
                                ".": {},
                                "f:lastTransitionTime": {},
                                "f:message": {},
                                "f:reason": {},
                                "f:status": {},
                                "f:type": {}
                            },
                            "k:{\"type\":\"NamespaceFinalizersRemaining\"}": {
                                ".": {},
                                "f:lastTransitionTime": {},
                                "f:message": {},
                                "f:reason": {},
                                "f:status": {},
                                "f:type": {}
                            }
                        }
                    }
                },
                "manager": "kube-controller-manager",
                "operation": "Update",
                "time": "2022-01-05T08:05:47Z"
            }
        ],
        "name": "kubesphere-logging-system",
        "ownerReferences": [
            {
                "apiVersion": "tenant.kubesphere.io/v1alpha1",
                "blockOwnerDeletion": true,
                "controller": true,
                "kind": "Workspace",
                "name": "system-workspace",
                "uid": "6d535470-2592-4f3c-a155-eabc362c339d"
            }
        ],
        "resourceVersion": "7376520",
        "uid": "2b76e9b1-75f2-4a2e-a819-73b36aea188e"
    },
    "spec": {
        "finalizers": [
        ]
    },
    "status": {
        "conditions": [
            {
                "lastTransitionTime": "2022-01-05T08:05:47Z",
                "message": "All resources successfully discovered",
                "reason": "ResourcesDiscovered",
                "status": "False",
                "type": "NamespaceDeletionDiscoveryFailure"
            },
            {
                "lastTransitionTime": "2022-01-05T08:05:47Z",
                "message": "All legacy kube types successfully parsed",
                "reason": "ParsedGroupVersions",
                "status": "False",
                "type": "NamespaceDeletionGroupVersionParsingFailure"
            },
            {
                "lastTransitionTime": "2022-01-05T08:05:47Z",
                "message": "All content successfully deleted, may be waiting on finalization",
                "reason": "ContentDeleted",
                "status": "False",
                "type": "NamespaceDeletionContentFailure"
            },
            {
                "lastTransitionTime": "2022-01-05T08:05:47Z",
                "message": "Some resources are remaining: fluentbits.logging.kubesphere.io has 1 resource instances",
                "reason": "SomeResourcesRemain",
                "status": "True",
                "type": "NamespaceContentRemaining"
            },
            {
                "lastTransitionTime": "2022-01-05T08:05:47Z",
                "message": "Some content in the namespace has finalizers remaining: fluentbit.logging.kubesphere.io in 1 resource instances",
                "reason": "SomeFinalizersRemain",
                "status": "True",
                "type": "NamespaceFinalizersRemaining"
            }
        ],
        "phase": "Terminating"
    }
}

执行清理命令

现在我们只需要一条命令 就可以彻底删除这个 Namespace

1
 kubectl replace --raw "/api/v1/namespaces/kubesphere-logging-system/finalize" -f ./kubesphere-logging-system.json 

执行完以后,你需要等待一会
再次执行命令检查 namespaces

1
 kubectl replace --raw "/api/v1/namespaces/kubesphere-logging-system/finalize" -f ./kubesphere-logging-system.json 

最后的检查

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
kubectl get  ns  kubesphere-logging-system
Error from server (NotFound): namespaces "kubesphere-logging-system" not found

[root@ip-10-0-100-206 ~]# kubectl get  ns 
NAME                              STATUS   AGE
default                           Active   23d
kubesphere-controls-system        Active   9d
kubesphere-devops-system          Active   9d
kubesphere-devops-worker          Active   16h
kubesphere-monitoring-federated   Active   9d
kubesphere-monitoring-system      Active   9d
kubesphere-sample-dev             Active   8d
kubesphere-system                 Active   9d

在此查看的时候,它已经不存在了。


作者:SRE运维博客
博客地址:https://www.cnsre.cn/
文章地址:https://www.cnsre.cn/posts/220107906441/
相关话题:https://www.cnsre.cn/tags/eks/


您的鼓励是我最大的动力
alipay QR Code
wechat QR Code

Avatar
作者
CNSRE
一位只会重启的运维


目录