【笔记记录】【敏感信息已混淆】k8s生态下 kubectl命令、pod性能验证及监控

trading# k8s生态下 kubectl命令操作pod性能监控

环境

1C2G 1POD

架构拓扑

查看命名空间，ns就是namespaces

[root@pr-dr-13-47 .kube]# kubectl get ns
NAME                STATUS   AGE
default             Active   327d
itwork              Active   326d
kube-public         Active   327d
kube-system         Active   327d
projectName-dev     Active   326d
projectName-sit     Active   323d
aaaaa-company-dev   Active   326d
aaaaa-company-sit   Active   326d
uat                 Active   255d

查看Node或Pod的资源使用情况，需要在集群中运行Metrics Service

[root@pr-dr-13-47 .kube]# kubectl top nodes 
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
100.20.30.40    152m         1%     24004Mi         83%       
100.20.30.41    225m         1%     24515Mi         85%       
100.20.30.42    287m         1%     24028Mi         83%       
100.20.30.43    148m         2%     20784Mi         72%       
100.20.30.44    327m         4%     21441Mi         74%       
100.20.30.45    107m         2%     2247Mi          68%       
100.20.30.46    69m          1%     2076Mi          63%       
100.20.30.47    126m         1%     20126Mi         70%       
100.20.30.48    107m         2%     2012Mi          61%       
100.20.30.49    163m         2%     20408Mi         71%       
100.20.30.50    158m         2%     23506Mi         81%       
100.20.30.51    135m         1%     19663Mi         68%       
100.20.30.52    150m         2%     17834Mi         62%

查看所有pod的cpu和内存

[root@pr-dr-13-47 .kube]# kubectl top pod -n projectName-sit
NAME                                              CPU(cores)   MEMORY(bytes)   
aa-aaaaa-web-16c9b-6cc4b47f66-wtdwh               6m           2269Mi          
agent-web-7d23a-65645ddb85-ppqbp                  5m           2874Mi          
aaaaaaaa-src-5bd63-648fd86446-5mzz8               8m           1992Mi          
back-aaaaaaa-web-fd620-69447b56d9-xbw7c           103m         2549Mi          
back-aaaaaaa-service-2dd48-76d4dd74fd-btszn       34m          2980Mi          
back-aaaaaaa-web-dcc32-54bc89d9f8-p6p7t           22m          2661Mi          
back-promotion-web-c9efd-776867f767-462hh         10m          2075Mi          
basics-promotion-service-4a92d-7c54b687c6-klqg4   6m           1925Mi          
certificate-9d281-7bfdbfcbd4-pb44m                6m           1536Mi          
aaa-web-2a866-d9c767d6-wf76c                      7m           1956Mi          
aaa-web-0341b-6c5b7c45fc-z86w8                    6m           1829Mi          
frontier-aaaaa-web-2a87b-6558f5c4cd-rsjbh         9m           1699Mi          
h5-b2b-ba02f-866445c976-w7tjl                     0m           148Mi           
h5-b2c-51b44-7b8c45d87f-sgksc                     1m           146Mi           
index-3eb41-5c7b4f5d77-blxbk                      18m          4035Mi          
aaa-web-da544-6dbcd75646-r64d5                    6m           2458Mi          
aaaa-web-e57c4-69458f59f4-c8cn7                   28m          2346Mi          
aaa-scheduler-bccbd-5786b44d98-k7tjs              8m           2332Mi          
aaa-api-e29c7-794c6b974f-qvwqv                    10m          2726Mi          
aaa-dataex-4b411-8655df4df6-mqc6z                 12m          2351Mi          
aaa-task-4eea2-6b7c5b85d7-48t2j                   7m           2967Mi          
aaa-task-54d4c-f974cf699-w95rw                    12m          2431Mi          
aaa-web-15126-6f9c9d7fc7-txtdl                    7m           2577Mi          
aaaa-web-94990-6b8b7b685f-skbzn                   9m           3760Mi          
aaaa-web-a3bfc-7f8f5bf55d-r2rqs                   8m           2764Mi          
aaaa-routing-d0728-69f79c47bf-gmnxg               3m           898Mi           
aaaaa-web-e2c0c-7795b88d97-sdgsq                  15m          2186Mi          
portal-f48a7-f587596cd-4lwtd                      1m           22Mi            
search-2baad-6d85c5b95c-z7mbv                     18m          3224Mi          
aaaaaa-back-web-6c194-c5c6bc88b-zzfgm             7m           1910Mi          
aaaaaa-web-8ca6e-8675d64898-5nj7q                 7m           2019Mi          
vue-aaaaaa-c1ef5-7c857db699-f4dlz                 1m           8Mi             
xxl-aaa-admin-260d7-5477469576-65sww              25m          2087Mi

在Pod和本地之间复制文件

#把Pod上的/etc/fstab复制到本地的/tmp目录
kubectl cp nginx-6ddbbc47-sfdcv:/etc/fstab /tmp

watch kubectl top pod -n projectName-sit serviceName-fd620-69447b56d9-xbw7

top -Hp 1

安装kubectl

window和linux都可以，按照官网步骤走就行

https://kubernetes.io/docs/tasks/tools/

为了让 kubectl 能发现并访问 Kubernetes 集群，home目录下新建.kube 目录并新增文件 config，通常，kubectl 的配置信息存放于文件 ~/.kube/config 中。

apiVersion: v1
clusters:
- cluster:
    cert-authority-data: dsalj98ew7e98w==
    server: https://1.2.3.4:60002
  name: local
contexts:
- context:
    cluster: local
    user: admin
  name: master
current-context: master
kind: Config
preferences: {}
users:
- name: admin
  user:
    token: dsjakhdkjsahdkj28e791e8wqdewqddsa

验证kubectl配置

如果返回一个 URL，则意味着 kubectl 成功的访问到了你的集群

[root@pr-dr-13-47 ~]# kubectl cluster-info
Kubernetes control plane is running at https://1.2.3.4:60002
CoreDNS is running at https://1.2.3.4:60002/api/v1/namespaces/kube-system/services/kube-dns:dns-tcp/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

三秒刷新一次宿主机的cpu和内存资源占用及百分比

[root@pr-dr-13-47 ~]# watch -n 3 kubectl top nodes 111.222.203.200

Every 3.0s: kubectl top nodes 111.222.203.200                                                            
Tue Feb  8 12:57:25 2022

NAME          CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
111.222.203.200   183m         1%     17663Mi         61%

三秒刷新一次容器的cpu和内存资源占用

[root@pr-dr-13-47 ~]# watch -n 3 kubectl top pod -n projectName-sit aaa-web-a07a8-cc75f8b79-k2xrq

Every 3.0s: kubectl top pod -n projectName-sit aaa-web-a07a8-cc75f8b79-k2xrq                                                   Tue Feb  8 12:58:18 2022

NAME                            CPU(cores)   MEMORY(bytes)
aaa-web-a07a8-cc75f8b79-k2xrq   23m          1760Mi

进入aaa-web容器

[root@pr-dr-13-47 ~]# kubectl exec -it aaa-web-a07a8-cc75f8b79-k2xrq  -n projectName-sit -- bash
Defaulted container "aaa-web-a07a8" out of: aaa-web-a07a8, skywalking-agent (init), jmx-agent (init)
root@aaa-web-a07a8-cc75f8b79-k2xrq:/usr/local/tomcat#

查看当前容器的进程id

root@aaa-web-a07a8-cc75f8b79-k2xrq:/usr/local/tomcat# jps
1 Bootstrap
263 Jps

导出进程堆栈

root@aaa-web-a07a8-cc75f8b79-k2xrq:/usr/local/tomcat# jstack 1 > log2022.txt
1: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding

通过命令kubectl cp把堆栈日志从容器拷贝到宿主机上(当前所在目录)

[root@pr-dr-13-47 ~]# kubectl cp projectName-sit/aaa-web-a07a8-cc75f8b79-k2xrq:/usr/local/tomcat/log202202081247 log202202081247.txt
Defaulted container "aaa-web-a07a8" out of: aaa-web-a07a8, skywalking-agent (init), jmx-agent (init)
tar: Removing leading `/' from member names

从宿主机(node)上下载到本地做分析(命令sz或者其他文件上传下载工具)