使用Kubeadm搭建Kubernetes集群
kubeadm
kubeadm概述
kubernetes有多种部署方式，已知的主要有kind、minikube、kubeadm、二进制包、Rancher
kubeadm是官方社区推出的一个用于快速部署kubernetes集群的工具。这个工具能通过两条指令完成一个kubernetes集群的部署
# 创建一个Master节点
kubeadm init
# 将一个Node节点加入到当前集群中
kubeadm join <Master 节点的IP 和端口>
官网：https://kubernetes.io/zh/
中文社区: https://www.kubernetes.org.cn/
搭建过程
1.在所有节点上安装Docker和kubeadm
2.部署Kubernetes Master
4.部署Kubernetes Node，将节点加入Kubernetes集群中
3.部署容器网络插件
5.Kubernetes的基本使用
初始化准备
关闭防火墙
# 临时
systemctl stop firewalld
# 永久
systemctl disable firewalld
关闭Selinux
# 永久
sed -i 's/enforcing/disabled/' /etc/selinux/config
# 临时
setenforce 0
关闭Swap
# 临时
swapoff -a
# 永久;把文件中带有swap的行注释
vim /etc/fstab
添加主机名与IP对应关系
hostnamectl set-hostname <hostname>
172.29.234.1 hostnamectl set-hostname node001
172.29.234.2 hostnamectl set-hostname node002
172.29.234.3 hostnamectl set-hostname node003
在master添加hosts设置
cat >> /etc/hosts << EOF
172.29.234.1 node001 node001
172.29.234.2 node002 node002
172.29.234.3 node003 node003
EOF
配置内核参数
配置内核参数，将桥接的IPv4流量传递到iptables的链
[root@node001 ~]# cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
配置时间同步
# 查看时间
date
yum install -y ntpdate
# 同步最新时间
ntpdate time.windows.com
搭建Kubernetes集群
安装Docker
在所有节点安装Docker
# 添加Docker阿里镜像源
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
# 所有docker版本
yum list docker-ce --showduplicates | sort -r
# 选择特定版本安装
yum install <FQPN>
# 默认最新稳定版
yum install docker-ce
# 设置开启启动并立即启动Docker
systemctl enable docker && systemctl start docker
# docker版本
docker --version
设置Yum软件源
设置kubernetes源 vim /etc/yum.repos.d/kubernetes.repo
# 添加阿里云Kubernetes yum镜像源
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
设置docker拉取镜像仓库的源 vim /etc/docker/daemon.json
[root@master ~]# cat /etc/docker/daemon.json
{
"registry-mirrors": ["https://a5amagpr.mirror.aliyuncs.com"]
}
将配置文件分发到其他节点
[root@node001 ~]# scp /etc/yum.repos.d/kubernetes.repo node002:/etc/yum.repos.d/
kubernetes.repo 100% 275 1.1MB/s 00:00
[root@node001 ~]# scp /etc/yum.repos.d/kubernetes.repo node003:/etc/yum.repos.d/
kubernetes.repo
# 注意:先每个节点: mkdir /etc/docker
[root@node001 ~]# scp /etc/docker/daemon.json node002:/etc/docker/
daemon.json 100% 68 346.0KB/s 00:00
[root@node001 ~]# scp /etc/docker/daemon.json node003:/etc/docker/
daemon.json 100% 68 331.5KB/s 00:00
[root@node001 ~]#
所有节点重启Docker
[root@node001 ~]# systemctl daemon-reload
[root@node001 ~]# systemctl restart docker
安装kubeadm、kubelet、kubectl
在所有节点安装，注意: 所有节点版本必须一致
# 默认下载最新版
yum install -y kubelet kubeadm kubectl
# 指定稳定版本下载
yum install -y kubelet-1.18.0 kubeadm-1.18.0 kubectl-1.18.0
# 开机自启
systemctl enable kubelet
部署Kubernetes Master
在Master节点(node001)部署Kubernetes Master
--apiserver-advertise-address：api server地址，指定机器IP，主节点内网地址
--image-repository： 默认拉取国外镜像地址k8s.gcr.io，修改指定拉取阿里云镜像仓库地址
--kubernetes-version：指定Kubernetes版本
--service-cidr：service网段，自定义使用IP，不冲突即可
--pod-network-cidr：pod的网段，自定义使用IP，不冲突即可
kubeadm init \
--apiserver-advertise-address=172.29.234.1 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.18.0 \
--service-cidr=10.10.0.0/16 \
--pod-network-cidr=10.20.0.0/16
注意：除了修改拉取镜像地址外，还可以先从国内仓库拉取镜像，然后给镜像打个Tag，也可以到阿里云镜像仓库找到对应镜像单独拉取。
批量脚本拉取如下：
#!/bin/bash
images=(
kube-apiserver:v1.17.3
kube-proxy:v1.17.3
kube-controller-manager:v1.17.3
kube-scheduler:v1.17.3
coredns:1.6.5
etcd:3.4.3-0
pause:3.1
)
for imageName in ${images[@]} ; do
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
# docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName
done
出现警告，需要执行systemctl enable kubelet与 swapoff -a关闭swap
[WARNING Swap]: swap is enabled; production deployments should disable swap unless testing the NodeSwap feature gate of the kubelet
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
出现异常：
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
在/etc/docker/daemon.json配置文件中加入"exec-opts": ["native.cgroupdriver=systemd"]，并重启Docker
[root@master ~]# cat /etc/docker/daemon.json
{
"registry-mirrors": ["https://a5amagpr.mirror.aliyuncs.com"],
"exec-opts": ["native.cgroupdriver=systemd"]
}
[root@master ~]# systemctl restart docker
清除kubeadm信息后重新初始化
kubeadm reset -f
当出现Your Kubernetes control-plane has initialized successfully!即安装成功
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 172.29.234.1:6443 --token j3yoqc.n0puscwoaw024cer \
--discovery-token-ca-cert-hash sha256:a319ee2c305bcf30661cc70ae4a1c8790e450ef77fb908e5473f813019c03c19
查看拉取的相关镜像
[root@node001 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cd4ae6cff8e1 43940c34f24f "/usr/local/bin/kube…" 43 seconds ago Up 43 seconds k8s_kube-proxy_kube-proxy-8nhxg_kube-system_68da55c6-8cc3-456d-9ef0-40c08c79476c_0
07baba228a19 registry.aliyuncs.com/google_containers/pause:3.2 "/pause" 44 seconds ago Up 43 seconds k8s_POD_kube-proxy-8nhxg_kube-system_68da55c6-8cc3-456d-9ef0-40c08c79476c_0
177181973b54 a31f78c7c8ce "kube-scheduler --au…" About a minute ago Up About a minute k8s_kube-scheduler_kube-scheduler-node001_kube-system_ca2aa1b3224c37fa1791ef6c7d883bbe_0
ad07f05aaafe d3e55153f52f "kube-controller-man…" About a minute ago Up About a minute k8s_kube-controller-manager_kube-controller-manager-node001_kube-system_c3e36b1ca7f02f5d08f86ab49a445523_0
f63eb32e060c 303ce5db0e90 "etcd --advertise-cl…" About a minute ago Up About a minute k8s_etcd_etcd-node001_kube-system_1650fba102ae7fd48af91ae7227a8918_0
4ff5afd36074 74060cea7f70 "kube-apiserver --ad…" About a minute ago Up About a minute k8s_kube-apiserver_kube-apiserver-node001_kube-system_c7f4ce820e11c35aea50e314696db0c5_0
66de0ba4c586 registry.aliyuncs.com/google_containers/pause:3.2 "/pause" About a minute ago Up About a minute k8s_POD_kube-scheduler-node001_kube-system_ca2aa1b3224c37fa1791ef6c7d883bbe_0
5795be27b3d6 registry.aliyuncs.com/google_containers/pause:3.2 "/pause" About a minute ago Up About a minute k8s_POD_kube-controller-manager-node001_kube-system_c3e36b1ca7f02f5d08f86ab49a445523_0
402958c279a1 registry.aliyuncs.com/google_containers/pause:3.2 "/pause" About a minute ago Up About a minute k8s_POD_kube-apiserver-node001_kube-system_c7f4ce820e11c35aea50e314696db0c5_0
0fe24c7bf783 registry.aliyuncs.com/google_containers/pause:3.2 "/pause" About a minute ago Up About a minute k8s_POD_etcd-node001_kube-system_1650fba102ae7fd48af91ae7227a8918_0
按初始化提示执行
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
执行kubectl get nodes获取所有节点
[root@node001 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
node001 NotReady master 2m35s v1.18.0
Node节点加入K8S集群
获取Node001节点执行kubeadm init输出的kubeadm join命令：
kubeadm join 172.29.234.1:6443 --token j3yoqc.n0puscwoaw024cer \
--discovery-token-ca-cert-hash sha256:a319ee2c305bcf30661cc70ae4a1c8790e450ef77fb908e5473f813019c03c19
拿到此命令后，在所有Node节点执行该命令，向集群添加新节点，Mast节点不能执行该命令
[root@node002 ~]# kubeadm join 172.29.234.1:6443 --token j3yoqc.n0puscwoaw024cer \
> --discovery-token-ca-cert-hash sha256:a319ee2c305bcf30661cc70ae4a1c8790e450ef77fb908e5473f813019c03c19
W0319 13:23:34.581734 14132 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.13. Latest validated version: 19.03
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
master节点查询集群状态和数量： kubectl get nodes
[root@node001 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
node001 NotReady master 7m18s v1.18.0
node002 NotReady <none> 3m9s v1.18.0
node003 NotReady <none> 84s v1.18.0
安装Pod网络插件
节点状态：NotReady，因为corednspod没有启动，缺少网络Pod，安装Pod网络插件
# 下载kube-flannel.yml
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# 将kube-flannel.yml里的配置应用到flannel中
kubectl apply -f kube-flannel.yml
注意：kube-flannel.yml中指定的images可能无法访问，可以去docker hub、阿里镜像仓库找一个然后修改其地址
查看创建了哪些pod
kubectl get pods -n kube-system 查看指定名称空间的pods
kubectl get pods –all-namespace 查看所有名称空间的pods
[root@node001 ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-7ff77c879f-nh4sw 0/1 Pending 0 9m38s
coredns-7ff77c879f-rmm7q 0/1 Pending 0 9m38s
etcd-node001 1/1 Running 0 9m47s
kube-apiserver-node001 1/1 Running 0 9m47s
kube-controller-manager-node001 1/1 Running 0 9m47s
kube-flannel-ds-sskm9 0/1 Init:0/2 0 45s
kube-flannel-ds-t7jkg 0/1 Init:0/2 0 45s
kube-flannel-ds-zmxsv 0/1 Init:0/2 0 45s
kube-proxy-4dtsb 1/1 Running 0 4m3s
kube-proxy-8nhxg 1/1 Running 0 9m38s
kube-proxy-z2b58 1/1 Running 0 5m48s
kube-scheduler-node001 1/1 Running 0 9m47s
需耐心等待所有Pod启动成功
[root@node001 ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-7ff77c879f-nh4sw 0/1 Running 0 13m
coredns-7ff77c879f-rmm7q 0/1 Running 0 13m
etcd-node001 1/1 Running 0 13m
kube-apiserver-node001 1/1 Running 0 13m
kube-controller-manager-node001 1/1 Running 0 13m
kube-flannel-ds-sskm9 1/1 Running 0 4m39s
kube-flannel-ds-t7jkg 1/1 Running 0 4m39s
kube-flannel-ds-zmxsv 1/1 Running 0 4m39s
kube-proxy-4dtsb 1/1 Running 0 7m57s
kube-proxy-8nhxg 1/1 Running 0 13m
kube-proxy-z2b58 1/1 Running 0 9m42s
kube-scheduler-node001 1/1 Running 0 13m
master节点查询集群状态和数量： kubectl get nodes
[root@node001 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
node001 Ready master 14m v1.18.0
node002 Ready <none> 10m v1.18.0
node003 Ready <none> 8m42s v1.18.0
kubernetes集群测试
建一个资源对象deployment，服务名称：nginx，拉取镜像：nginx
kubectl create deployment nginx --image=nginx
暴露端口
# Pod的80(--port)映射容器的80(--target-port)
# service代理Pod的80
kubectl expose deployment nginx --port=80 --target-port=80 --type=NodePort
查看nginx运行在哪个节点上，还有具体的访问端口
[root@node001 ~]# kubectl get pod,svc -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/nginx-f89759699-6bbsk 1/1 Running 0 2m8s 10.20.1.2 node002 <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/kubernetes ClusterIP 10.10.0.1 <none> 443/TCP 20m <none>
service/nginx NodePort 10.10.181.43 <none> 80:30102/TCP 119s app=nginx
访问Nginx运行节点（node002:30102）
Kubernetes基本操作
应用部署
创建一个名称为nginx-test的Deployment，同时指定应用镜像
kubectl create deployment nginx-test --image=nginx
查看所有Deployment：kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-test 1/1 1 1 18s
删除部署应用
kubectl delete deployment nginx-test
查看应用信息
查看所有Pod的状态：kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-test-84b478f9c5-vz7bw 1/1 Running 0 29s
查看Pod的详细状态:kubectl describe pods
Name: nginx-test-84b478f9c5-vz7bw
Namespace: default
Priority: 0
Node: minikube/192.168.49.2
Start Time: Tue, 22 Mar 2022 09:03:09 +0800
Labels: app=nginx-test
pod-template-hash=84b478f9c5
Annotations: <none>
Status: Running
IP: 172.17.0.3
IPs:
IP: 172.17.0.3
Controlled By: ReplicaSet/nginx-test-84b478f9c5
Containers:
nginx:
Container ID: docker://8f20af263a8c7dce564fa6d49943fbef4fe151aaaef24e3564e57e13787c7213
Image: nginx
Image ID: docker-pullable://nginx@sha256:e1211ac17b29b585ed1aee166a17fad63d344bc973bc63849d74c6452d549b3e
Port: <none>
Host Port: <none>
State: Running
Started: Tue, 22 Mar 2022 09:03:13 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2hldl (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-2hldl:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 39s default-scheduler Successfully assigned default/nginx-test-84b478f9c5-vz7bw to minikube
Normal Pulling 39s kubelet Pulling image "nginx"
Normal Pulled 36s kubelet Successfully pulled image "nginx" in 2.952573727s
Normal Created 35s kubelet Created container nginx
Normal Started 35s kubelet Started container nginx
将Pod名称设置为环境变量，方便使用$POD_NAME来应用Pod的名称
export NGINX_POD=nginx-test-84b478f9c5-vz7bw
查看Pod打印的日志：kubectl logs $NGINX_POD
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2022/03/22 01:03:13 [notice] 1#1: using the "epoll" event method
2022/03/22 01:03:13 [notice] 1#1: nginx/1.21.6
2022/03/22 01:03:13 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
2022/03/22 01:03:13 [notice] 1#1: OS: Linux 3.10.0-1160.59.1.el7.x86_64
2022/03/22 01:03:13 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2022/03/22 01:03:13 [notice] 1#1: start worker processes
2022/03/22 01:03:13 [notice] 1#1: start worker process 32
2022/03/22 01:03:13 [notice] 1#1: start worker process 33
使用exec在Pod的容器中执行命令
kubectl exec nginx-test-84b478f9c5-vz7bw -- echo hello world
进入容器内部并执行bash命令，退出容器使用exit命令
kubectl exec -it nginx-test-84b478f9c5-vz7bw -- bash
公开暴露应用
默认Pod无法被集群外部访问，需要创建Service并暴露端口才能被外部访问。
创建Service暴露nginx-test这个Deployment，通过NodePort属性得到暴露到外部的端口
kubectl expose deployment nginx-test --type=NodePort --port 80
查看所有Service的状态:kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 16h
nginx-test NodePort 10.101.176.18 <none> 80:32299/TCP 77s
查看Service的详情
kubectl describe services nginx-test
访问服务：IP：32299
Name: nginx-test
Namespace: default
Labels: app=nginx-test
Annotations: <none>
Selector: app=nginx-test
Type: NodePort
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.101.176.18
IPs: 10.101.176.18
Port: <unset> 80/TCP
TargetPort: 80/TCP
NodePort: <unset> 32299/TCP
Endpoints: 172.17.0.3:80
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
标签的使用
通过给资源添加Label，可以方便地管理资源
查看Deployment中所包含的Label：kubectl describe deployment
Name: nginx-test
Namespace: default
CreationTimestamp: Tue, 22 Mar 2022 09:03:09 +0800
Labels: app=nginx-test
Annotations: deployment.kubernetes.io/revision: 1
Selector: app=nginx-test
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=nginx-test
Containers:
nginx:
Image: nginx
Port: <none>
Host Port: <none>
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: nginx-test-84b478f9c5 (1/1 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 32m deployment-controller Scaled up replica set nginx-test-84b478f9c5 to 1
通过Label查询Pod：kubectl get pods -l app=nginx-test
NAME READY STATUS RESTARTS AGE
nginx-test-84b478f9c5-vz7bw 1/1 Running 0 34m
通过Label查询Service：kubectl get services -l app=nginx-test
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-test NodePort 10.101.176.18 <none> 80:32299/TCP 16m
给Pod添加Label：kubectl label pod nginx-test-84b478f9c5-vz7bw env_role=dev
查看Pod的详细信息：kubectl describe pods nginx-test-84b478f9c5-vz7bw
Name: nginx-test-84b478f9c5-vz7bw
Namespace: default
Priority: 0
Node: minikube/192.168.49.2
Start Time: Tue, 22 Mar 2022 09:03:09 +0800
Labels: app=nginx-test
env_role=dev
通过Label删除服务：kubectl delete service -l app=nginx-test
service "nginx-test" deleted
kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 16h
524
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
