实验 详解K8S多节点部署群集

一 环境部署

主机名IP地址部署的服务
master0120.0.0.101apiserver、scheduler、controller-manager、etcd
master0220.0.0.104apiserver、scheduler、controller-manager、etcd
node0120.0.0.102kubelet、kube-proxy、docker、flannel、etcd
node0220.0.0.103kubelet、kube-proxy、docker、flannel、etcd
lb0120.0.0.105nginx、keepalived
lb0220.0.0.106nginx、keepalived
VIP20.0.0.100

二 实验操作步骤

在单节点实验基础上操作(详情见博客)

2.1 master02部署

##在master01上操作##
1. 复制kubernetes目录到master02
[root@localhost k8s]# scp -r /opt/kubernetes/ root@20.0.0.104:/opt/
The authenticity of host '20.0.0.104 (20.0.0.104)' can't be established.
ECDSA key fingerprint is SHA256:EM5og8Q/El6tYfMXxgXbd5ccgh2knP0HkFALiBNwd30.
ECDSA key fingerprint is MD5:4d:aa:26:2b:3c:c2:bd:12:92:1e:57:60:6d:72:26:1f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '20.0.0.104' (ECDSA) to the list of known hosts.
root@20.0.0.104's password: 
token.csv                                                              100%   84   140.6KB/s   00:00    
kube-apiserver                                                         100%  914   889.0KB/s   00:00    
kube-scheduler                                                         100%   85   118.4KB/s   00:00    
kube-controller-manager                                                100%  483   849.6KB/s   00:00    
kube-apiserver                                                         100%  184MB 126.8MB/s   00:01    
kubectl                                                                100%   55MB 114.9MB/s   00:00    
kube-controller-manager                                                100%  155MB 135.0MB/s   00:01    
kube-scheduler                                                         100%   55MB 135.0MB/s   00:00    
ca-key.pem                                                             100% 1675     1.2MB/s   00:00    
ca.pem                                                                 100% 1359   624.2KB/s   00:00    
server-key.pem                                                         100% 1675     1.9MB/s   00:00    
server.pem                                                             100% 1643     1.8MB/s   00:00 

2. 复制master01中的三个组件启动脚本kube-apiserver.service kube-controller-maneger.service kube-scheduler.service
[root@localhost k8s]# scp /usr/lib/systemd/system/{kube-apiserver,kube-controller-manager,kube-scheduler}.service root@20.0.0.104:/usr/lib/systemd/system
root@20.0.0.104's password: 
kube-apiserver.service                                                 100%  282   245.9KB/s   00:00    
kube-controller-manager.service                                        100%  317   286.7KB/s   00:00    
kube-scheduler.service                                                 100%  281   378.9KB/s   00:00   


##在master02上操作##
1. 查看复制的文件
[root@localhost ~]# ls /opt/kubernetes/bin/
kube-apiserver  kube-controller-manager  kubectl  kube-scheduler
[root@localhost ~]# ls /opt/kubernetes/cfg/
kube-apiserver  kube-controller-manager  kube-scheduler  token.csv
[root@localhost ~]# ls /opt/kubernetes/ssl/
ca-key.pem  ca.pem  server-key.pem  server.pem
[root@localhost ~]# ls /usr/lib/systemd/system | grep "kube"
kube-apiserver.service
kube-controller-manager.service
kube-scheduler.service



2. 修改配置文件
[root@localhost ~]# cd /opt/kubernetes/cfg/
[root@localhost cfg]# vim kube-apiserver 
KUBE_APISERVER_OPTS="--logtostderr=true \
--v=4 \
--etcd-servers=https://20.0.0.101:2379,https://20.0.0.102:2379,https://20.0.0.103:2379 \
--bind-address=20.0.0.104 \		##改为本地IP
--secure-port=6443 \
--advertise-address=20.0.0.104 \	####改为本地IP
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--kubelet-https=true \
--enable-bootstrap-token-auth \
--token-auth-file=/opt/kubernetes/cfg/token.csv \
--service-node-port-range=30000-50000 \
--tls-cert-file=/opt/kubernetes/ssl/server.pem  \
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/etcd/ssl/ca.pem \
--etcd-certfile=/opt/etcd/ssl/server.pem \
--etcd-keyfile=/opt/etcd/ssl/server-key.pem"

##在master01上操作##
1. 拷贝master01上已有的etcd证书给master02使用
//特别注意:master02一定要有etcd证书,否则apiserver服务无法启动
[root@localhost k8s]# scp -r /opt/etcd/ root@20.0.0.104:/opt/
root@20.0.0.104's password: 
etcd                                                                   100%  488   297.5KB/s   00:00    
etcd                                                                   100%   18MB 101.2MB/s   00:00    
etcdctl                                                                100%   15MB 119.9MB/s   00:00    
ca-key.pem                                                             100% 1679     1.5MB/s   00:00    
ca.pem                                                                 100% 1265   607.2KB/s   00:00    
server-key.pem                                                         100% 1675     1.9MB/s   00:00    
server.pem                                                             100% 1338   643.5KB/s   00:00



##在master02上操作##
1. 启动master三个组件服务
[root@localhost cfg]# systemctl start kube-apiserver.service 
[root@localhost cfg]# systemctl enable kube-apiserver.service
[root@localhost cfg]# systemctl start kube-controller-manager.service 
[root@localhost cfg]# systemctl enable kube-controller-manager.service
[root@localhost cfg]# systemctl start kube-scheduler.service
[root@localhost cfg]# systemctl enable kube-scheduler.service 

[root@localhost cfg]# systemctl status kube-apiserver.service 
[root@localhost cfg]# systemctl status kube-controller-manager.service 
[root@localhost cfg]# systemctl status kube-scheduler.service 

2. 增加环境变量
[root@localhost cfg]# vim /etc/profile
export PATH=$PATH:/opt/kubernetes/bin/		##末尾添加
[root@localhost cfg]# source /etc/profile

3. 查看nodes节点状态信息
[root@localhost cfg]# kubectl get nodes
NAME         STATUS   ROLES    AGE   VERSION
20.0.0.102   Ready    <none>   55m   v1.12.3
20.0.0.103   Ready    <none>   48m   v1.12.3

2.2 nginx负载均衡集群部署

##在lb01,lb02上操作##
以下以lb01为例
1. 设置主机名,关闭防火墙,核心防护
[root@localhost ~]# hostnamectl set-hostname lb01
[root@localhost ~]# su
[root@lb01 ~]# iptables -F
[root@lb01 ~]# setenforce 0
setenforce: SELinux is disabled

2. 把nginx.sh和keepalived.conf脚本拷贝到家目录
[root@lb01 ~]# rz -E
rz waiting to receive.
[root@lb01 ~]# ls
anaconda-ks.cfg       keepalived.conf  公共  视频  文档  音乐
initial-setup-ks.cfg  nginx.sh         模板  图片  下载  桌面

3. 编辑nginx线网源地址,下载安装nginx
[root@lb01 ~]# vim /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
[root@lb01 ~]# yum -y install nginx

4. 添加四层转发,开启nginx服务
[root@lb01 ~]# vim /etc/nginx/nginx.conf 
events {
    worker_connections  1024;
}

stream {					##添加四层转发

   log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
    access_log  /var/log/nginx/k8s-access.log  main;

    upstream k8s-apiserver {
        server 20.0.0.101:6443;
        server 20.0.0.104:6443;
    }
    server {
                listen 6443;
                proxy_pass k8s-apiserver;
    }
    }
[root@lb01 ~]# systemctl start nginx

5. 安装keepalived
[root@lb01 ~]# yum -y install keepalived
[root@lb01 ~]# cp keepalived.conf /etc/keepalived/keepalived.conf 
cp:是否覆盖"/etc/keepalived/keepalived.conf"yes
[root@lb01 ~]# vim /etc/keepalived/keepalived.conf 
! Configuration File for keepalived

global_defs {
   # 接收邮件地址
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   } 
   # 邮件发送地址
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1 
   smtp_connect_timeout 30
   router_id NGINX_MASTER 
}  

vrrp_script check_nginx {				##check_nginx名称可以修改
    script "/etc/nginx/check_nginx.sh"

vrrp_instance VI_1 {
    state MASTER 
    interface ens33					##改为ens33
    virtual_router_id 51				##VRRP 路由 ID实例,每个实例是唯一的 
    priority 100					##优先级,备服务器设置 90 
    advert_int 1    					##指定VRRP 心跳包通告间隔时间,默认1秒 
    authentication {
        auth_type PASS
        auth_pass 1111
    }   
    virtual_ipaddress {
        20.0.0.100/24 					##vip地址
    }   
    track_script {
        check_nginx					##与上面的名称一致
    }   
}   

-----------------------lb02配置-------------------
! Configuration File for keepalived

global_defs {
   # 接收邮件地址
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   # 邮件发送地址
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id NGINX_MASTER
}

vrrp_script check_nginx {
     script "/etc/nginx/check_nginx.sh"
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        20.0.0.100/24
    }
    track_script {
        check_nginx
    }
}


6. 编辑check_nginx.sh脚本
[root@lb01 ~]# vim /etc/nginx/check_nginx.sh
count=$(ps -ef | grep nginx | egrep -cv "grep|$$")

if [ "$count" -eq 0 ];then
    systemctl stop keepalived
fi
[root@lb01 ~]# chmod +x /etc/nginx/check_nginx.sh 
[root@lb01 ~]# systemctl start keepalived.service 

7. 查看lb01,lb02地址信息
[root@lb01 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:bf:f4:82 brd ff:ff:ff:ff:ff:ff
    inet 20.0.0.105/24 brd 20.0.0.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 20.0.0.100/24 scope global secondary ens33				##vip地址
       valid_lft forever preferred_lft forever
    inet6 fe80::324f:a4dd:6fe:7a09/64 scope link 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
    link/ether 52:54:00:7b:a1:c1 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000
    link/ether 52:54:00:7b:a1:c1 brd ff:ff:ff:ff:ff:ff

--------------------------lb02----------------------------------
[root@lb02 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:83:e9:37 brd ff:ff:ff:ff:ff:ff
    inet 20.0.0.106/24 brd 20.0.0.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::bff5:d805:be6c:19e4/64 scope link 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
    link/ether 52:54:00:c1:8b:92 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000
    link/ether 52:54:00:c1:8b:92 brd ff:ff:ff:ff:ff:ff


8. 验证地址漂移
##在lb01上操作##
1. 杀死nginx进程
[root@lb01 ~]# pkill nginx

##在lb02上操作##
1. 查看vip地址漂移
[root@lb02 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:83:e9:37 brd ff:ff:ff:ff:ff:ff
    inet 20.0.0.106/24 brd 20.0.0.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 20.0.0.100/24 scope global secondary ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::bff5:d805:be6c:19e4/64 scope link 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
    link/ether 52:54:00:c1:8b:92 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000
    link/ether 52:54:00:c1:8b:92 brd ff:ff:ff:ff:ff:ff


9. 恢复lb01为master
##在lb01上操作##
1. 先启动nginx,再启动keepalived
[root@lb01 ~]# systemctl start nginx
[root@lb01 ~]# systemctl start keepalived.service

2. 查看vip地址漂移
[root@lb01 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:bf:f4:82 brd ff:ff:ff:ff:ff:ff
    inet 20.0.0.105/24 brd 20.0.0.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 20.0.0.100/24 scope global secondary ens33				##恢复成功
       valid_lft forever preferred_lft forever
    inet6 fe80::324f:a4dd:6fe:7a09/64 scope link 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
    link/ether 52:54:00:7b:a1:c1 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000
    link/ether 52:54:00:7b:a1:c1 brd ff:ff:ff:ff:ff:ff

##在lb02上操作##
1. 查看vip是否漂移
[root@lb02 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:83:e9:37 brd ff:ff:ff:ff:ff:ff
    inet 20.0.0.106/24 brd 20.0.0.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::bff5:d805:be6c:19e4/64 scope link 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
    link/ether 52:54:00:c1:8b:92 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000
    link/ether 52:54:00:c1:8b:92 brd ff:ff:ff:ff:ff:ff



10. 修改node节点配置文件,统一VIP(bootstrap.kubeconfig,kubelet.kubeconfig)
##以下以node01为例
1. 修改为VIP地址
[root@node01 ~]# cd /opt/kubernetes/cfg/
[root@node01 cfg]# ls
bootstrap.kubeconfig  kubelet         kubelet.kubeconfig  kube-proxy.kubeconfig
flanneld              kubelet.config  kube-proxy
[root@node01 cfg]# vim bootstrap.kubeconfig
[root@node01 cfg]# vim kubelet.kubeconfig 
[root@node01 cfg]# vim kube-proxy.kubeconfig
##统统修改为VIP
server: https://20.0.0.100:6443

2. 自检
[root@node01 cfg]# grep 100 *
bootstrap.kubeconfig:    server: https://20.0.0.100:6443
kubelet.kubeconfig:    server: https://20.0.0.100:6443
kube-proxy.kubeconfig:    server: https://20.0.0.100:6443

3. 在lb01上查看nginx的k8s日志
使用lb虚拟机自带的火狐浏览器访问20.0.0.106-----------welcome nginx!

[root@lb01 ~]# tail /var/log/nginx/k8s-access.log 
20.0.0.1 20.0.0.101:6443 - [24/Nov/2020:17:39:14 +0800] 200 170
20.0.0.1 20.0.0.104:6443 - [24/Nov/2020:17:39:14 +0800] 200 170
20.0.0.1 20.0.0.101:6443 - [24/Nov/2020:17:39:14 +0800] 200 170
20.0.0.1 20.0.0.104:6443 - [24/Nov/2020:17:39:14 +0800] 200 170


##在master01上操作##
1. 测试创建pod
[root@master01 ~]# kubectl run nginx --image=nginx
kubectl run --generator=deployment/apps.v1beta1 is DEPRECATED and will be removed in a future version. Use kubectl create instead.
deployment.apps/nginx created

2. 查看状态
[root@master01 ~]# kubectl get pods
NAME                    READY   STATUS              RESTARTS   AGE
nginx-dbddb74b8-jj54s   0/1     ContainerCreating   0          24s		##正在创建中
[root@master01 ~]# kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE
nginx-dbddb74b8-jj54s   1/1     Running   0          79s			##创建完成,运行中

##注意日志问题
[root@master01 ~]# kubectl logs nginx-dbddb74b8-jj54s
Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy) ( pods/log nginx-dbddb74b8-jj54s)

解决方法:
[root@master01 ~]# kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous
clusterrolebinding.rbac.authorization.k8s.io/cluster-system-anonymous created

[root@master01 ~]# kubectl logs nginx-dbddb74b8-jj54s		##成功查看日志
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up


3. 查看网络
[root@master01 ~]# kubectl get pods -o wide
NAME                    READY   STATUS    RESTARTS   AGE     IP            NODE         NOMINATED NODE
nginx-dbddb74b8-jj54s   1/1     Running   0          5m58s   172.17.86.2   20.0.0.102   <none>

4. 在对应网段的node节点上操作直接访问
打开20.0.0.102上的火狐浏览器:172.17.86.2------welcome nginx!

5. 回到master01上查看日志
[root@master01 ~]# kubectl logs nginx-dbddb74b8-jj54s
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
172.17.86.1 - - [24/Nov/2020:10:36:49 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0" "-"
  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值