kubenetes容器编排

1、k8s的架构

k8s架构核心组件

其他组件

2、k8s集群的安装

2.1、准备环境(主机IP、主机名、hosts解析)


10.0.0.11    k8s-master
10.0.0.12    k8s-node-1
10.0.0.13    k8s-node-2

2.2、master节点安装etcd


1)安装etcd
yum install etcd -y vim 

2)修改配置文件
 vim  /etc/etcd/etcd.conf 
6行:ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379" 
21行: ETCD_ADVERTISE_CLIENT_URLS="http://10.0.0.11:2379" 

3)启动etcd
systemctl start etcd.service 
systemctl enable etcd.service

4)插入key和值并检查
etcdctl set testdir/testkey0 0 
etcdctl get testdir/testkey0

5)检查集群健康状态 
etcdctl -C http://10.0.0.11:2379 cluster-health

2.3、master节点安装kubernetes

yum install kubernetes-master.x86_64 -y

vim /etc/kubernetes/apiserver 
8行: KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"  
11行:KUBE_API_PORT="--port=8080" 
14 KUBELET_PORT="--kubelet-port=10250" 
17行:KUBE_ETCD_SERVERS="--etcd-servers=[http://10.0.0.11:2379](http://10.0.0.11:2379)"  
23行:KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota"

vim /etc/kubernetes/config  
22行:KUBE_MASTER="--master=[http://10.0.0.11:8080](http://10.0.0.11:8080)"

systemctl enable kube-apiserver.service  
systemctl restart kube-apiserver.service  
systemctl enable kube-controller-manager.service  
systemctl restart kube-controller-manager.service  
systemctl enable kube-scheduler.service  
systemctl restart kube-scheduler.service

2.4、 node节点安装kubernetes


yum install kubernetes-node.x86_64 -y

vim /etc/kubernetes/config 
22行:KUBE_MASTER="--master=[http://10.0.0.11:8080](http://10.0.0.11:8080)"

vim /etc/kubernetes/kubelet  
5行:KUBELET_ADDRESS="--address=0.0.0.0"  
8行:KUBELET_PORT="--port=10250"  
11行:KUBELET_HOSTNAME="--hostname-override=10.0.0.12"  
14行:KUBELET_API_SERVER="--api-servers=[http://10.0.0.11:8080](http://10.0.0.11:8080)"

systemctl enable kubelet.service  
systemctl start kubelet.service  
systemctl enable kube-proxy.service  
systemctl start kube-proxy.service

在10.0.0.11上检查
kubectl get nodes

2.5、所有节点配置flannel网络


yum install flannel -y

sed -i 's#http://127.0.0.1:2379#http://10.0.0.11:2379#g' /etc/sysconfig/flanneld


master节点:  
etcdctl mk /atomic.io/network/config '{ "Network": "172.16.0.0/16" }'
yum install docker -y  
systemctl enable flanneld.service 
systemctl restart flanneld.service 
service docker restart  
systemctl restart kube-apiserver.service  
systemctl restart kube-controller-manager.service  
systemctl restart kube-scheduler.service

node节点:  
systemctl enable flanneld.service 
systemctl restart flanneld.service 
service docker restart  
systemctl restart kubelet.service  
systemctl restart kube-proxy.service


所有节点上传镜像
wget http://192.168.12.201/docker_image/docker_busybox.tar.gz
docker load -i docker_busybox.tar.gz
docker run -it docker_docker.io/busybox:latest
发现主机之间不能ping通,解决方法:
 vim /usr/lib/systemd/system/docker.service
18行添加  ExecStartPost=/usr/sbin/iptables  -P FORWARD ACCEPT
systemctl daemon-reload
systemctl restart docker
启动容器主机之间互相ping验证


2.6、配置master为镜像仓库


#所有节点
vim /etc/sysconfig/docker  
将第四行替换成下面
OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false --registry-mirror=[https://registry.docker-cn.com](https://registry.docker-cn.com) --insecure-registry=10.0.0.11:5000'

systemctl restart docker

#master节点
wget http://192.168.12.201/docker_image/registry.tar.gz
docker load -i registry.tar.gz
docker run -d -p 5000:5000 --restart=always --name registry -v /opt/myregistry:/var/lib/registry registry

2.7、node节点操作


 wget http://192.168.12.201/docker_image/pod-infrastructure-latest.tar.gz
 docker load -i pod-infrastructure-latest.tar.gz
 docker tag docker.io/tianyebj/pod-infrastructure 10.0.0.72:5000/pod-infrastructure

vim /etc/kubernetes/kubelet 
KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=10.0.0.72:5000/pod-infrastructure:latest"




3、k8s介绍

3.1、k8s的核心功能


自愈:重新启动失败的容器,在节点不可用时,替换和重新调度节点上的容器,对用户定义的健康检查不响应的容器会被终止,并且在容器准备好服务之前不会把其向客户端广播

弹性伸缩

服务的自动发现和负载均衡

滚动升级和一键回滚

3.2、k8s的历史


2014年  docker容器编排工具
2015年  发布kubernetes 1.0,加入CNCF
2016年   干掉 docker swarm, mesos
2017
2018年   k8s从CNCF毕业
2019年   1.13,  1.14,  1.15

3.3、k8s安装


yum 安装   1.5版本   最容易成功。最适合学习的

源码编译安装      难度最大,可以安装最新版

二进制安装     步骤繁琐    可以安装最新版    shell,ansible,saltstack

kubeadm    安装最容易,需要网络   可以安装最新版

minikube    适合开发人员体验k8s,网络

3.4、k8s的应用场景


k8s最适合跑微服务项目

什么是微服务?
一个功能一个集群
微服务支持更大的并发
减少发布更新的时间
降低开发的难度
集群健壮性更高

4、k8s常用的资源

4.1、创建pod资源

k8s yaml的主要组成

apiVersion: v1  api版本
kind: pod   资源类型
metadata:   属性
spec:       详细

创建pod


vim k8s_pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    app: web
spec:
  containers:
    - name: nginx
      image: 10.0.0.11:5000/nginx:1.13
      ports:
        - containerPort: 80


pod资源:至少有两个容器组成,pod基础容器和业务容器

pod是k8s最小的资源单位

4.2、ReplicationController资源


rc:保证指定数量的pod始终存活,rc通过标签选择器来关联pid


k8s资源的常见操作:
kubectl   create  -f   xxx.yaml
kubectl   get  pod|rc
kubectl  describe  pod  nginx
kubectl  delete   pod  nginx   或者kubectl delete  -f  xxx.yaml
kubectl  edit  pod   nginx

创建一个rc


apiVersion: v1
kind: ReplicationController
metadata:
  name: nginx
spec:
  replicas: 5
  selector:
    app: myweb
  template:
    metadata:
      labels:
        app: myweb
    spec:
      containers:
      - name: myweb
        image: 10.0.0.11:5000/nginx:1.13
        ports:
        - containerPort: 80




rc的滚动升级
新建一个nginx-rc1.15.yaml
![1563093237250](assets/1563093237250.png)

升级
kubectl rolling-update nginx -f nginx-rc1.15.yaml --update-period=10s

回滚
kubectl rolling-update nginx2 -f nginx-rc.yaml --update-period=1s

4.3、 service资源


service帮助pod暴露端口

创建一个service


apiVersion: v1
kind: Service
metadata:
  name: myweb
spec:
  type: NodePort
  ports:
    - port: 80
      nodePort: 30000
      targetPort: 80
  selector:
    app: myweb2

修改nodePort范围


vim  /etc/kubernetes/apiserver
KUBE_API_ARGS="--service-node-port-range=3000-50000"


service默认使用iptables来实现负载均衡,新版本中推荐使用lvs(四层负载均衡)

4.3、deployment资源


rc在滚动升级之后,会造成服务访问中断,于是引入了deployment资源

创建deployment


apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: 10.0.0.11:5000/nginx:1.13
        ports:
        - containerPort: 80
        resources:
          limits:
            cpu: 100m
          requests:
            cpu: 100m

deployment升级和回滚


命令行创建deployment
kubectl run   nginx  --image=10.0.0.11:5000/nginx:1.13 --replicas=3 --record

命令行升级版本
kubectl set image deploy nginx nginx=10.0.0.11:5000/nginx:1.15

查看deployment所有历史版本
kubectl rollout history deployment nginx

deployment回滚到上一个版本
kubectl rollout undo deployment nginx

deployment回滚到指定版本
kubectl rollout undo deployment nginx --to-revision=2


对外开放端口(随机端口)
kubectl expose deployment nginx --port=80 --type=NodePort


4.5、tomcat+mysql练习


node节点操作
1)上传mysql、tomcat镜像

2)上传镜像
docker load -i docker-mysql-5.7.tar.gz 
docker load -i tomcat-app-v2.tar.gz 

3)打标签
docker tag docker.io/mysql:5.7 10.0.0.11:5000/mysql:5.7
docker tag docker.io/kubeguide/tomcat-app:v2 10.0.0.11:5000/tomcat-app:v2



master节点操作
1)vim k8s/tomcat_demo/mysql-rc.yml 
apiVersion: v1
kind: ReplicationController
metadata:
  name: mysql
spec:
  replicas: 1
  selector:
    app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
        - name: mysql
          image: 10.0.0.11:5000/mysql:5.7
          ports:
          - containerPort: 3306
          env:
          - name: MYSQL_ROOT_PASSWORD
            value: '123456'
2)kubectl create -f mysql-rc.yml 

3)vim k8s/tomcat_demo/mysql-svc.yml 
apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  ports:
    - port: 3306
      targetPort: 3306
  selector:
    app: mysql
4)kubectl create -f mysql-svc.yml 

5)vim k8s/tomcat_demo/tomcat-rc.yml (注意:env下的value配置CLUSTER-IP,vIP可以用kubectl get all -o wide查看svc/mysql下的CLUSTER-IP地址)
apiVersion: v1
kind: ReplicationController
metadata:
  name: myweb
spec:
  replicas: 1
  selector:
    app: myweb
  template:
    metadata:
      labels:
        app: myweb
    spec:
      containers:
        - name: myweb
          image: 10.0.0.11:5000/tomcat-app:v2
          ports:
          - containerPort: 8080
          env:
          - name: MYSQL_SERVICE_HOST
            value: '10.254.43.97 '                 <----------------------------------------   
          - name: MYSQL_SERVICE_PORT
            value: '3306'
6)kubectl create -f tomcat-rc.yml

7)vim k8s/tomcat_demo/tomcat-svc.yml 
apiVersion: v1
kind: Service
metadata:
  name: myweb
spec:
  type: NodePort
  ports:
    - port: 8080
      nodePort: 30008
  selector:
    app: myweb
8)  kubectl create -f tomcat-svc.yml 

9)kubectl get all -o wide
找到tomcat8080端口对应的对外端口,网页进行访问,有数据就连接上了


网页打不开就执行下面命令
sysctl net.ipv4.ip_forward=1
iptables -P FORWARD ACCEPT



5、k8s的附加组件

5.1、dns服务


安装dns镜像
1)下载dns_docker镜像包
wget http://192.168.12.201/docker_image/docker_k8s_dns.tar.gz

2)node2节点导入镜像
docker load -i k8s_dns.tar.gz

3)创建dns服务
kubectl  create  -f   skydns-rc.yaml
kubectl  create  -f   skydns-svc.yaml

4)检查
kubectl get all --namespace=kube-system

5)修改所有node节点kubelet的配置文件

vim  /etc/kubernetes/kubelet

KUBELET_ARGS="--cluster_dns=10.254.230.254 --cluster_domain=cluster.local"

systemctl   restart kubelet

5.2、namespace命令空间



kubectl create namespace lwl

sed -i '3a  \ \ namespace: lwl' *.yml|head -5

kubectl get all --namespace lwl

5.3、健康检查

5.3.1、探针的种类


livenessProbe:健康状态检查,周期性检查服务是否存活,检查结果失败,将重启容器

readinessProbe:可用性检查,周期性检查服务是否可用,不可用将从service的endpoints中移除

5.3.2、探针的检测方法


exec:执行一段命令
httpGet:检测某个 http 请求的返回状态码
tcpSocket:测试某个端口是否能够连接

5.3.3 liveness探针的exec使用


vi  nginx_pod_exec.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: exec
spec:
  containers:
    - name: nginx
      image: 10.0.0.11:5000/nginx:1.13
      ports:
        - containerPort: 80
      args:
        - /bin/sh
        - -c
        - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600
      livenessProbe:
        exec:
          command:
            - cat
            - /tmp/healthy
        initialDelaySeconds: 5
        periodSeconds: 5


5.3.4 liveness探针的httpGet使用


vi   nginx_pod_httpGet.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: httpget
spec:
  containers:
    - name: nginx
      image: 10.0.0.11:5000/nginx:1.13
      ports:
        - containerPort: 80
      livenessProbe:
        httpGet:
          path: /index.html
          port: 80
        initialDelaySeconds: 3
        periodSeconds: 3


5.3.5 liveness探针的tcpSocket使用


vi   nginx_pod_tcpSocket.yaml
apiVersion: v1
kind: Pod
metadata:
  name: tcpSocket
spec:
  containers:
    - name: nginx
      image: 10.0.0.11:5000/nginx:1.13
      ports:
        - containerPort: 80
      livenessProbe:
        tcpSocket:
          port: 80
        initialDelaySeconds: 3
        periodSeconds: 3


5.3.6 readiness探针的httpGet使用


vi   nginx-rc-httpGet.yaml
apiVersion: v1
kind: ReplicationController
metadata:
  name: readiness
spec:
  replicas: 2
  selector:
    app: readiness
  template:
    metadata:
      labels:
        app: readiness
    spec:
      containers:
      - name: readiness
        image: 10.0.0.11:5000/nginx:1.13
        ports:
        - containerPort: 80
        readinessProbe:
          httpGet:
            path: /qiangge.html
            port: 80
          initialDelaySeconds: 3
          periodSeconds: 3

5.4、dashboard服务


1:上传并导入镜像,打标签

2:创建dashborad的deployment和service

3:访问[http://10.0.0.11:8080/ui/](http://10.0.0.11:8080/ui/)


5.5 通过apiservicer反向代理访问service


第一种:NodePort类型 
type: NodePort
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30008

第二种:ClusterIP类型
 type: ClusterIP
  ports:
    - port: 80
      targetPort: 80

6、 k8s弹性伸缩


k8s弹性伸缩,需要附加插件heapster监控

6.1 安装heapster监控



1:上传并导入镜像,打标签

ls *.tar.gz
for n in `ls *.tar.gz`;do docker load -i $n ;done
docker tag docker.io/kubernetes/heapster_grafana:v2.6.0 10.0.0.11:5000/heapster_grafana:v2.6.0
docker tag  docker.io/kubernetes/heapster_influxdb:v0.5 10.0.0.11:5000/heapster_influxdb:v0.5
docker tag docker.io/kubernetes/heapster:canary 10.0.0.11:5000/heapster:canary

2:上传配置文件,kubectl create   -f   .

3:打开dashboard验证

6.2、弹性伸缩


1:修改rc的配置文件

 containers:
 - name: myweb
 image: 10.0.0.11:5000/nginx:1.13
 ports:
 - containerPort: 80
 resources:
 limits:
 cpu: 100m
 requests:
 cpu: 100m</pre>

2:创建弹性伸缩规则

kubectl autoscale -n qiangge replicationcontroller myweb --max=8 --min=1 --cpu-percent=8

3:测试

ab -n 1000000 -c 40 http://172.16.28.6/index.html


7、持久化存储


pv: persistent volume

pvc: persistent volume  claim

7.1、安装nfs服务端(10.0.0.11)


yum install nfs-utils.x86_64 -y
mkdir /data
vim /etc/exports
/data  10.0.0.0/24(rw,async,no_root_squash,no_all_squash)
systemctl start rpcbind
systemctl start nfs


7.2、在node节点安装nfs客户端


yum install nfs-utils.x86_64 -y
showmount -e 10.0.0.11


7.3、创建pv和pvc


上传yaml配置文件,创建pv和pvc

7.4、创建mysql-rc,pod模板里使用volume


          volumeMounts:
          - name: mysql
            mountPath: /var/lib/mysql
      volumes:
      - name: mysql
        persistentVolumeClaim:
          claimName: tomcat-mysql       

7.5、 验证持久化


验证方法1:删除mysql的pod,数据库不丢

kubectl delete pod mysql-gt054

验证方法2:查看nfs服务端,是否有mysql的数据文件

7.6、 分布式存储glusterfs

7.6.1、 什么是glusterfs


Glusterfs是一个开源分布式文件系统,具有强大的横向扩展能力,可支持数PB存储容量和数千客户端,通过网络互联成一个并行的网络文件系统。具有可扩展性、高性能、高可用性等特点。

7.6.2、安装gusterfs


所有节点:
yum install  centos-release-gluster -y
yum install  install glusterfs-server -y
systemctl start glusterd.service
systemctl enable glusterd.service
mkdir -p /gfs/test1
mkdir -p /gfs/test2


fdisk -l
mkfs.xfs /dev/sdb
mkfs.xfs /dev/sdc
blkid(磁盘UUID)
vim /etc/fatsb(添加sdb和sdc的UUID。。。)
mount -a
df -h

7.6.3、添加存储资源池


master节点:
gluster pool list
gluster peer probe glusterfs02
gluster peer probe glusterfs02
gluster pool list

7.6.4、glusterfs卷管理


创建分布式复制卷
gluster volume create lwl replica 2 glusterfs01:/gfs/test1 glusterfs01:/gfs/test2 glusterfs02:/gfs/test1 glusterfs02:/gfs/test2 force

启动卷
gluster volume start lwl

查看卷
gluster volume info lwl

挂载卷
mount -t glusterfs 10.0.0.14:/lwl /mnt


7.6.5、分布式复制卷详解


pool   池
peer   节点
volume  卷
add-brick  扩容卷

7.6.6、分布式卷扩容


扩容前查看容量:
df   -h

扩容命令:
gluster volume add-brick lwl glusterfs:/gfs/test1 glusterfs:/gfs/test2 force

扩容后查看容量:
df   -h


7.7、 k8s 对接glusterfs存储

创建endpoint


vi  glusterfs-ep.yaml
aapiVersion: v1
kind: Endpoints
metadata:
  name: glusterfs
  namespace: default
subsets:
- addresses:
  - ip: 10.0.0.13
  - ip: 10.0.0.14
  - ip: 10.0.0.15
  ports:
  - port: 49152
    protocol: TCP

创建service


vi  glusterfs-svc.yaml
aapiVersion: v1
kind: Service
metadata:
  name: glusterfs
  namespace: default
spec:
  ports:
  - port: 49152
    protocol: TCP
    targetPort: 49152
  sessionAffinity: None
  type: ClusterIP



创建gluster类型pv


apiVersion: v1
kind: PersistentVolume
metadata:
  name: gluster
  labels:
    type: glusterfs
spec:
  capacity:
    storage: 50Gi
  accessModes:
    - ReadWriteMany
  glusterfs:
    endpoints: "glusterfs"
    path: "lwl"
    readOnly: false

创建pvc


kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: gluster
spec:
  selector:
    matchLabels:
      type: glusterfs
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi



使用gluster


vim mysql-rc.yml 
apiVersion: v1
kind: ReplicationController
metadata:
  name: mysql
spec:
  replicas: 1
  selector:
    app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      volumes:
        - name: mysql
          persistentVolumeClaim:
            claimName: gluster
      containers:
        - name: mysql
          image: 10.0.0.11:5000/mysql:5.7
          volumeMounts:
            - name: mysql
              mountPath: /var/lib/mysql
          ports:
          - containerPort: 3306
          env:
          - name: MYSQL_ROOT_PASSWORD
            value: '123456'



8、与jenkins集成实现ci/cd

8.1、安装gitlab并上传代码


#a:安装
wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/gitlab-ce-11.9.11-ce.0.el7.x86_64.rpm
yum localinstall gitlab-ce-11.9.11-ce.0.el7.x86_64.rpm -y
#b:配置
vim /etc/gitlab/gitlab.rb
external_url 'http://10.0.0.15'
prometheus_monitoring['enable'] = false
#c:应用并启动服务
gitlab-ctl reconfigure

#使用浏览器访问http://10.0.0.13,修改root用户密码,创建project

#上传代码到git仓库
cd /srv/
rz -E
unzip xiaoniaofeifei.zip 
rm -fr xiaoniaofeifei.zip 

git config --global user.name "Administrator"
git config --global user.email "admin@example.com"
git init
git remote add origin http://10.0.0.15/root/xiaoniao.git
git add .
git commit -m "Initial commit"
git push -u origin master

vim dockerfile
FROM 10.0.0.11:5000/nginx:1.13
ADD . /usr/share/nginx/html
docker build -t xiaoniao:v1 .
docker run -d -p 88:80 xiaoniao:v1
git add .
git commit -m "create dockerfile"
git push -u origin master

8.2、访问jenkins


访问[http://10.0.0.12:8080/](http://10.0.0.12:8080/),默认账号密码admin:123456

8.3、配置jenkins拉去gitlab代码凭据


a:在jenkins上生成秘钥对

ssh-keygen -t rsa

b:复制公钥粘贴gitlab上

c:jenkins上创建全局凭据


8.4、拉取代码测试

8.5、点击jenkins立即构建,自动构建docker镜像并上传到私有仓库


修改jenkins 工程配置
docker  build  -t  10.0.0.11:5000/test:v$BUILD_ID  .
docker  push 10.0.0.11:5000/test:v$BUILD_ID


8.6、jenkins自动部署应用到k8s


#!/bin/bash
if [ -f /tmp/xiaoniao.lock ];then
    docker  build  -t  10.0.0.11:5000/xiaoniao:v$BUILD_ID  .
    docker  push 10.0.0.11:5000/xiaoniao:v$BUILD_ID
    kubectl -s 10.0.0.11:8080 set image  -n xiaoniao deploy xiaoniao xiaoniao=10.0.0.11:5000/xiaoniao:v$BUILD_ID
    echo "更新成功"
else
    docker  build  -t  10.0.0.11:5000/xiaoniao:v$BUILD_ID  .
    docker  push 10.0.0.11:5000/xiaoniao:v$BUILD_ID
    kubectl  -s 10.0.0.11:8080  create  namespace  xiaoniao
    kubectl  -s 10.0.0.11:8080  run   xiaoniao  -n xiaoniao  --image=10.0.0.11:5000/xiaoniao:v$BUILD_ID --replicas=3 --record
    kubectl  -s 10.0.0.11:8080   expose -n xiaoniao deployment xiaoniao --port=80 --type=NodePort
    port=`kubectl -s 10.0.0.11:8080  get svc -n xiaoniao|grep -oP '(?<=80:)\d+'`
    echo "你的项目地址访问是http://10.0.0.13:$port"
    touch /tmp/xiaoniao.lock
fi



jenkins一键回滚

kubectl  -s 10.0.0.11:8080 rollout undo -n xiaoniao  deployment xiaoniao

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值