K8S - 理解volumeMounts 中的subpath

在上一篇文章中
springboot service如何动态读取外部配置文件

介绍了springboot 中如何实时读取外部配置文件的内容



部署在K8S

接下来我把它部署在k8s



首先, 我们把配置文件放入项目某个目录

这步部是必须的, 毕竟我们要引入是项目外部的文件, 这一步只是方便在不同CICD 环境下的docker 构建
我们就放在 src 外面的configs folder
在这里插入图片描述


修改docker file

# use the basic openjdk image
FROM openjdk:17

# setup working directory
WORKDIR /app

# create /app/config
RUN mkdir -p /app/config/config2

# copy configs files from project folder to /app/config
# When using COPY with more than one source file, the destination must be a directory and end with a /
COPY configs/external-config.properties /app/config/
COPY configs/external-config2.properties /app/config/config2/

# copy and rename jar file into container
COPY target/*.jar app.jar

# expose port
EXPOSE 8080

# define environment variable, and provide a default value
ENV APP_ENVIRONMENT=dev

# define the default startup command, it could be overridden in k8s
# CMD java -jar -Dserver.port=8080 -Dspring.config.name=application-${APP_ENVIRONMENT} app.jar
CMD java -jar -Dserver.port=8080 app.jar --spring.profiles.active=${APP_ENVIRONMENT}

需要创建/app/configs 并把配置文件复制到这里, 因为springboot service 会从这个path读取
注意有两个config 文件, 他们并不在同1个folder
external-config2.properties 在subfolder config2里面



构建镜像和部署镜像到GAR

这一步我们用google cloudbuild 完成
至于cloudbuild 的介绍请参考
初探 Google 云原生的CICD - CloudBuild

cloudbuild-gcr.yaml

# just to update the docker image to GAR with the pom.xml version

steps:
  - id: run maven install
    name: maven:3.9.6-sapmachine-17 # https://hub.docker.com/_/maven
    entrypoint: bash
    args:
      - '-c'
      - |
        whoami
        set -x
        pwd
        mvn install
        cat pom.xml | grep -m 1 "<version>" | sed -e 's/.*<version>\([^<]*\)<\/version>.*/\1/' > /workspace/version.txt
        echo "Version: $(cat /workspace/version.txt)"


  - id: build and push docker image
    name: 'gcr.io/cloud-builders/docker'
    entrypoint: bash
    args:
      - '-c'
      - |
        set -x
        echo "Building docker image with tag: $(cat /workspace/version.txt)"
        docker build -t $_GAR_BASE/$PROJECT_ID/$_DOCKER_REPO_NAME/${_APP_NAME}:$(cat /workspace/version.txt) .
        docker push $_GAR_BASE/$PROJECT_ID/$_DOCKER_REPO_NAME/${_APP_NAME}:$(cat /workspace/version.txt)


logsBucket: gs://jason-hsbc_cloudbuild/logs/
options: # https://cloud.google.com/cloud-build/docs/build-config#options
  logging: GCS_ONLY # or CLOUD_LOGGING_ONLY https://cloud.google.com/cloud-build/docs/build-config#logging



substitutions:
  _DOCKER_REPO_NAME: my-docker-repo
  _APP_NAME: cloud-order
  _GAR_BASE: europe-west2-docker.pkg.dev

部署完成后, 我们得到1个gar的对应image的url
europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/cloud-order:1.1.0



编写k8s yaml 和部署到k8s

apiVersion: apps/v1
kind: Deployment
metadata:
  labels: # label of this deployment
    app: cloud-order # custom defined
    author: nvd11
  name: deployment-cloud-order # name of this deployment
  namespace: default
spec:
  replicas: 3            # desired replica count, Please note that the replica Pods in a Deployment are typically distributed across multiple nodes.
  revisionHistoryLimit: 10 # The number of old ReplicaSets to retain to allow rollback
  selector: # label of the Pod that the Deployment is managing,, it's mandatory, without it , we will get this error 
            # error: error validating data: ValidationError(Deployment.spec.selector): missing required field "matchLabels" in io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector ..
    matchLabels:
      app: cloud-order
  strategy: # Strategy of upodate
    type: RollingUpdate # RollingUpdate or Recreate
    rollingUpdate:
      maxSurge: 25% # The maximum number of Pods that can be created over the desired number of Pods during the update
      maxUnavailable: 25% # The maximum number of Pods that can be unavailable during the update
  template: # Pod template
    metadata:
      labels:
        app: cloud-order # label of the Pod that the Deployment is managing. must match the selector, otherwise, will get the error Invalid value: map[string]string{"app":"bq-api-xxx"}: `selector` does not match template `labels`
    spec: # specification of the Pod
      containers:
      - image: europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/cloud-order:1.1.0 # image of the container
        imagePullPolicy: Always
        name: container-cloud-order
        command: ["bash"]
        args: 
          - "-c"
          - |
            java -jar -Dserver.port=8080 app.jar --spring.profiles.active=$APP_ENVIRONMENT
        env: # set env varaibles
        - name: APP_ENVIRONMENT # name of the environment variable
          value: prod # value of the environment variable
            
            
      restartPolicy: Always # Restart policy for all containers within the Pod
      terminationGracePeriodSeconds: 10 # The period of time in seconds given to the Pod to terminate gracefully

没什么特别

部署:

gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl apply -f deployment-cloud-order-with-subpath.yaml 
deployment.apps/deployment-cloud-order created
gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl get pods
NAME                                         READY   STATUS      RESTARTS       AGE
deployment-bq-api-service-6f6ffc7866-8djx9   1/1     Running     3 (12d ago)    17d
deployment-bq-api-service-6f6ffc7866-g4854   1/1     Running     12 (12d ago)   61d
deployment-bq-api-service-6f6ffc7866-lwxt7   1/1     Running     14 (12d ago)   63d
deployment-bq-api-service-6f6ffc7866-mxwcq   1/1     Running     11 (12d ago)   61d
deployment-cloud-order-58ddcf894d-8pjsz      1/1     Running     0              5s
deployment-cloud-order-58ddcf894d-mp4js      1/1     Running     0              5s
deployment-cloud-order-58ddcf894d-sszjd      1/1     Running     0              5s

检查容器内的配置文件:

gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl exec -it deployment-cloud-order-58ddcf894d-8pjsz -- /bin/bash
bash-4.4# pwd
/app
bash-4.4# ls config
config2  external-config.properties
bash-4.4# ls config/config2/
external-config2.properties
bash-4.4# 



测试api

gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ curl http://www.jp-gcp-vms.cloud:8085/cloud-order/actuator/info | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1842    0  1842    0     0   1827      0 --:--:--  0:00:01 --:--:--  1829
{
  "app": "Cloud Order Service",
  "appEnvProfile": "prod",
  "version": "1.1.0",
  "hostname": "deployment-cloud-order-58ddcf894d-8pjsz",
  "dbUrl": "jdbc:mysql://192.168.0.42:3306/demo_cloud_order?useUnicode=true&characterEncoding=utf-8&useSSL=false&allowPublicKeyRetrieval=true",
  "description": "This is a simple Spring Boot application to for cloud order...",
  "customConfig1": "value of config1",
  "customConfig2": "value of config2",
  }
}

注意 customConfig1 和 customConfig2 的值, 部署是成功的



For customConfig2 使用configMap

由于customConfig2 是实时更新的
我们尝试用configmap 来取代 上面external-config2.properties 的配置
注意这里的值改成 value of config2 - from k8s configmap 方便区分



构建1个external-config2 的configmap 资源对象

configmap-cloud-order-external-config2.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: configmap-cloud-order-external-config2
data:
  external.custom.config2: external.custom.config2=value of config2 - from k8s configmap

部署:

gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl apply -f configmap-cloud-order-external-config2.yaml 
configmap/configmap-cloud-order-external-config2 created
gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl describe cm configmap-cloud-order-external-config2
Name:         configmap-cloud-order-external-config2
Namespace:    default
Labels:       <none>
Annotations:  <none>

Data
====
external.custom.config2:
----
external.custom.config2=value of config2 - from k8s configmap

BinaryData
====

Events:  <none>



修改deployment yaml 引入configmap

apiVersion: apps/v1
kind: Deployment
metadata:
  labels: # label of this deployment
    app: cloud-order # custom defined
    author: nvd11
  name: deployment-cloud-order # name of this deployment
  namespace: default
spec:
  replicas: 3            # desired replica count, Please note that the replica Pods in a Deployment are typically distributed across multiple nodes.
  revisionHistoryLimit: 10 # The number of old ReplicaSets to retain to allow rollback
  selector: # label of the Pod that the Deployment is managing,, it's mandatory, without it , we will get this error 
            # error: error validating data: ValidationError(Deployment.spec.selector): missing required field "matchLabels" in io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector ..
    matchLabels:
      app: cloud-order
  strategy: # Strategy of upodate
    type: RollingUpdate # RollingUpdate or Recreate
    rollingUpdate:
      maxSurge: 25% # The maximum number of Pods that can be created over the desired number of Pods during the update
      maxUnavailable: 25% # The maximum number of Pods that can be unavailable during the update
  template: # Pod template
    metadata:
      labels:
        app: cloud-order # label of the Pod that the Deployment is managing. must match the selector, otherwise, will get the error Invalid value: map[string]string{"app":"bq-api-xxx"}: `selector` does not match template `labels`
    spec: # specification of the Pod
      containers:
      - image: europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/cloud-order:1.1.0 # image of the container
        imagePullPolicy: Always
        name: container-cloud-order
        command: ["bash"]
        args: 
          - "-c"
          - |
            java -jar -Dserver.port=8080 app.jar --spring.profiles.active=$APP_ENVIRONMENT
        env: # set env varaibles
        - name: APP_ENVIRONMENT # name of the environment variable
          value: prod # value of the environment variable
        volumeMounts:
          - name: volume-external-config2
            mountPath: /app/config/config2/
	    volumes:
	      - name: volume-external-config2
	        configMap:
	          name: configmap-cloud-order-external-config2
	          items:
	            - key: external.custom.config2
	              path: external-config2.properties # name of the file to be mounted
            
            
      restartPolicy: Always # Restart policy for all containers within the Pod
      terminationGracePeriodSeconds: 10 # The period of time in seconds given to the Pod to terminate gracefully

注意这里使用了 volume 和 volumemount



重新部署 cloud-order service

gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl delete deploy deployment-cloud-order
deployment.apps "deployment-cloud-order" deleted
gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl apply -f deployment-cloud-order-with-subpath.yaml 
deployment.apps/deployment-cloud-order created



测试api

gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ curl http://www.jp-gcp-vms.cloud:8085/cloud-order/actuator/info | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1863    0  1863    0     0   3888      0 --:--:-- --:--:-- --:--:--  3889
{
  "app": "Cloud Order Service",
  "appEnvProfile": "prod",
  "version": "1.1.0",
  "hostname": "deployment-cloud-order-69d4cd76d6-hwtfj",
  "dbUrl": "jdbc:mysql://192.168.0.42:3306/demo_cloud_order?useUnicode=true&characterEncoding=utf-8&useSSL=false&allowPublicKeyRetrieval=true",
  "description": "This is a simple Spring Boot application to for cloud order...",
  "customConfig1": "value of config1",
  "customConfig2": "value of config2 - from k8s configmap",

并没什么大问题, 证明configmap 的配置是可以覆盖docker里面定义的配置文件的



实时修改configmap 的配置

这时我们修改一下 configmap configmap-cloud-order-external-config2 里的值
由 external.custom.config2: external.custom.config2=value of config2 - from k8s configmap
改成 external.custom.config2: external.custom.config2=value of config2 - from k8s configmap updated!

apiVersion: v1
kind: ConfigMap
metadata:
  name: configmap-cloud-order-external-config2
data:
  external.custom.config2: external.custom.config2=value of config2 - from k8s configmap updated!

更新:

gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl apply -f configmap-cloud-order-external-config2.yaml 
configmap/configmap-cloud-order-external-config2 configured

等半分钟后 (1是 k8s 需要时间把 configmap的值刷新到 pods里的volumes, 2时是springboot本身需要定时器去重新获取值)
再测试api

gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ curl http://www.jp-gcp-vms.cloud:8085/cloud-order/actuator/info | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1872    0  1872    0     0   3900      0 --:--:-- --:--:-- --:--:--  3900
{
  "app": "Cloud Order Service",
  "appEnvProfile": "prod",
  "version": "1.1.0",
  "hostname": "deployment-cloud-order-69d4cd76d6-qj98f",
  "dbUrl": "jdbc:mysql://192.168.0.42:3306/demo_cloud_order?useUnicode=true&characterEncoding=utf-8&useSSL=false&allowPublicKeyRetrieval=true",
  "description": "This is a simple Spring Boot application to for cloud order...",
  "customConfig1": "value of config1",
  "customConfig2": "value of config2 - from k8s configmap updated!",

果然customConfig2的值自动更新了, 正是我们想要的
而且容器里的文件也的确更新了

gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl exec -it deployment-cloud-order-69d4cd76d6-hwtfj -- /bin/bash
bash-4.4# pwd
/app
bash-4.4# ls config
config2  external-config.properties
bash-4.4# cat config/config2/external-config2.properties 
external.custom.config2=value of config2 - from k8s configmap updated!



volumes mount 会覆盖整个文件夹(其他文件被删除)

下面来点整活
我们修改一下deployment 的yaml 配置, 把configmap的值 mark成另1个文件名

  volumeMounts:
          - name: volume-external-config2
            mountPath: /app/config/config2/
	    volumes:
	      - name: volume-external-config2
	        configMap:
	          name: configmap-cloud-order-external-config2
	          items:
	            - key: external.custom.config2
	              path: external-config2-1.properties # name of the file to be mounted

按照设想, 容器内的configmap里的
/app/config/config2/ 文件内将会有两个文件

dockerfile 定义的external-config2.properties
和 k8s 定义的 external-config2-1.properties

实际部署测试:

gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ curl http://www.jp-gcp-vms.cloud:8085/cloud-order/actuator/info | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1837    0  1837    0     0    926      0 --:--:--  0:00:01 --:--:--   926
{
  "app": "Cloud Order Service",
  "appEnvProfile": "prod",
  "version": "1.1.0",
  "hostname": "deployment-cloud-order-6878b85d44-cdvlc",
  "dbUrl": "jdbc:mysql://192.168.0.42:3306/demo_cloud_order?useUnicode=true&characterEncoding=utf-8&useSSL=false&allowPublicKeyRetrieval=true",
  "description": "This is a simple Spring Boot application to for cloud order...",
  "customConfig1": "value of config1",
  "customConfig2": "not defined",

customConfig2 居然是not defined
检查容器
发现 external-config2.properties 没了

gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl exec -it deployment-cloud-order-6878b85d44-cdvlc -- /bin/bash
bash-4.4# ls
app.jar  config
bash-4.4# ls config
config2  external-config.properties
bash-4.4# ls config/config2/
external-config2-1.properties
bash-4.4# 

原因是 当k8s 把 volume volume-external-config2 mount在 /app/config/config2 中的时候, 原来的文件就会消失



尝试把新文件mount到1个subfolder

解决方法1:
把 volume volume-external-config2 mount在 /app/config/config2/config-sub
应该可以解决

我们修改deployment yaml

  volumeMounts:
          - name: volume-external-config2
            mountPath: /app/config/config2/config-sub
	    volumes:
	      - name: volume-external-config2
	        configMap:
	          name: configmap-cloud-order-external-config2
	          items:
	            - key: external.custom.config2
	              path: external-config2-1.properties # name of the file to be mounted

mountPath 改成了/app/config/config2/config-sub

这种方法是可行的

gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl exec -it deployment-cloud-order-654974f855-cmdz7 -- /bin/bash
bash-4.4# ls
app.jar  config
bash-4.4# cd config
bash-4.4# ls
config2  external-config.properties
bash-4.4# cd config2
bash-4.4# ls
config-sub  external-config2.properties
bash-4.4# cd config-sub
bash-4.4# ls
external-config2-1.properties
bash-4.4# 

而且证明了, mountpath中的路径不存在的话, k8s会尝试创建。



利用subpath 去把文件mount在同样的folder

但是上面的做法, 能把新文件mount在1个字母中, 的确不会另旧文件消失,但是并直接真正解决问题

K8S 提供了subpath 的方法:
我们修改deployment yaml

        volumeMounts:
          - name: volume-external-config2
            mountPath: /app/config/config2/external-config2-1.properties # if we need to use subpath, need to provide the filename as well
            subPath: external-config2-1.properties # name of the file to be mounted, if we use subpath, the other files in that same folder will not dispear
      volumes:
        - name: volume-external-config2
          configMap:
            name: configmap-cloud-order-external-config2
            items:
              - key: external.custom.config2
                path: external-config2-1.properties # name of the file to be mounted                                                               

注意这里有两个改动:

  1. mountpath 上加上了文件名
  2. 添加subpath 配置, 其值还是文件名

这次的确能令两个配置文件同时存在, 原来的文件external-config2.properties 并没有被抹除

gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl exec -it deployment-cloud-order-7775b8c7cd-jnv7v -- /bin/bash
bash-4.4# pwd
/app
bash-4.4# ls
app.jar  config
bash-4.4# ls config
config2  external-config.properties
bash-4.4# ls config/config2/
external-config2-1.properties  external-config2.properties
bash-4.4# 



subpath的一些限制

k8s 设计的 subpath 其实并不是那么直接
参考:
https://hackmd.io/@maelvls/kubernetes-subpath?utm_source=preview-mode&utm_medium=rec

第1个限制就是 subpath 只能1个文件1个文件地mount, 不支持mount整个folder

参考:
https://kubernetes.io/docs/concepts/configuration/configmap/

第2个限制就是, 用subpath mount的configmap , 即使configmap的值被外部修改, 也不会同步到容器…

  • 9
    点赞
  • 16
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
### 回答1: 在 Kubernetes ,你可以通过在 Deployment 或者 StatefulSet 配置一个 "subPath" 来实现 subpath。 在 Deployment 使用 subPath 的示例配置如下: ``` apiVersion: apps/v1 kind: Deployment metadata: name: my-deployment spec: template: spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /path/to/mount subPath: my-subpath volumes: - name: my-volume configMap: name: my-configmap ``` 在 StatefulSet 使用 subPath 的示例配置如下: ``` apiVersion: apps/v1 kind: StatefulSet metadata: name: my-statefulset spec: template: spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /path/to/mount subPath: my-subpath volumes: - name: my-volume configMap: name: my-configmap ``` 其,"subPath" 应该指向 configMap 需要挂载的目录或文件的路径。 需要注意的是,这个只适用于 configMap 与 Secret 这两种 volume。 ### 回答2: K8sKubernetes)的Subpath是一种配置方式,用于将容器内的特定路径挂载到Pod的特定路径上。接下来我将用300字回答如何配置K8sSubpath。 首先,要使用Subpath,需要在Pod的配置文件的“volumes”部分定义一个Volume,并指定其类型和相关配置。例如,可以定义一个ConfigMap类型的Volume,通过将容器内的文件或目录路径挂载到Pod的某个路径上,实现文件共享和持久化。 接下来,在同一个Pod配置文件的“volumeMounts”部分,将定义的Volume挂载到容器内的特定路径上。同样需要指定路径和相关配置。例如,可以将Volume挂载到容器内的/mounted_path路径上。 最后,确保在Pod的配置文件,将定义的Volume和对应的VolumeMounts以及容器的其他相关配置相连接。 以下为示例配置: ``` apiVersion: v1 kind: Pod metadata: name: mypod spec: containers: - name: mycontainer image: myimage volumeMounts: - name: myvolume mountPath: /mounted_path readOnly: true # 是否设置只读 volumes: - name: myvolume configMap: name: myconfigmap ``` 在上述示例,定义了一个名为myvolume的Volume,类型为ConfigMap,将myconfigmap映射到Pod。然后,在容器mycontainer,将myvolume挂载到/mounted_path路径上,并设置为只读。 通过这样配置,容器内的特定路径和Pod的特定路径就能进行映射,可以实现文件的共享和持久化,便于资源的管理和访问。 总结起来,使用Subpath配置,需要在Pod的配置文件定义Volume和VolumeMounts,并确保将它们连接到容器的相关路径上。这样就能实现容器内某个路径的挂载到Pod的特定路径上,方便文件的管理和访问。 ### 回答3: 在KubernetesSubPath用于将容器的单个文件或目录挂载到容器的特定路径上。在进行SubPath配置时,需要在容器的卷挂载配置添加subPath字段,并指定宿主机的文件或目录路径。 首先,在Pod的配置文件,定义一个Volume,指定它的类型和名称。例如,可以使用emptyDir作为Volume类型,名称为"myvolume": ```yaml ... volumes: - name: myvolume emptyDir: {} ... ``` 接下来,在容器的卷挂载配置,使用上面定义的Volume,并添加subPath字段来指定容器内挂载点的路径。例如,将宿主机上的"/data/myfile.txt"文件挂载到容器内的"/app/data"目录下: ```yaml ... volumeMounts: - name: myvolume mountPath: /app/data subPath: myfile.txt ... ``` 完成上述配置后,当Pod启动时,"/data/myfile.txt"文件将会被挂载到容器内的"/app/data/myfile.txt"路径上。 需要注意的是,subPath只能用于将单个文件或目录挂载到容器,无法将多个文件或目录挂载到同一个路径上。此外,修改subPath字段的值可能会导致Pod重新启动,以确保卷的正确挂载和文件的可访问性。 总结起来,通过在Pod的Volumes定义创建一个Volume,并在容器的VolumeMounts指定subPath字段,可以配置KubernetesSubPath。这样可以将宿主机上的文件或目录挂载到容器指定的路径上。
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

nvd11

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值