Kubernetes operator系列:kubebuilder 实战演练 之 开发多版本CronJob

云原生学习路线导航页(持续更新中)

1.本项目开发的 多版本CronJob 介绍

1.1.什么情况下需要用到多版本CRD

  • 大多数项目都是从一个 alpha API 开始的,我们可以将其作为发布版本供用户使用。
  • 但增加一些重要特性后,大多数项目还是需要发布一个更稳定的 API版本。一旦 API 版本稳定,就不能对其进行重大更改。
  • 这就是 API 多版本发挥作用的地方。

1.2.本文基于前一篇开发的 CronJob:v1

1.3.两个版本的差异

  • 本文基于前一篇开发的 CronJob:v1,添加一个新的版本v2,版本的差异如下:
    • v1版本的CronJob,Spec中Schedule字段是string字符串,没有结构化
    • v2版本的CronJob,我们对Schedule字段,进行结构化,更便于使用
  • 本文仅仅是为了演示多版本的开发方法,所以v2中只对Spec进行结构化,其他的全部和v1一样

1.4.Kubernetes 版本 与 CRD 转换方法的关系

1.5.完整代码github仓库

2.CronJob:v2 开发

2.1.创建新的API:v2

  • 接下来的操作,全部基于 Kubernetes operator(九) kubebuilder 实战演练 之 自定义CronJob 得到的项目
  • 执行命令
    kubebuilder create api --group batch --version v2 --kind CronJob
    # 询问中,创建Resource回答y,创建Controller回答n
    
  • 执行命令实践,结果如下
    # 执行创建API的命令
    [root@localhost cronJob-operator]# kubebuilder create api --group batch --version v2 --kind CronJob
    INFO Create Resource [y/n]
    y
    INFO Create Controller [y/n]
    n
    INFO Writing kustomize manifests for you to edit...
    INFO Writing scaffold for you to edit...
    INFO api/v2/cronjob_types.go
    INFO api/v2/groupversion_info.go
    INFO Update dependencies:
    $ go mod tidy
    go: downloading github.com/stretchr/testify v1.8.4
    go: downloading github.com/pmezard/go-difflib v1.0.0
    go: downloading go.uber.org/goleak v1.3.0
    go: downloading github.com/evanphx/json-patch v4.12.0+incompatible
    go: downloading gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c
    go: downloading github.com/kr/pretty v0.3.1
    go: downloading github.com/rogpeppe/go-internal v1.10.0
    go: downloading github.com/kr/text v0.2.0
    INFO Running make:
    $ make generate
    /root/zgy/project/share-code-operator-study/cronJob-operator/bin/controller-gen-v0.14.0 object:headerFile="hack/boilerplate.go.txt" paths="./..."
    Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
    $ make manifests
    
    # 执行命令后,得到的项目目录如下
    [root@localhost cronJob-operator]# tree
    .
    ├── api
    │   ├── v1
    │   │   ├── cronjob_types.go
    │   │   ├── cronjob_webhook.go
    │   │   ├── cronjob_webhook_test.go
    │   │   ├── groupversion_info.go
    │   │   ├── webhook_suite_test.go
    │   │   └── zz_generated.deepcopy.go
    │   └── v2
    │       ├── cronjob_types.go
    │       ├── groupversion_info.go
    │       └── zz_generated.deepcopy.go
    ├── bin
    │   ├── controller-gen-v0.14.0
    │   └── kustomize-v5.3.0
    ├── cmd
    │   └── main.go
    ├── config
    │   ├── certmanager
    │   │   ├── certificate.yaml
    │   │   ├── kustomization.yaml
    │   │   └── kustomizeconfig.yaml
    │   ├── crd
    │   │   ├── bases
    │   │   │   └── batch.graham924.com_cronjobs.yaml
    │   │   ├── kustomization.yaml
    │   │   ├── kustomizeconfig.yaml
    │   │   └── patches
    │   │       ├── cainjection_in_cronjobs.yaml
    │   │       └── webhook_in_cronjobs.yaml
    │   ├── default
    │   │   ├── kustomization.yaml
    │   │   ├── manager_auth_proxy_patch.yaml
    │   │   ├── manager_config_patch.yaml
    │   │   ├── manager_webhook_patch.yaml
    │   │   └── webhookcainjection_patch.yaml
    │   ├── manager
    │   │   ├── kustomization.yaml
    │   │   └── manager.yaml
    │   ├── prometheus
    │   │   ├── kustomization.yaml
    │   │   └── monitor.yaml
    │   ├── rbac
    │   │   ├── auth_proxy_client_clusterrole.yaml
    │   │   ├── auth_proxy_role_binding.yaml
    │   │   ├── auth_proxy_role.yaml
    │   │   ├── auth_proxy_service.yaml
    │   │   ├── cronjob_editor_role.yaml
    │   │   ├── cronjob_viewer_role.yaml
    │   │   ├── kustomization.yaml
    │   │   ├── leader_election_role_binding.yaml
    │   │   ├── leader_election_role.yaml
    │   │   ├── role_binding.yaml
    │   │   ├── role.yaml
    │   │   └── service_account.yaml
    │   ├── samples
    │   │   ├── batch_v1_cronjob.yaml
    │   │   ├── batch_v2_cronjob.yaml
    │   │   └── kustomization.yaml
    │   └── webhook
    │       ├── kustomization.yaml
    │       ├── kustomizeconfig.yaml
    │       ├── manifests.yaml
    │       └── service.yaml
    ├── Dockerfile
    ├── go.mod
    ├── go.sum
    ├── hack
    │   └── boilerplate.go.txt
    ├── internal
    │   └── controller
    │       ├── cronjob_controller.go
    │       ├── cronjob_controller_test.go
    │       └── suite_test.go
    ├── Makefile
    ├── PROJECT
    ├── README.md
    └── test
        ├── e2e
        │   ├── e2e_suite_test.go
        │   └── e2e_test.go
        └── utils
            └── utils.go
    
    22 directories, 61 files
    

2.2.修改 api/v2/cronjob_types.go

  • 从 2.1 输出的目录可以看到,创建完 v2 版本的 API,在api 目录下多出一个 v2 目录,v2 目录下是 新版本的CronJob实体类相关资源
  • 我们修改 api/v2/cronjob_types.go,CronJobSpec 的Schedule写成结构化,其他所有的内容都和 v1版本的CronJob 一样
  • 和 v1 版本的差异处
    // represents a Cron field specifier.
    type CronField string
    
    // describes a Cron schedule.
    type CronSchedule struct {
    	// specifies the minute during which the job executes.
    	// +optional
    	Minute *CronField `json:"minute,omitempty"`
    	// specifies the hour during which the job executes.
    	// +optional
    	Hour *CronField `json:"hour,omitempty"`
    	// specifies the day of the month during which the job executes.
    	// +optional
    	DayOfMonth *CronField `json:"dayOfMonth,omitempty"`
    	// specifies the month during which the job executes.
    	// +optional
    	Month *CronField `json:"month,omitempty"`
    	// specifies the day of the week during which the job executes.
    	// +optional
    	DayOfWeek *CronField `json:"dayOfWeek,omitempty"`
    }
    
    // CronJobSpec defines the desired state of CronJob
    type CronJobSpec struct {
    	// The schedule in Cron format, see https://en.wikipedia.org/wiki/Cron.
    	Schedule CronSchedule `json:"schedule"`
    	......
    }
    
  • 完整的 api/v2/cronjob_types.go
    package v2
    
    import (
    	batchv1 "k8s.io/api/batch/v1"
    	corev1 "k8s.io/api/core/v1"
    	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    )
    
    // EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!
    // NOTE: json tags are required.  Any new fields you add must have json tags for the fields to be serialized.
    
    // ConcurrencyPolicy describes how the job will be handled.
    // Only one of the following concurrent policies may be specified.
    // If none of the following policies is specified, the default one
    // is AllowConcurrent.
    // +kubebuilder:validation:Enum=Allow;Forbid;Replace
    type ConcurrencyPolicy string
    
    const (
    	// AllowConcurrent allows CronJobs to run concurrently.
    	AllowConcurrent ConcurrencyPolicy = "Allow"
    
    	// ForbidConcurrent forbids concurrent runs, skipping next run if previous
    	ForbidConcurrent ConcurrencyPolicy = "Forbid"
    
    	// ReplaceConcurrent cancels currently running job and replaces it with a new one.
    	ReplaceConcurrent ConcurrencyPolicy = "Replace"
    )
    
    // represents a Cron field specifier.
    type CronField string
    
    // describes a Cron schedule.
    type CronSchedule struct {
    	// specifies the minute during which the job executes.
    	// +optional
    	Minute *CronField `json:"minute,omitempty"`
    	// specifies the hour during which the job executes.
    	// +optional
    	Hour *CronField `json:"hour,omitempty"`
    	// specifies the day of the month during which the job executes.
    	// +optional
    	DayOfMonth *CronField `json:"dayOfMonth,omitempty"`
    	// specifies the month during which the job executes.
    	// +optional
    	Month *CronField `json:"month,omitempty"`
    	// specifies the day of the week during which the job executes.
    	// +optional
    	DayOfWeek *CronField `json:"dayOfWeek,omitempty"`
    }
    
    // CronJobSpec defines the desired state of CronJob
    type CronJobSpec struct {
    	// The schedule in Cron format, see https://en.wikipedia.org/wiki/Cron.
    	Schedule CronSchedule `json:"schedule"`
    
    	// +kubebuilder:validation:Minimum=0
    
    	// Optional deadline in seconds for starting the job if it misses scheduled
    	// time for any reason.  Missed jobs executions will be counted as failed ones.
    	// +optional
    	StartingDeadlineSeconds *int64 `json:"startingDeadlineSeconds,omitempty"`
    
    	// Specifies how to treat concurrent executions of a Job.
    	// Valid values are:
    	// - "Allow" (default): allows CronJobs to run concurrently;
    	// - "Forbid": forbids concurrent runs, skipping next run if previous run hasn't finished yet;
    	// - "Replace": cancels currently running job and replaces it with a new one
    	// +optional
    	ConcurrencyPolicy ConcurrencyPolicy `json:"concurrencyPolicy,omitempty"`
    
    	// This flag tells the controller to suspend subsequent executions, it does
    	// not apply to already started executions.  Defaults to false.
    	// +optional
    	Suspend *bool `json:"suspend,omitempty"`
    
    	// Specifies the job that will be created when executing a CronJob.
    	JobTemplate batchv1.JobTemplateSpec `json:"jobTemplate"`
    
    	// +kubebuilder:validation:Minimum=0
    
    	// The number of successful finished jobs to retain.
    	// This is a pointer to distinguish between explicit zero and not specified.
    	// +optional
    	SuccessfulJobsHistoryLimit *int32 `json:"successfulJobsHistoryLimit,omitempty"`
    
    	// +kubebuilder:validation:Minimum=0
    
    	// The number of failed finished jobs to retain.
    	// This is a pointer to distinguish between explicit zero and not specified.
    	// +optional
    	FailedJobsHistoryLimit *int32 `json:"failedJobsHistoryLimit,omitempty"`
    }
    
    // CronJobStatus defines the observed state of CronJob
    type CronJobStatus struct {
    	// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
    	// Important: Run "make" to regenerate code after modifying this file
    
    	// A list of pointers to currently running jobs.
    	// +optional
    	Active []corev1.ObjectReference `json:"active,omitempty"`
    
    	// Information when was the last time the job was successfully scheduled.
    	// +optional
    	LastScheduleTime *metav1.Time `json:"lastScheduleTime,omitempty"`
    }
    
    //+kubebuilder:object:root=true
    //+kubebuilder:subresource:status
    
    // CronJob is the Schema for the cronjobs API
    type CronJob struct {
    	metav1.TypeMeta   `json:",inline"`
    	metav1.ObjectMeta `json:"metadata,omitempty"`
    
    	Spec   CronJobSpec   `json:"spec,omitempty"`
    	Status CronJobStatus `json:"status,omitempty"`
    }
    
    //+kubebuilder:object:root=true
    
    // CronJobList contains a list of CronJob
    type CronJobList struct {
    	metav1.TypeMeta `json:",inline"`
    	metav1.ListMeta `json:"metadata,omitempty"`
    	Items           []CronJob `json:"items"`
    }
    
    func init() {
    	SchemeBuilder.Register(&CronJob{}, &CronJobList{})
    }
    

2.3.设置etcd的存储版本

  • 当API有多个版本时,对于一个API资源,etcd不知道统一保存哪个版本的资源,需要我们指定一个 存储版本
  • 这样etcd会将该API资源,统一转成存储版本,加以存储
  • 我们决定将v1版本设置为存储版本,设置方法为:在v1版本的CronJob结构体上方,使用 +kubebuilder:storageversion 标记
  • api/v1/cronjob_types.go内容如下
    //+kubebuilder:object:root=true
    //+kubebuilder:subresource:status
    //+kubebuilder:storageversion
    
    // CronJob is the Schema for the cronjobs API
    type CronJob struct {
    	metav1.TypeMeta   `json:",inline"`
    	metav1.ObjectMeta `json:"metadata,omitempty"`
    
    	Spec   CronJobSpec   `json:"spec,omitempty"`
    	Status CronJobStatus `json:"status,omitempty"`
    }
    

2.5.编写版本间的转换方法

2.5.1.controller-runtime的Hubs、spokes概念

  • 存在多个版本的API,用户可以请求任何一个版本,所以必须定义一种可以在多个版本之间来回转换的方法

  • 版本转换 有两种解决方案

    • 两两版本间转换:每两个版本之间,就写一套转换方法
    • 中心轴条式转换(hub-spokes):定义一个中心版本,其他版本 只写 转成中心版本的方法,版本间相互转换通过中心版本做中转
      • 中心版本,称为Hub
      • 其他所有版本,称为Spokes

    在这里插入图片描述

  • 很明显,第二种 中心轴条式转换(hub-spokes) 更优异,不需要维护那么多转换方法,易扩展,controller-runtime 也是如此

2.5.2.controller-runtime 的 Hub 和 Convertible 接口

  • controller-runtimepkg/conversion 包下提供了两个接口:
    • Hub 接口
      • 具有多版本的API,要从中选择一个版本作为 中心版本
      • 中心版本需要实现 Hub 接口,相当于完成了标记
        type Hub interface {
        	runtime.Object
        	Hub()
        }
        
    • Convertible 接口
      • 具有多版本的API,每个Spokes版本,都需要 实现 Convertible 接口,实现 ConvertTo、ConvertFrom 方法,用于和Hub版本之间相互转换
        type Convertible interface {
        	runtime.Object
        	// 将 当前版本 转成 Hub中心版本
        	ConvertTo(dst Hub) error
        	// 将 Hub版本 转成 当前版本
        	ConvertFrom(src Hub) error
        }
        

2.5.3.将 CronJob:v1 版本作为Hub中心版本

  • api/v1 目录下创建一个 cronjob_conversion.go 文件,用于让 v1 版本的 CronJob 实现 Hub 接口
  • 实现 Hub 方法,空就行
  • api/v1/cronjob_conversion.go 内容
    package v1
    
    // Hub marks this type as a conversion hub.
    func (*CronJob) Hub() {}
    

2.5.4.将 CronJob:v2 版本作为Spoke轴条版本

  • api/v2 目录下创建一个 cronjob_conversion.go 文件,用于让 v2 版本的 CronJob 实现 Convertible 接口
  • 编写 ConvertTo 和 ConvertFrom 方法,用于和Hub版本之间相互转换
  • api/v2/cronjob_conversion.go 内容
    package v2
    
    import (
    	"fmt"
    	v1 "graham924.com/cronJob-operator/api/v1"
    	"sigs.k8s.io/controller-runtime/pkg/conversion"
    	"strings"
    )
    
    func (src *CronJob) ConvertTo(dstRaw conversion.Hub) error {
    	dst := dstRaw.(*v1.CronJob)
    
    	sched := src.Spec.Schedule
    	scheduleParts := []string{"*", "*", "*", "*", "*"}
    	if sched.Minute != nil {
    		scheduleParts[0] = string(*sched.Minute)
    	}
    	if sched.Hour != nil {
    		scheduleParts[1] = string(*sched.Hour)
    	}
    	if sched.DayOfMonth != nil {
    		scheduleParts[2] = string(*sched.DayOfMonth)
    	}
    	if sched.Month != nil {
    		scheduleParts[3] = string(*sched.Month)
    	}
    	if sched.DayOfWeek != nil {
    		scheduleParts[4] = string(*sched.DayOfWeek)
    	}
    	dst.Spec.Schedule = strings.Join(scheduleParts, " ")
    	/*
    		The rest of the conversion is pretty rote.
    	*/
    	// ObjectMeta
    	dst.ObjectMeta = src.ObjectMeta
    
    	// Spec
    	dst.Spec.StartingDeadlineSeconds = src.Spec.StartingDeadlineSeconds
    	dst.Spec.ConcurrencyPolicy = v1.ConcurrencyPolicy(src.Spec.ConcurrencyPolicy)
    	dst.Spec.Suspend = src.Spec.Suspend
    	dst.Spec.JobTemplate = src.Spec.JobTemplate
    	dst.Spec.SuccessfulJobsHistoryLimit = src.Spec.SuccessfulJobsHistoryLimit
    	dst.Spec.FailedJobsHistoryLimit = src.Spec.FailedJobsHistoryLimit
    
    	// Status
    	dst.Status.Active = src.Status.Active
    	dst.Status.LastScheduleTime = src.Status.LastScheduleTime
    
    	return nil
    }
    func (dst *CronJob) ConvertFrom(srcRaw conversion.Hub) error {
    	src := srcRaw.(*v1.CronJob)
    	schedParts := strings.Split(src.Spec.Schedule, " ")
    	if len(schedParts) != 5 {
    		return fmt.Errorf("invalid schedule: not a standard 5-field schedule")
    	}
    
    	partIfNeeded := func(raw string) *CronField {
    		if raw == "*" {
    			return nil
    		}
    		part := CronField(raw)
    		return &part
    	}
    
    	dst.Spec.Schedule = CronSchedule{
    		Minute:     partIfNeeded(schedParts[0]),
    		Hour:       partIfNeeded(schedParts[1]),
    		DayOfMonth: partIfNeeded(schedParts[2]),
    		Month:      partIfNeeded(schedParts[3]),
    		DayOfWeek:  partIfNeeded(schedParts[4]),
    	}
    
    	/*
    		The rest of the conversion is pretty rote.
    	*/
    	// ObjectMeta
    	dst.ObjectMeta = src.ObjectMeta
    
    	// Spec
    	dst.Spec.StartingDeadlineSeconds = src.Spec.StartingDeadlineSeconds
    	dst.Spec.ConcurrencyPolicy = ConcurrencyPolicy(src.Spec.ConcurrencyPolicy)
    	dst.Spec.Suspend = src.Spec.Suspend
    	dst.Spec.JobTemplate = src.Spec.JobTemplate
    	dst.Spec.SuccessfulJobsHistoryLimit = src.Spec.SuccessfulJobsHistoryLimit
    	dst.Spec.FailedJobsHistoryLimit = src.Spec.FailedJobsHistoryLimit
    
    	// Status
    	dst.Status.Active = src.Status.Active
    	dst.Status.LastScheduleTime = src.Status.LastScheduleTime
    
    	return nil
    }
    

2.6.多版本转换需要使用Webhook运行

2.6.1.CRD转换方法需要使用Webhook运行

  • Kubernetes CRD 的 conversion 方法通常需要使用 webhook 来实现。
  • Conversion webhook 是一种 Kubernetes API 扩展机制,它允许开发者在 CRD 对象在 API Server 中存储之前或之后对其进行版本升级、字段转换、默认值设置等处理

2.6.2.为 CronJob v2 创建webhook

  • 我们仅仅为了测试创建的是用于conversion的webhook,因此参数只使用 --conversion
    • 如果你想要为v2版本也进行 校验和设置默认值,那么就像 v1 版本那样,使用 --defaulting --programmatic-validation 参数
    • 我们这里就不测试 校验和设置默认值 了
  • 执行命令
    kubebuilder create webhook --group batch --version v2 --kind CronJob --conversion
    
  • 执行之后,项目目录如下:
    • 可以看到,api/v2 目录下生成了 cronjob_webhook.gocronjob_webhook_test.go 两个文件
    • 其中,cronjob_webhook.go 文件内容如下
    package v2
    
    import (
    	ctrl "sigs.k8s.io/controller-runtime"
    	logf "sigs.k8s.io/controller-runtime/pkg/log"
    )
    
    // log is for logging in this package.
    var cronjoblog = logf.Log.WithName("cronjob-resource")
    
    // SetupWebhookWithManager will setup the manager to manage the webhooks
    func (r *CronJob) SetupWebhookWithManager(mgr ctrl.Manager) error {
    	return ctrl.NewWebhookManagedBy(mgr).
    		For(r).
    		Complete()
    }
    
    // TODO(user): EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!
    

2.6.3.查看main方法的变化

  • 为 CronJob v2 的创建完webhook之后,main方法有一些变化
  • 我们新创建了一个webhook,自然要在main方法中调用 SetupWebhookWithManager 方法,将 这个Webhook 启动起来
  • 查看main.go中,果然多了一段 batchv2.CronJob 的 Webhook 启动的逻辑
    ......
    if os.Getenv("ENABLE_WEBHOOKS") != "false" {
    	if err = (&batchv1.CronJob{}).SetupWebhookWithManager(mgr); err != nil {
    		setupLog.Error(err, "unable to create webhook", "webhook", "CronJob")
    		os.Exit(1)
    	}
    }
    if os.Getenv("ENABLE_WEBHOOKS") != "false" {
    	if err = (&batchv2.CronJob{}).SetupWebhookWithManager(mgr); err != nil {
    		setupLog.Error(err, "unable to create webhook", "webhook", "CronJob")
    		os.Exit(1)
    	}
    }
    ......
    

3.部署和验证

3.1.部署

  • 环境配置前提:安装cert-manager,修改配置文件,请看:Kubernetes operator(九) kubebuilder 实战演练 之 自定义CronJob 的 6.2
  • 确保webhook的证书可以正确提供,再执行命令
    make manifests
    make install
    export ENABLE_WEBHOOKS=true
    make docker-build docker-push IMG=gesang321/cronjob-operator:v3
    make deploy IMG=gesang321/cronjob-operator:v3
    
  • make deploy过程中可能遇到这个错误:
    • error: resource mapping not found for name: "cronjob-operator-controller-manager-metrics-monitor" namespace: "cronjob-operator-system" from "STDIN": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1" ensure CRDs are installed first ensure CRDs are installed first
    • github上有人遇到相同的错误,可以参考一下
    • 看他的意思,好像是通过安装 kube-prometheus-stack 解决了问题,如果遇到可以试一下

3.2.验证

  • 编写测试 yaml

  • 修改 config/samples/batch_v2_cronjob.yaml

  • 内容如下

    apiVersion: batch.tutorial.kubebuilder.io/v2
    kind: CronJob
    metadata:
      labels:
        app.kubernetes.io/name: cronjob
        app.kubernetes.io/instance: cronjob-sample
        app.kubernetes.io/part-of: project
        app.kubernetes.io/managed-by: kustomize
        app.kubernetes.io/created-by: project
      name: cronjob-sample
    spec:
      schedule:
        minute: "*/1"
      startingDeadlineSeconds: 60
      concurrencyPolicy: Allow # explicitly specify, but Allow is also default.
      jobTemplate:
        spec:
          template:
            spec:
              containers:
              - name: hello
                image: busybox
                args:
                - /bin/sh
                - -c
                - date; echo Hello from the Kubernetes cluster
              restartPolicy: OnFailure
    
  • 创建 batch.tutorial.kubebuilder.io/v2 的 CronJob

    kubectl apply -f config/samples/batch_v2_cronjob.yaml
    
  • 验证转换逻辑可以生效

    • 下面两个命令,都能获取到我们的资源
    • 当获取v1版本资源的时候,apiserver会自动调用我们运行的 conversion webhook,进行 v2->Hub->v1 的转换,进而获取到 v1 版本的资源
    • 并且获取到的v1版本的Schedule,是 "*/1 * * * *",与我们编写的v2版本的yaml,是等价的
    kubectl get cronjobs.v2.batch.graham924.com
    kubectl get cronjobs.v1.batch.graham924.com
    
  • 21
    点赞
  • 22
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值