golang-进程崩溃后如何输出错误日志?

最近更新

最后更新时间:2022-08-09

勘误

最近抽空看了下以前写的文章,发现有一些误导,主要如下:

  1. 对go panic机制理解不够。误以为和C++进程一样,会收到操作系统发送的信号(内存错误),这样在进程退出前,可以记录下退出时间。
  2. 后台启动进程,不要自己开发 daemon ,直接使用 nohup 命令后台启动即可。
  3. 使用 go 关键字创建routine,但是在里面没有使用 recover() 恢复,这是直接导致进程崩溃的最主要原因。

下面说一下如何正确的理解go中的异常处理。

Go中异常处理

原则1:main协程不需要recover

错误用法:对main进行recover

func main() {
	defer func() {
		recover()
		fmt.Println("recover")
	}()

	fmt.Println("join party")
	panic("break")

	fmt.Println("continues")
	time.Sleep(time.Second * 5)
}

输出:

join party
recover

上面这个示例,其实没有什么实际意义,程序在启动阶段,且启动失败时,是允许崩溃的。如果加上recover() 没处理返回值,把异常和堆栈吃掉了,反而影响问题的排查。

正确用法:

func main() {
	fmt.Println("join party")
    
    // mock
	panic("config miss server.ip filed")

	fmt.Println("continues")
	time.Sleep(time.Second * 5)
}

输出:

join party
panic: config miss server.ip filed

goroutine 1 [running]:
main.main()
        /Users/fei.xu/repo/github/ToBeTopGo-Gitbook/code/go/04_panic_and_crash/main.go:10 +0x70
exit status 2

此时,能快速找到错误原因。

陷进1:错误信息被猫吃了

在原则1的代码中,我们是这样来处理错误的:

错误的写法:

defer func() {
	recover();
	fmt.Println("recover")
}()

这个是新人很可能会犯错的地方,如果你这样写,你上线后,报了一堆 recover ,就是没有打印具体的错误原因和堆栈,你可能会骂人。

正确的写法:

	defer func() {
		if err := recover(); err != nil {
			fmt.Println("panic error:", err)
		}
		fmt.Println("recover")
	}()

输出:

join party
panic: config miss server.ip filed
recover

陷进2:堆栈被猫吃了

你以为上面就万事大吉了?看看解决了陷阱1之后的输出:

join party
panic error: config miss server.ip filed
recover

发现什么了吗?错误堆栈呢?被猫吃了吗?

那要怎么办?

我们打印下堆栈即可(参考:https://stackoverflow.com/questions/52103182/how-to-get-the-stacktrace-of-a-panic-and-store-as-a-variable):

defer func() {
	if err := recover(); err != nil {
		fmt.Println("panic error:", err)
		fmt.Println("stacktrace from panic:" + string(debug.Stack()))
	}
	fmt.Println("recover")
}()

输出:

join party
panic error: config miss server.ip filed
stacktrace from panic:goroutine 1 [running]:
runtime/debug.Stack()
        /opt/homebrew/Cellar/go/1.18/libexec/src/runtime/debug/stack.go:24 +0x68
main.main.func1()
        /Users/fei.xu/repo/github/ToBeTopGo-Gitbook/code/go/04_panic_and_crash/main.go:14 +0x7c
panic({0x1024d3680, 0x1024e3b70})
        /opt/homebrew/Cellar/go/1.18/libexec/src/runtime/panic.go:838 +0x204
main.main()
        /Users/fei.xu/repo/github/ToBeTopGo-Gitbook/code/go/04_panic_and_crash/main.go:20 +0x8c

recover

当然,这里的堆栈层级多了三层,需要处理下。

原则2:不要轻易使用原生go关键字,启动routine

如果你是新手,你以为在main.go中写一个 recover,就万事大吉了:

func main() {
	// func main() {
	defer func() {
		recover()
		fmt.Println("recover")
	}()

    // 你的业务逻辑,在某个地方,你可以启动了一个协程来异步处理一些事情 
	// 错误:直接使用go关键字创建routine,如果出现异常,整个程序崩溃
	go func() {
		time.Sleep(time.Second * 1)
		// a dengerous action
		panic("throw a error")
	}()

	fmt.Println("server start success")
	time.Sleep(time.Second * 10)
}

可惜,事与愿违,又崩溃了:

server start success
panic: throw a error

goroutine 18 [running]:
main.main.func2()
        /Users/fei.xu/repo/github/ToBeTopGo-Gitbook/code/go/04_panic_and_crash/main.go:44 +0x3c
created by main.main
        /Users/fei.xu/repo/github/ToBeTopGo-Gitbook/code/go/04_panic_and_crash/main.go:34 +0x44
exit status 2

问题出在哪里?

func main() {
	// 捕获 main 所在 go routine
	// defer func() {
	// 	recover()
	// 	fmt.Println("recover")
	// }()

	// 错误:直接使用go关键字创建routine,如果出现异常,整个程序崩溃
	go func() {
		time.Sleep(time.Second * 1)

		// 正确的做法是:为每个routine都执行 recover() 以捕获和恢复
		defer func() {
			recover()
			fmt.Println("recover")
		}()

		// a dengerous action
		panic("throw a error")
	}()

	fmt.Println("server start success")
	time.Sleep(time.Second * 10)
}

执行:

server start success
recover
^Csignal: interrupt

最佳实践

在研究 go-zero 框架的过程中,我发现他封住了安全的go关键字,以应对新开go routine 但是忘记写recover的场景,从此,再也不用担心程序崩溃了!:

routines.go

package threading

import (
	"bytes"
	"runtime"
	"strconv"

	"github.com/zeromicro/go-zero/core/rescue"
)

// GoSafe runs the given fn using another goroutine, recovers if fn panics.
func GoSafe(fn func()) {
	go RunSafe(fn)
}

// RoutineId is only for debug, never use it in production.
func RoutineId() uint64 {
	b := make([]byte, 64)
	b = b[:runtime.Stack(b, false)]
	b = bytes.TrimPrefix(b, []byte("goroutine "))
	b = b[:bytes.IndexByte(b, ' ')]
	// if error, just return 0
	n, _ := strconv.ParseUint(string(b), 10, 64)

	return n
}

// RunSafe runs the given fn, recovers if fn panics.
func RunSafe(fn func()) {
	defer rescue.Recover()

	fn()
}

recover.go:

package rescue

import "github.com/zeromicro/go-zero/core/logx"

// Recover is used with defer to do cleanup on panics.
// Use it like:
//  defer Recover(func() {})
func Recover(cleanups ...func()) {
	for _, cleanup := range cleanups {
		cleanup()
	}

	if p := recover(); p != nil {
		logx.ErrorStack(p)
	}
}

使用示例:

func main(){
   GoSafe(func(){
       // your logic
   })
}

可以按需抽取到自己项目中使用,别忘记 logx.ErrorStack(p) 改成自己的log输出,以把错误信息和堆栈写入到日志文件中!


原文开始

——————————————————————————————
2020-04-22 日更新,增加damon进程启用core dump示例代码
——————————————————————————————

在之前文章中:https://blog.csdn.net/xmcy001122/article/details/103743249 使用golang实现了优雅退出的功能。

近期程序线上运行出现了进程崩溃的问题,于是度娘加入了 defer 代码以便打印错误堆栈,运行半天后,发现进程不见了,但是log里面却没有任何错误。于是反复检测,终于找到问题所在。

出问题的代码:

func main(){
     // ... 业务逻辑
     
     // 崩溃的时候,这段代码没有执行
	defer func() {
		// 如果是崩溃导致的信号,打印错误
		err := recover()
		if err != nil {
			logger.Sugar.Error(err)
		}
	}()

     // 优雅退出
	c := make(chan os.Signal)
	signal.Notify(c, syscall.SIGHUP, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT)
	waitExit(c)
}

func waitExit(c chan os.Signal) {
	for i := range c {
		switch i {
		case syscall.SIGHUP, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT:
			_ = logger.Sugar.Sync()
			logger.Sugar.Info("exit...")
			
			os.Exit(1)
		}
	}
}

运行的时候,模拟崩溃,goland IDE里面输出了错误,但是log里面依然不见错误输出。

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x13a79af]

goroutine 36 [running]:
robot_server/internal.(*RouteServerConn).onHandleMsgJoinNotify(0xc0001640d0, 0xc00000e700, 0xc0001a6010, 0xc, 0x27f0)
        /Users/xuyc/repo/zhaogang.com/go/src/robot_server/internal/route_server_conn.go:92 +0x15f
robot_server/internal.(*RouteServerConn).onHandleData(0xc0001640d0, 0xc00000e700, 0xc0001a6010, 0xc, 0x27f0)
        /Users/xuyc/repo/zhaogang.com/go/src/robot_server/internal/route_server_conn.go:65 +0x148
robot_server/base.(*ImClient).NetLoop(0xc0000b4180, 0xc0001ea000, 0x149ab3b)
        /Users/xuyc/repo/zhaogang.com/go/src/robot_server/base/im_client.go:168 +0x1f6
robot_server/internal.read(0xc0001640d0, 0xc00016c0f0, 0xb, 0xc0001a200a)
        /Users/xuyc/repo/zhaogang.com/go/src/robot_server/internal/route_server_conn.go:41 +0x3f1
created by robot_server/internal.(*RouteServerConn).Start
        /Users/xuyc/repo/zhaogang.com/go/src/robot_server/internal/route_server_conn.go:26 +0x5d

Process finished with exit code 2

解决方案

尝试了多种方法,最后使用了系统生成core dump文件的方案。

启用core dump

1.创建coredump.sh,并写入如下内容:

#!/bin/bash

# Filename: coredumpshell.sh
# Description: enable coredump and format the name of core file on centos system

# enable coredump whith unlimited file-size for all users
echo -e "\n# enable coredump whith unlimited file-size for all users\n* soft core unlimited" >> /etc/security/limits.conf

# set the path of core file with permission 777
cd /data && mkdir imcorefile && chmod 777 imcorefile

# format the name of core file.   
# %% – 符号%
# %p – 进程号
# %u – 进程用户id
# %g – 进程用户组id
# %s – 生成core文件时收到的信号
# %t – 生成core文件的时间戳(seconds since 0:00h, 1 Jan 1970)
# %h – 主机名
# %e – 程序文件名    
# for centos7 system(update 2017.4.2 21:44)
echo -e "\nkernel.core_pattern=/data/imcorefile/core-%e-%s-%u-%g-%p-%t" >> /etc/sysctl.conf
echo -e "\nkernel.core_uses_pid = 1" >> /etc/sysctl.conf

sysctl -p /etc/sysctl.conf

2.永久启用core dump功能

chmod 777 coredump.sh
./coredump.sh
# 重新打开终端
ulimit -a 

输出以下代表成功:
在这里插入图片描述
3.验证(不建议省略)

vim test.c 

输入如下内容:

#include <stdio.h>
int main( int argc, char * argv[] ) { char a[1]; scanf( "%s", a ); return 0; }
gcc test.c -o test  # 编译
./test              # 执行test,然后任意输入一串字符后按回车,如zhaogang.com

ls /data/corefile   # 在此目录下如果生成了相应的core文件core-test-*,代表成功

运行

按照上述设置后,在进程前加上 env GOTRACEBACK=crash

env GOTRACEBACK=crash ./robot_server

如果进程崩溃,将在 /data/imcorefile 生成core文件,比如:

[root@10-0-59-229 imcorefile]# ls
core-robot_server-6-0-0-25365-1587471512

PS:关于 GOTRACEBACK 可以参考 Go – 通过GOTRACEBACK生成程序崩溃后core文件的方法(gcore gdb)

调试

gdb

gdb robot_server ../imcorefile/core-robot_server-6-0-0-25365-1587471512

输出:

(gdb) bt  # bt查看错误堆栈
#0  runtime.raise () at /usr/local/go/src/runtime/sys_linux_amd64.s:150
#1  0x00000000004405bb in runtime.dieFromSignal (sig=6) at /usr/local/go/src/runtime/signal_unix.go:424
#2  0x0000000000440b3d in runtime.sigfwdgo (sig=6, info=0xc00015fd70, ctx=0xc00015fc40, ~r3=<optimized out>)
    at /usr/local/go/src/runtime/signal_unix.go:629
#3  0x000000000043fc60 in runtime.sigtrampgo (sig=<optimized out>, info=0xc00015fd70, ctx=0xc00015fc40)
    at /usr/local/go/src/runtime/signal_unix.go:289
#4  0x0000000000459d43 in runtime.sigtramp () at /usr/local/go/src/runtime/sys_linux_amd64.s:357
#5  0x0000000000459e30 in ?? () at /usr/local/go/src/runtime/sys_linux_amd64.s:441
#6  0x0000000000000000 in ?? ()
(gdb) eixt
Undefined command: "eixt".  Try "help".
(gdb)
Undefined command: "eixt".  Try "help".
(gdb) quit # 退出

结论:虽然生成了core文件,但是看不到具体的错误,所以需要使用delve这个工具了。

Delve

1.安装

go get -u github.com/derekparker/delve/cmd/dlv

go env # 查看 GOPATH="/home/go"

[root@10-0-59-229 robot-server.2020-04-14]# cd /home/go/
[root@10-0-59-229 go]# ls
bin  src
[root@10-0-59-229 go]# cd bin
[root@10-0-59-229 bin]# ls
dlv
[root@10-0-59-229 bin]# cp dlv /usr/bin/   # 这样就可以直接使用dlv命令了
cp:是否覆盖"/usr/bin/dlv"? y
[root@10-0-59-229 bin]# dlv # 使用dlv命令

2.调试
格式:dlv core < executable> < core> [flags]

[root@10-0-59-231 robot-server.2020-04-22]# dlv core robot_server /data/imcorefile/core-robot_server-6-0-0-32402-1587555657
Type 'help' for list of commands.
(dlv)
(dlv)
(dlv) bt # 先使用bt显示错误堆栈 
 0  0x0000000000459be1 in runtime.raise
    at /usr/local/go/src/runtime/sys_linux_amd64.s:150
 1  0x000000000044074b in runtime.dieFromSignal
    at /usr/local/go/src/runtime/signal_unix.go:424
 2  0x0000000000440ccd in runtime.sigfwdgo
    at /usr/local/go/src/runtime/signal_unix.go:629
 3  0x000000000043fdf0 in runtime.sigtrampgo
    at /usr/local/go/src/runtime/signal_unix.go:289
 4  0x0000000000459ed3 in runtime.sigtramp
    at /usr/local/go/src/runtime/sys_linux_amd64.s:357
 5  0x0000000000459fc0 in runtime.sigreturn
    at /usr/local/go/src/runtime/sys_linux_amd64.s:449
 6  0x00000000004408ea in runtime.crash
    at /usr/local/go/src/runtime/signal_unix.go:518
 7  0x000000000042c1a4 in runtime.fatalpanic
    at /usr/local/go/src/runtime/panic.go:717
 8  0x000000000042bb55 in runtime.gopanic
    at /usr/local/go/src/runtime/panic.go:565
 9  0x0000000000440661 in runtime.panicmem
    at /usr/local/go/src/runtime/panic.go:82
10  0x0000000000440661 in runtime.sigpanic
    at /usr/local/go/src/runtime/signal_unix.go:390
11  0x00000000007aba4f in robot_server/internal.(*RouteServerConn).onHandleMsgJoinNotify
    at /home/go/src/robot_server/internal/route_server_conn.go:97
12  0x00000000007ab598 in robot_server/internal.(*RouteServerConn).onHandleData
    at /home/go/src/robot_server/internal/route_server_conn.go:65
13  0x00000000007aebd2 in robot_server/internal.(*RouteServerConn).onHandleData-fm
    at /home/go/src/robot_server/internal/route_server_conn.go:60
14  0x000000000079bff6 in robot_server/base.(*ImClient).NetLoop
    at /home/go/src/robot_server/base/im_client.go:168
15  0x00000000007ab3e1 in robot_server/internal.read
    at /home/go/src/robot_server/internal/route_server_conn.go:41
16  0x00000000004582d1 in runtime.goexit
    at /usr/local/go/src/runtime/asm_amd64.s:1337

如上图所示,11行找到了错误位置。

11  0x00000000007aba4f in robot_server/internal.(*RouteServerConn).onHandleMsgJoinNotify
    at /home/go/src/robot_server/internal/route_server_conn.go:97

如果运行机器上有代码的话(也可以把进程和core文件考到编译机器查看),可以显示具体错误行:

[root@10-0-59-231 robot-server.2020-04-22]# dlv core robot_server /data/imcorefile/core-robot_server-6-0-0-32402-1587555657
Type 'help' for list of commands.
(dlv)
(dlv)
(dlv) bt # 先使用bt显示错误堆栈 
 0  0x0000000000459be1 in runtime.raise
    at /usr/local/go/src/runtime/sys_linux_amd64.s:150
 1  0x000000000044074b in runtime.dieFromSignal
    at /usr/local/go/src/runtime/signal_unix.go:424
 2  0x0000000000440ccd in runtime.sigfwdgo
    at /usr/local/go/src/runtime/signal_unix.go:629
 3  0x000000000043fdf0 in runtime.sigtrampgo
    at /usr/local/go/src/runtime/signal_unix.go:289
 4  0x0000000000459ed3 in runtime.sigtramp
    at /usr/local/go/src/runtime/sys_linux_amd64.s:357
 5  0x0000000000459fc0 in runtime.sigreturn
    at /usr/local/go/src/runtime/sys_linux_amd64.s:449
 6  0x00000000004408ea in runtime.crash
    at /usr/local/go/src/runtime/signal_unix.go:518
 7  0x000000000042c1a4 in runtime.fatalpanic
    at /usr/local/go/src/runtime/panic.go:717
 8  0x000000000042bb55 in runtime.gopanic
    at /usr/local/go/src/runtime/panic.go:565
 9  0x0000000000440661 in runtime.panicmem
    at /usr/local/go/src/runtime/panic.go:82
10  0x0000000000440661 in runtime.sigpanic
    at /usr/local/go/src/runtime/signal_unix.go:390
11  0x00000000007aba4f in robot_server/internal.(*RouteServerConn).onHandleMsgJoinNotify
    at /home/go/src/robot_server/internal/route_server_conn.go:97
12  0x00000000007ab598 in robot_server/internal.(*RouteServerConn).onHandleData
    at /home/go/src/robot_server/internal/route_server_conn.go:65
13  0x00000000007aebd2 in robot_server/internal.(*RouteServerConn).onHandleData-fm
    at /home/go/src/robot_server/internal/route_server_conn.go:60
14  0x000000000079bff6 in robot_server/base.(*ImClient).NetLoop
    at /home/go/src/robot_server/base/im_client.go:168
15  0x00000000007ab3e1 in robot_server/internal.read
    at /home/go/src/robot_server/internal/route_server_conn.go:41
16  0x00000000004582d1 in runtime.goexit
    at /usr/local/go/src/runtime/asm_amd64.s:1337





(dlv) frame 11  # frame+行号,显示错误的错误信息
> runtime.raise() /usr/local/go/src/runtime/sys_linux_amd64.s:150 (PC: 0x459be1)
Warning: debugging optimized function
Frame 11: /home/go/src/robot_server/internal/route_server_conn.go:97 (PC: 7aba4f)
    92:		sessionId := *msg.SessionId
    93:		//if msg.SessionId != nil {
    94:		//	sessionId = *msg.SessionId
    95:		//}
    96:
=>  97:		logger.Sugar.Infof("onHandleWelcomeMsgList from_id:%d,to_id:%d,session_type:%d,app_id:%d", *msg.UserId,
    98:			sessionId, *msg.SessionType, *msg.AppId)
    99:		DefaultFaqHttpQueryPool.PushGetWelcomeReq(header, msg, false)
   100:	}

所以从上面能看到,应该是 msg.SessionId 空指针导致的崩溃。

daemon下如何设置

一般我们的服务都需要后台运行(守护进程),配合 restart.sh 等重启脚本。那么脚本里面如何设置 env GOTRACEBACK=crash 呢?

笔者尝试直接在shell脚本里面加好像没有用:

function start() {
    # 没用
    ./daemon env GOTRACEBACK=crash ./robot_server --log_dir=${log_dir} --conf=robotserver.conf &

    sleep 2
    ps -ef | grep robot_server
}

正确的做法是daemon进程里使用 exec.Command设置env环境 变量:

cmd := exec.Command(fullPath, args...)
// 在Start()之前设置,启用go core dump 功能
cmd.Env = append(cmd.Env, "GOTRACEBACK=crash")
err := cmd.Start()

完整代码见下面。

daemon.go

package main

import (
	"flag"
	"fmt"
	"os/exec"
	"path/filepath"
	"time"
)

var (
	appPath string
)

func main() {
	flag.Parse()

	args := make([]string, 0)
	for i := range flag.Args() {
		if i == 0 {
			appPath = flag.Arg(i)
		} else {
			args = append(args, flag.Arg(i))
		}
	}

	//fmt.Println(flag.Args())
	if appPath == "" {
		fmt.Println("Usage:./daemon [appPath] [args]")
		return
	}

	fullPath, _ := filepath.Abs(appPath)
	cmd := exec.Command(fullPath, args...)
	// 启用go core dump 功能
	cmd.Env = append(cmd.Env, "GOTRACEBACK=crash")
	err := cmd.Start()
	time.Sleep(time.Second)

	if err != nil {
		fmt.Printf("daemon error:%s \n", err.Error())
	} else {
		fmt.Println("daemon success")
	}
}

附录

restart.sh脚本

#!/bin/sh
#nohup ./robot_server -log_dir=log -alsologtostderr=true &

log_dir=./log

function create_log_dir() {
    if [[ ! -d "${log_dir}" ]];then
        mkdir -p ${log_dir}
    fi
}

function start() {
    ./daemon ./robot_server --log_dir=${log_dir} --conf=robotserver.conf &

    sleep 2
    ps -ef | grep robot_server
}

function stop() {
    if [[ -e server.pid  ]]; then
        pid=`cat server.pid`
        echo "kill pid=$pid"
        kill ${pid}
    fi
}

stop
create_log_dir
start

关闭core dump

ulimit -c        # 查看core dump状态,0代表关闭,unlimited代表打开
vim /etc/profile 

加入如下一句话:

# No core files by default
ulimit -S -c 0 > /dev/null 2>&1 

查看是否生效:

# 重新打开终端 如果要启用,把上面那句话注释重新打开终端即可
ulimit -c  如果输出0,代表关闭成功

重定向方案

参考weixin_34167043,这个没试过:

logFile, err := os.OpenFile("./log/fatal.log", os.O_CREATE|os.O_APPEND|os.O_RDWR, 0660)
if err != nil {
    log.Println("服务启动出错",  "打开异常日志文件失败" , err)
    return
}
// 将进程标准出错重定向至文件,进程崩溃时运行时将向该文件记录协程调用栈信息
syscall.Dup2(int(logFile.Fd()), int(os.Stderr.Fd())) 

参考

golang调试工具Delve
https://www.cnblogs.com/li-peng/p/8522592.html

golang coredump分析
https://blog.csdn.net/yunlilang/article/details/83014468?depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromBaidu-9&utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromBaidu-9

golang程序因未知错误崩溃时如何记录异常 https://blog.csdn.net/weixin_34167043/article/details/94651600?depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromBaidu-3&utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromBaidu-3

golang runtime
https://golang.org/src/runtime/crash_unix_test.go?m=text

关于

推荐下自己的开源IM,纯Golang编写:

CoffeeChat:
https://github.com/xmcy0011/CoffeeChat
opensource im with server(go) and client(flutter+swift)

参考了TeamTalk、瓜子IM等知名项目,包含服务端(go)和客户端(flutter),单聊和机器人(小微、图灵、思知)聊天功能已完成,目前正在研发群聊功能,欢迎对golang和跨平台开发flutter技术感兴趣的小伙伴Star加关注。

————————————————
版权声明:本文为CSDN博主「许非」的原创文章,遵循 CC 4.0 BY-SA 版权协议,转载请附上原文出处链接及本声明。
原文链接:https://blog.csdn.net/xmcy001122/article/details/103921991

评论 4
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值