go 进阶 gin实战相关: 五. gin_scaffold 企业脚手架

一. gin_scaffold 企业级脚手架

1. 为什么使用脚手架,防止重复造轮子,只关注业务开发即可
2. gin_scaffold 优点:

1. 提供了能够覆盖mysql/redis/request的链路功能,日志功能
2. 支持多语言错误信息提示以及自定义错误提示
3. 支持多环境配置
4. 封装了log/redis/mysql/http.client常用方法等
5. 支持swagger文档

二. gin_scaffold 脚手架安装及使用演示

1. gin_scaffold地址: https://github.com/e421083458/gin_scaffold
   

文件分层解释

├── README.md
├── conf            配置文件夹
│   └── dev         开发环境配置文件(多环境是创建对应其它环境的配置文件)
│       ├── base.toml
│       ├── mysql_map.toml
│       └── redis_map.toml
├── controller      控制器
│   └── demo.go
├── dao             DB数据层
│   └── demo.go
├── docs            swagger文件层
├── dto             输入输出结构层
│   └── demo.go
├── go.mod
├── go.sum
├── main.go         入口文件
├── middleware      中间件层
│   ├── panic.go
│   ├── response.go
│   ├── token_auth.go
│   └── translation.go
├── public          公共文件
│   ├── log.go
│   ├── mysql.go
│   └── validate.go
└── router          路由层
│   ├── httpserver.go
│   └── route.go
└── services        逻辑处理层

开始使用

1. 配置开启go mod 功能

参考:go 基础入门二十九 go mod

2. 下载 安装 gin_scaffold

1. 执行clone命令下载gin_scaffold到本地,地址: git clone git@github.com:e421083458/gin_scaffold.git
2. 下载完毕后跳转到下载的gin_scaffold目录下,
3. 执行"go mod tidy"命令,安装gin_scaffold相关类库
4. 使用ide编辑工具打开下载的gin_scaffold,找到"conf/mysql_map.toml","conf/redis_map.toml"配置文件,配置连接自己的mysql,redis
   
   
5. 执行go run main.go 启动测试一下

实际就是拉取一个空的服务baes,测试启动一下,如果能够启动,在这个base上进行业务开发

3. 整合 golang_common

1. gin_scaffold 下整合了golang_common包,在该包中提供了一下功能:

1. 支持多环境运行设置，比如：dev、prod。
2. 支持mysql、redis 多套数据源配置。
3. 支持默认和自定义日志实例，自动滚动日志。
4. 封装了 mysql(gorm.io/gorm v1.22.4)、redis(redigo)、http.client整合及常用方法
5. 支持 mysql(gorm.io/gorm v1.22.4)、redis(redigo)、http.client 请求链路日志输出
   

2. 官方参考文档
3. 注意在使用golang_common中提供的相关方法时,对应组件可能要提供对应的配置文件,要将这些配置文件放到项目中,官方提供了相关配置文件示例
   
4. 实际就是gin_scaffold 整合了一个golang_common工具包,这个工具包中封装了一些针对mysql,redis,httpclient的相关方法,整合了以后,如果在项目中使用mysql,redis时,直接通过golang_common中提供的方法使用即可,在golang_common中整合mysql,redis,httpclient时可能会用到相关的配置文件,将对应的配置文件放到项目中即可
5. 注意: 如果拉下来的gin_scaffold脚手架项目中没有golang_common这个包,或者已经存在的项目后续开发中整合golang_common时:

1. 项目中创建golang_common文件夹,访问官方文档地址,获取lib,log放到新建的golang_common文件夹下,
2. 获取相关配置文件,放入conf目录下,
3. 执行 “go get -v github.com/e421083458/golang_common” 拉取golang_common包
4. 使用golang_common下提供的方法即可
   

4. 测试链路

1. 执行sql脚本,创建测试数据

CREATE TABLE `area` (
 `id` bigint(20) NOT NULL AUTO_INCREMENT,
 `area_name` varchar(255) NOT NULL,
 `city_id` int(11) NOT NULL,
 `user_id` int(11) NOT NULL,
 `update_at` datetime NOT NULL,
 `create_at` datetime NOT NULL,
 `delete_at` datetime NOT NULL,
 PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=utf8 COMMENT='area';
INSERT INTO `area` (`id`, `area_name`, `city_id`, `user_id`, `update_at`, `create_at`, `delete_at`) VALUES (NULL, 'area_name', '1', '2', '2019-06-15 00:00:00', '2019-06-15 00:00:00', '2019-06-15 00:00:00');

2. 请求接口,查看响应,其中"trace_id"就是链路唯一id

curl 'http://127.0.0.1:8880/demo/dao?id=1'
{
    "errno": 0,
    "errmsg": "",
    "data": "[{\"id\":1,\"area_name\":\"area_name\",\"city_id\":1,\"user_id\":2,\"update_at\":\"2019-06-15T00:00:00+08:00\",\"create_at\":\"2019-06-15T00:00:00+08:00\",\"delete_at\":\"2019-06-15T00:00:00+08:00\"}]",
    "trace_id": "c0a8fe445d05b9eeee780f9f5a8581b0"
}

3. 查看链路日志

//gin_scaffold.inf.log下存放了所有正常info级别日志
tail -f gin_scaffold.inf.log
//gin_scaffold.wf.log下存放了所有的异常日志
tail -f gin_scaffold.wf.log

5. swagger文档生成

1. 下载对应操作系统的执行文件到$GOPATH/bin下面(下载后解压获取到bin文件夹下的"swag"文件,复制到$GOPATH/bin下)

//如
➜  gin_scaffold git:(master) ✗ ll -r $GOPATH/bin
total 434168
-rwxr-xr-x  1 niuyufu  staff    13M  4  3 17:38 swag

2. 编写接口,增加swagger注释,参考一下示例

// ListPage godoc
// @Summary 测试数据绑定
// @Description 测试数据绑定
// @Tags 用户
// @ID /demo/bind
// @Accept  json
// @Produce  json
// @Param polygon body dto.DemoInput true "body"
// @Success 200 {object} middleware.Response{data=dto.DemoInput} "success"
// @Router /demo/bind [post]

3. 执行命令生成接口文档: swag init, 生成docs文件夹,生成swagger相关文件
4. 启动服务, swagger整合完毕后访问接口文档页面: “http://127.0.0.1:8880/swagger/index.html”
5. gin 集成swagger参考博客
6. swagger命令报错问题

6. gin_scaffold 下conf 下配置文件

1. base.toml 配置文件

# This is base config

[base]
    debug_mode="debug"
    time_location="Asia/Chongqing"

[http]
    addr =":8880"                       # 监听地址, default ":8700"
    read_timeout = 10                   # 读取超时时长
    write_timeout = 10                  # 写入超时时长
    max_header_bytes = 20               # 最大的header大小，二进制位长度
    allow_ip = [                        # 白名单ip列表
        "127.0.0.1",
        "192.168.1.1"
    ]

[session]
    redis_server = "127.0.0.1:6379"   #redis session server
    redis_password = ""

[log]
    log_level = "trace"         #日志打印最低级别
    [log.file_writer]           #文件写入配置
        on = true
        log_path = "./logs/go_gateway.inf.log"
        rotate_log_path = "./logs/go_gateway.inf.log.%Y%M%D%H"
        wf_log_path = "./logs/go_gateway.wf.log"
        rotate_wf_log_path = "./logs/go_gateway.wf.log.%Y%M%D%H"
    [log.console_writer]        #工作台输出
        on = false
        color = false

[cluster]
    cluster_ip="127.0.0.1"
    cluster_port="8080"
    cluster_ssl_port="4433"

[swagger]
    title="go_gateway swagger API"
    desc="This is a sample server celler server."
    host="127.0.0.1:8880"
    base_path=""

2. mysql_map.toml配置文件

# this is mysql config
[list]
    [list.default]
        driver_name = "mysql"
        data_source_name = "root:123456@tcp(127.0.0.1:3306)/go_gateway2?charset=utf8&parseTime=true&loc=Asia%2FChongqing"
        max_open_conn = 20
        max_idle_conn = 10
        max_conn_life_time = 100

3. redis_map.toml 配置文件

# this is redis config file
[list]
    [list.default]
        proxy_list = ["127.0.0.1:6379"]
        conn_timeout = 500
        password = ""
        db = 0
        read_timeout = 1000
        write_timeout = 1000
        max_active = 200
        max_idle = 500

7. gin_scaffold main方法启动服务解释

1. 初始化路由,启动服务方法,与关闭服务方法

import (
	"context"
	"go_test/golang_common/lib"
	"github.com/gin-gonic/gin"
	"log"
	"net/http"
	"time"
)

var (
	HttpSrvHandler *http.Server
)

//1.初始化路由,以及监听端口启动服务
func HttpServerRun() {
	//1.默认在debug下运行
	gin.SetMode(lib.GetStringConf("base.base.debug_mode"))
	//2.初始化路由(自定义函数在下方)
	r := InitRouter()
	//3.创建server服务
	HttpSrvHandler = &http.Server{
		Addr:           lib.GetStringConf("base.http.addr"),
		Handler:        r,
		ReadTimeout:    time.Duration(lib.GetIntConf("base.http.read_timeout")) * time.Second,
		WriteTimeout:   time.Duration(lib.GetIntConf("base.http.write_timeout")) * time.Second,
		MaxHeaderBytes: 1 << uint(lib.GetIntConf("base.http.max_header_bytes")),
	}
	
	//4.启动服务
	go func() {
		log.Printf(" [INFO] HttpServerRun:%s\n",lib.GetStringConf("base.http.addr"))
		if err := HttpSrvHandler.ListenAndServe(); err != nil {
			log.Fatalf(" [ERROR] HttpServerRun:%s err:%v\n", lib.GetStringConf("base.http.addr"), err)
		}
	}()
}

//2.关闭服务方法
func HttpServerStop() {
	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
	defer cancel()
	if err := HttpSrvHandler.Shutdown(ctx); err != nil {
		log.Fatalf(" [ERROR] HttpServerStop err:%v\n", err)
	}
	log.Printf(" [INFO] HttpServerStop stopped\n")
}

2. main

import (
	"flag"
	"go_test/golang_common/lib"
	"go_test/router"
	"os"
	"os/signal"
	"syscall"
)

func main(){
	//1.设置环境变量,如果第一个参数configPath为空,则会通过命令行读取"-config=./conf/prod/"下的文件
	//整合是注意此处的路径是否正确
	config := flag.String("config", "./config/dev/", "input config file like ./config/dev/")
	//lib是golang_common包下的
	lib.InitModule(*config)
	//2.退出
	defer lib.Destroy()
	
	//3.调用上面的初始化router,开启服务方法
	router.HttpServerRun()

	//4.监听系统的停止信号(也就是监听syscall.SIGINT与syscall.SIGTERM)
	quit:=make(chan os.Signal)
	signal.Notify(quit,syscall.SIGINT, syscall.SIGTERM)
	<-quit

	//5.当拿到监听的信号后说明需要关闭服务,调用上面的关闭服务方法
	router.HttpServerStop()
}

8. gin_scaffold 中初始化路由简单解释

func InitRouter(middlewares ...gin.HandlerFunc) *gin.Engine {
	//读取conf下配置文件中自定义配置数据,
	//通过gin官方提供的NewRedisStore,在redis获取session
	//store, err := sessions.NewRedisStore(10, "tcp", lib.GetStringConf("base.session.redis_server"), lib.GetStringConf("base.session.redis_password"), []byte("secret"))
	//if err != nil {
	//	log.Fatalf("sessions.NewRedisStore err:%v", err)
	//}

	//1.设置swagger相关
	docs.SwaggerInfo.Title = lib.GetStringConf("base.swagger.title")
	docs.SwaggerInfo.Description = lib.GetStringConf("base.swagger.desc")
	docs.SwaggerInfo.Version = "1.0"
	docs.SwaggerInfo.Host = lib.GetStringConf("base.swagger.host")
	docs.SwaggerInfo.BasePath = lib.GetStringConf("base.swagger.base_path")
	docs.SwaggerInfo.Schemes = []string{"http", "https"}

	//2.获取到默认的router
	router := gin.Default()
	router.Use(middlewares...)
	//3.绑定默认提供的/ping接口
	router.GET("/ping", func(c *gin.Context) {
		c.JSON(200, gin.H{
			"message": "pong",
		})
	})
	
	//4.绑定swagger相关接口
	router.GET("/swagger/*any", ginSwagger.WrapHandler(swaggerFiles.Handler))

	//5.设置group前缀
	adminLoginRouter := router.Group("/admin_login")
	//路由注册
	adminLoginRouter.Use( //设置中间件(下方只是示例)
		//sessions.Sessions("mysession", store), //session相关中间件
		middleware.RecoveryMiddleware(),       //用来捕获panic异常的中间件
		middleware.RequestLog(),               //打印链路日志相关的中间件
		middleware.IPAuthMiddleware(),         //权限相关中间件
		middleware.SessionAuthMiddleware(),	//校验session是否登录的中间件
		middleware.TranslationMiddleware()) //多语言翻译中间件
	{	
		//6.调用controller下的接口
		controller.AdminLoginRegister(adminLoginRouter)
	}
}

9. gin_scaffold 中controller简单示例

import (
	"encoding/json"
	"fmt"
	"github.com/e421083458/go_gateway/dao"
	"github.com/e421083458/go_gateway/dto"
	"github.com/e421083458/go_gateway/middleware"
	"github.com/e421083458/go_gateway/public"
	"github.com/e421083458/go_gateway/golang_common/lib"
	"github.com/gin-gonic/contrib/sessions"
	"github.com/gin-gonic/gin"
)

//1.创建controller结构体
type AdminController struct{}

//2.AdminController 接口体上绑定的业务方法(下方的注释时swagger相关的)
// AdminInfo godoc
// @Summary 管理员信息
// @Description 管理员信息
// @Tags 管理员接口
// @ID /admin/admin_info
// @Accept  json
// @Produce  json
// @Success 200 {object} middleware.Response{data=dto.AdminInfoOutput} "success"
// @Router /admin/admin_info [get]
func (adminlogin *AdminController) AdminInfo(c *gin.Context) {
	//1.获取session
	sess := sessions.Default(c)
	sessInfo := sess.Get(public.AdminSessionInfoKey)
	//2.在session中读取接收到的数据转换为结构体
	adminSessionInfo := &dto.AdminSessionInfo{}
	if err := json.Unmarshal([]byte(fmt.Sprint(sessInfo)), adminSessionInfo); err != nil {
		middleware.ResponseError(c, 2000, err)
		return
	}
	
	//3.设置响应
	out := &dto.AdminInfoOutput{
		ID:           adminSessionInfo.ID,
		Name:         adminSessionInfo.UserName,
	}
	middleware.ResponseSuccess(c, out)
}

//业务方法2
// ChangePwd godoc
// @Summary 修改密码
// @Description 修改密码
// @Tags 管理员接口
// @ID /admin/change_pwd
// @Accept  json
// @Produce  json
// @Param body body dto.ChangePwdInput true "body"
// @Success 200 {object} middleware.Response{data=string} "success"
// @Router /admin/change_pwd [post]
func (adminlogin *AdminController) ChangePwd(c *gin.Context) {
	//1.读取并校验请求参数
	params := &dto.ChangePwdInput{}
	if err := params.BindValidParam(c); err != nil {
		middleware.ResponseError(c, 2000, err)
		return
	}

	//2.在session读取用户信息到结构体
	sess := sessions.Default(c)
	sessInfo := sess.Get(public.AdminSessionInfoKey)
	adminSessionInfo := &dto.AdminSessionInfo{}
	if err := json.Unmarshal([]byte(fmt.Sprint(sessInfo)), adminSessionInfo); err != nil {
		middleware.ResponseError(c, 2000, err)
		return
	}

	//3.从数据库中读取 adminInfo
	//进行数据库操作时,先获取到默认的gorm连接池
	tx, err := lib.GetGormPool("default")
	if err != nil {
		middleware.ResponseError(c, 2001, err)
		return
	}
	//执行gorm操作
	adminInfo := &dao.Admin{}
	adminInfo, err = adminInfo.Find(c, tx, (&dao.Admin{UserName: adminSessionInfo.UserName}))
	if err != nil {
		middleware.ResponseError(c, 2002, err)
		return
	}

	//生成新密码 saltPassword
	saltPassword := public.GenSaltPassword(adminInfo.Salt, params.Password)
	adminInfo.Password = saltPassword

	//执行数据保存
	if err := adminInfo.Save(c, tx); err != nil {
		middleware.ResponseError(c, 2003, err)
		return
	}
	middleware.ResponseSuccess(c, "")
}


//3.当前controller注册到Group(也就是controller中第5步骤调用的方法)
func AdminRegister(group *gin.RouterGroup) {
	adminLogin := &AdminController{}
	group.GET("/admin_info", adminLogin.AdminInfo)
	group.POST("/change_pwd", adminLogin.ChangePwd)
}

10. gin_scaffold 操作数据库的dao示例

import (
	"github.com/e421083458/go_gateway/dto"
	"github.com/e421083458/go_gateway/public"
	"github.com/e421083458/gorm"
	"github.com/gin-gonic/gin"
	"github.com/pkg/errors"
	"time"
)

type Admin struct {
	Id        int       `json:"id" gorm:"primary_key" description:"自增主键"`
	UserName  string    `json:"user_name" gorm:"column:user_name" description:"管理员用户名"`
	Salt      string    `json:"salt" gorm:"column:salt" description:"盐"`
	Password  string    `json:"password" gorm:"column:password" description:"密码"`
	UpdatedAt time.Time `json:"update_at" gorm:"column:update_at" description:"更新时间"`
	CreatedAt time.Time `json:"create_at" gorm:"column:create_at" description:"创建时间"`
	IsDelete  int       `json:"is_delete" gorm:"column:is_delete" description:"是否删除"`
}

func (t *Admin) TableName() string {
	return "gateway_admin"
}

func (t *Admin) LoginCheck(c *gin.Context, tx *gorm.DB, param *dto.AdminLoginInput) (*Admin, error) {
	adminInfo, err := t.Find(c, tx, (&Admin{UserName: param.UserName, IsDelete: 0}))
	if err != nil {
		return nil, errors.New("用户信息不存在")
	}
	saltPassword := public.GenSaltPassword(adminInfo.Salt, param.Password)
	if adminInfo.Password != saltPassword {
		return nil, errors.New("密码错误，请重新输入")
	}
	return adminInfo, nil
}

func (t *Admin) Find(c *gin.Context, tx *gorm.DB, search *Admin) (*Admin, error) {
	out := &Admin{}
	err := tx.SetCtx(public.GetGinTraceContext(c)).Where(search).Find(out).Error
	if err != nil {
		return nil, err
	}
	return out, nil
}

func (t *Admin) Save(c *gin.Context, tx *gorm.DB) error {
	return tx.SetCtx(public.GetGinTraceContext(c)).Save(t).Error
}

11. gin_scaffold/middleware/下的中间件示例

1. 捕获panic异常的中间件示例

import (
	"errors"
	"fmt"
	"github.com/e421083458/go_gateway/public"
	"github.com/e421083458/go_gateway/golang_common/lib"
	"github.com/gin-gonic/gin"
	"runtime/debug"
)

// RecoveryMiddleware捕获所有panic，并且返回错误信息
func RecoveryMiddleware() gin.HandlerFunc {
	return func(c *gin.Context) {
		defer func() {
			if err := recover(); err != nil {
				//先做一下日志记录
				fmt.Println(string(debug.Stack()))
				public.ComLogNotice(c, "_com_panic", map[string]interface{}{
					"error": fmt.Sprint(err),
					"stack": string(debug.Stack()),
				})
				if lib.ConfBase.DebugMode != "debug" {
					ResponseError(c, 500, errors.New("内部错误"))
					return
				} else {
					ResponseError(c, 500, errors.New(fmt.Sprint(err)))
					return
				}
			}
		}()
		c.Next()
	}
}

2. 打印请求日志中间件示例

import (
	"bytes"
	"github.com/e421083458/go_gateway/public"
	"github.com/e421083458/go_gateway/golang_common/lib"
	"github.com/gin-gonic/gin"
	"io/ioutil"
	"time"
)

// 请求进入日志
func RequestInLog(c *gin.Context) {
	traceContext := lib.NewTrace()
	if traceId := c.Request.Header.Get("com-header-rid"); traceId != "" {
		traceContext.TraceId = traceId
	}
	if spanId := c.Request.Header.Get("com-header-spanid"); spanId != "" {
		traceContext.SpanId = spanId
	}

	c.Set("startExecTime", time.Now())
	c.Set("trace", traceContext)

	bodyBytes, _ := ioutil.ReadAll(c.Request.Body)
	c.Request.Body = ioutil.NopCloser(bytes.NewBuffer(bodyBytes)) // Write body back

	lib.Log.TagInfo(traceContext, "_com_request_in", map[string]interface{}{
		"uri":    c.Request.RequestURI,
		"method": c.Request.Method,
		"args":   c.Request.PostForm,
		"body":   string(bodyBytes),
		"from":   c.ClientIP(),
	})
}

// 请求输出日志
func RequestOutLog(c *gin.Context) {
	// after request
	endExecTime := time.Now()
	response, _ := c.Get("response")
	st, _ := c.Get("startExecTime")

	startExecTime, _ := st.(time.Time)
	public.ComLogNotice(c, "_com_request_out", map[string]interface{}{
		"uri":       c.Request.RequestURI,
		"method":    c.Request.Method,
		"args":      c.Request.PostForm,
		"from":      c.ClientIP(),
		"response":  response,
		"proc_time": endExecTime.Sub(startExecTime).Seconds(),
	})
}

func RequestLog() gin.HandlerFunc {
	return func(c *gin.Context) {
		//todo 优化点4
		if lib.GetBoolConf("base.log.file_writer.on") {
			RequestInLog(c)
			defer RequestOutLog(c)
		}
		c.Next()
	}
}

3. 封装响应的中间件示例

package middleware

import (
	"encoding/json"
	"fmt"
	"github.com/e421083458/go_gateway/golang_common/lib"
	"github.com/gin-gonic/gin"
	"strings"
)

type ResponseCode int

//1000以下为通用码，1000以上为用户自定义码
const (
	SuccessCode ResponseCode = iota
	UndefErrorCode
	ValidErrorCode
	InternalErrorCode

	InvalidRequestErrorCode ResponseCode = 401
	CustomizeCode           ResponseCode = 1000

	GROUPALL_SAVE_FLOWERROR ResponseCode = 2001
)

type Response struct {
	ErrorCode ResponseCode `json:"errno"`
	ErrorMsg  string       `json:"errmsg"`
	Data      interface{}  `json:"data"`
	TraceId   interface{}  `json:"trace_id"`
	Stack     interface{}  `json:"stack"`
}

func ResponseError(c *gin.Context, code ResponseCode, err error) {
	trace, _ := c.Get("trace")
	traceContext, _ := trace.(*lib.TraceContext)
	traceId := ""
	if traceContext != nil {
		traceId = traceContext.TraceId
	}

	stack := ""
	if c.Query("is_debug") == "1" || lib.GetConfEnv() == "dev" {
		stack = strings.Replace(fmt.Sprintf("%+v", err), err.Error()+"\n", "", -1)
	}

	resp := &Response{ErrorCode: code, ErrorMsg: err.Error(), Data: "", TraceId: traceId, Stack: stack}
	c.JSON(200, resp)
	response, _ := json.Marshal(resp)
	c.Set("response", string(response))
	c.AbortWithError(200, err)
}

func ResponseSuccess(c *gin.Context, data interface{}) {
	trace, _ := c.Get("trace")
	traceContext, _ := trace.(*lib.TraceContext)
	traceId := ""
	if traceContext != nil {
		traceId = traceContext.TraceId
	}

	resp := &Response{ErrorCode: SuccessCode, ErrorMsg: "", Data: data, TraceId: traceId}
	c.JSON(200, resp)
	response, _ := json.Marshal(resp)
	c.Set("response", string(response))
}

12. 将gin_scaffold脚手架修改为自己的指定服务

1. 复制一个gin_scaffold副本出来,重命名为自己的项目,
2. 新项目中去除拉取gin_scaffold是的git文件,以及".ide"一类的相关文件
3. 通过ide开发工具打开, 设置go module
   
4. 将原项目中gin_scaffold文件名,文件路径等等全局替换掉
5. 去除一下无用的controller,dao等等