导轨cad图标_为您的go项目构建一个类似于MigrationRunner的导轨

最新推荐文章于 2024-08-13 08:23:53 发布

郝ren

最新推荐文章于 2024-08-13 08:23:53 发布

阅读量231

点赞数

文章标签： python java vue ViewUI

原文链接：https://medium.com/better-programming/build-a-rails-like-migration-runner-for-your-go-projects-b72f551597a3

版权

导轨cad图标

Finding myself with a week free, I settled down to port an old Rails project into Go. I diligently spruced up the database schemas, planned the packages I’d use, and had an internal debate over whether to use serial IDs, UUIDs, or ULIDs. It was only when all the fine details were nailed that I realised the obvious — before I began my app, I’d need some way to manage migrations.

找到一周的空闲时间后，我决定将一个旧的Rails项目移植到Go中。我努力地整理了数据库架构，计划了我将要使用的软件包，并就是否使用序列ID，UUID或ULID进行了内部辩论。只有当所有细节都被钉上时，我才意识到这很明显–在我启动应用程序之前，我需要某种方式来管理迁移。

Rails is a full-stack web framework designed to make getting projects off the ground a breeze. Go is a programming language that excels in distributed systems and shuns frameworks on principle. Although Go web frameworks exist (Gin, for example), I felt there was more to be learned by minimising “magic” and writing some tooling of my own.

Rails是一个完整的Web框架，旨在使轻而易举地启动项目。 Go是一种编程语言，在分布式系统上表现出色，并且在原则上避免使用框架。尽管存在Go Web框架(例如，Gin)，但我认为，通过最小化“魔术”并编写自己的工具，还有很多要学习的东西。

Specifically, I needed the ability to reliably migrate development and test databases up and down so when I inevitably made a mess of application development, it’d be painless to roll the DB back and try again.

具体来说，我需要能够可靠地上下迁移开发和测试数据库的功能，因此当我不可避免地进行一堆应用程序开发时，回滚数据库并重试是很容易的。

1.环境 (1. Environment)

Migrate (golang-migrate/migrate) is a phenomenal package for running migrations, and it comes with both a library you can use in your Go applications and a command-line tool to generate and trigger migration files.

Migrate( golang-migrate/migrate )是一个golang-migrate/migrate软件包，用于运行迁移，它附带可在Go应用程序中使用的库以及命令行工具来生成和触发迁移文件。

The problem with using the CLI out of the box, however, is there’s a lot of faffing around setting environment variables and manually typing database addresses. I wanted seamlessness — a solution that’d load the correct environment config for my application and migrate the right database without needing me to hold its hand. That meant getting stuck into the library.

但是，开箱即用地使用CLI的问题在于设置环境变量和手动键入数据库地址方面存在很多麻烦。我想要无缝化—一种解决方案，可以为我的应用程序加载正确的环境配置并迁移正确的数据库，而无需我握住它的手。这意味着要陷入图书馆。

As the first step, I settled on a variable to signal the current environment to my program: FB05_ENV=[development|test|staging|production]. FB05 is the name of my application — call yours whatever you like. I wanted this to be the only env var I’d have to manually change.

作为第一步，我确定了一个变量以向程序发送当前环境信号： FB05_ENV=[development|test|staging|production] 。 FB05是我的应用程序的名称-随便叫什么。我希望这是我必须手动更改的唯一环境变量。

Second, I created environment.yaml in my project root. This will eventually hold every env var my app needs to run (taking care to conceal the sensitive ones, of course). So far it’s limited to the database config:

其次，我在项目根目录中创建了environment.yaml 。这最终将容纳我的应用程序需要运行的所有环境(当然，请注意隐藏敏感的环境)。到目前为止，它仅限于数据库配置：

development: &development
  FB05_DB_USER: fb05_dev
  FB05_DB_PASSWORD: password
  FB05_DB_HOST: localhost
  FB05_DB_PORT: 5432
  FB05_DB_NAME: fb05_development
test: &test
  <<: *development
  FB05_DB_NAME: fb05_test

This doesn’t have to be a YAML file. If you’re a fan of a .env or some other configuration format, that’ll work too.

这不必是YAML文件。如果您是.env或其他配置格式的粉丝，那也可以使用。

Assuming you’ve created the corresponding databases separately, that’s all the set up that’s necessary to start building our runner.

假设您已经分别创建了相应的数据库，那么这就是开始构建运行器所需的全部设置。

2.“ migrate.go” (2. ‘migrate.go’)

I’ve named the runner migrate.go, and it lives at cmd/migrate/migrate.go under the project root. Eventually, we’ll use make commands similar to Rail’s rake db:migrate to run this script with the appropriate arguments.

我已经将其命名为“ runner migrate.go ，它位于项目根目录下的cmd/migrate/migrate.go 。最终，我们将使用类似于Rail的rake db:migrate make命令来使用适当的参数运行此脚本。

Let’s start with the main function:

让我们从main函数开始：

func main() {
	// Flags to control environment variable loading.
	configName := flag.String(
		"configName",
		"environment",
		"The name of the config file (without extension) containing env vars required for the migration.",
	)
	configPath := flag.String(
		"configPath",
		".",
		"The path to the config file containing env vars required for the migration.",
	)
	flag.Parse()


	// Ensure a command has been provided.
	args := flag.Args()
	if len(args) == 0 {
		fmt.Println("migrate must be used with a migration command")
		fmt.Println("Usage: migrate down | drop | up | version | force number | step number | toVersion number [-configName string] [-configPath string]")
		os.Exit(1)
	}


	// Load environment variables
	var err error
	env, err = loadEnvVars(*configName, *configPath)
	if err != nil {
		fmt.Fprintf(os.Stderr, "migrate: %v\n", err)
		os.Exit(1)
	}


	// Run the specified command
	err = runMigration(args[0], args[1:])
	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}


	fmt.Println("Success!")
}

On lines 3-13, we specify and parse flags that control where our program should look for the environment config file. In this case, I’ve set the defaults to point to environment.yaml in the root of our project.

在第3-13行，我们指定并解析用于控制程序在何处查找环境配置文件的标志。在这种情况下，我将默认值设置为指向项目根目录中的environment.yaml 。

Next, we check to make sure the script has been called with at least one argument: the command getting the Migrate package to run. Some of these commands (like (m* Migrate) Steps(n int), for example), take arguments of their own, but we don’t check for that yet.

接下来，我们检查以确保已使用至少一个参数调用了脚本：使Migrate程序包运行的命令。其中一些命令(例如(m* Migrate) Steps(n int)例如)带有自己的参数，但我们尚未对其进行检查。

On lines 24-29, we load the environment file described by our flags. Behind the scenes, this is done by a package called Viper. More on that shortly.

在第24-29行，我们加载由标志描述的环境文件。在后台，这是通过名为Viper的软件包完成的。不久之后会更多。

With the environment successfully loaded, we’re in a position to run the migration with the specified command and any arguments that have been passed.

成功加载环境后，我们可以使用指定的命令和已传递的所有参数来运行迁移。

Let’s take a closer look at loadEnvVars and runMigration.

让我们仔细看看loadEnvVars和runMigration 。

`'loadEnvVars'` (`'loadEnvVars'`)

// targetEnvKey is the environment variable that specifies the
// current project environment, e.g. development, test, etc.
const targetEnvKey = "FB05_ENV"


func loadEnvVars(configName, configPath string) (*viper.Viper, error) {
	targetEnv := os.Getenv(targetEnvKey)
	if targetEnv == "" {
		return nil, fmt.Errorf("loadEnvVars: target environment unknown, %s is blank", targetEnvKey)
	}


	fmt.Printf("Loading environment %q...\n", targetEnv)
	viper.SetConfigName(configName)
	viper.SetConfigType("yaml")
	viper.AddConfigPath(configPath)
	if err := viper.ReadInConfig(); err != nil {
		return nil, fmt.Errorf("loadEnvVars: %v", err)
	}
	// Return only the variables for the target environment.
	return viper.Sub(targetEnv), nil
}

loadEnvVars takes the configName and configPath flags we parsed in main and returns a pointer to a viper.Viper.

loadEnvVars采用我们在main解析的configName和configPath标志，并返回指向viper.Viper的指针。

Viper is a high-powered configuration package for Go applications and, as such, it’s overkill for a script like this. However, I’m planning to use Viper elsewhere in my project, so it made sense to pull it in here too. If you’d prefer something lightweight, consider envconfig, GoDotEnv, or simply parsing the environment file yourself.

Viper是用于Go应用程序的高性能配置包，因此，对于这样的脚本而言，它是过高的。但是，我计划在项目的其他地方使用Viper，因此也可以将其引入此处。如果您希望使用轻量级的产品，请考虑使用envconfig ， GoDotEnv或自行解析环境文件。

On lines 12-14, we configure Viper to read from a file called environment.yaml located in the project root. Notice that I didn’t create a flag called configType — I’ve hardcoded yaml because I know my config files will always be YAML. You may decide differently.

在第12-14行中，我们将Viper配置为从位于项目根目录中的名为environment.yaml的文件读取。请注意，我没有创建名为configType的标志-我已经对yaml硬编码，因为我知道我的配置文件将始终为YAML。您可能会做出不同的决定。

viper.ReadInConfig causes Viper to read the env vars contained in the source file and create a *viper.Viper. When we want to access an environment variable, we query this struct with methods like (v *Viper) Get(key string) instead of running os.Getenv.

viper.ReadInConfig使Viper读取源文件中包含的环境变量，并创建一个*viper.Viper 。当我们想要访问环境变量时，我们使用(v *Viper) Get(key string)类的方法查询此结构，而不是运行os.Getenv 。

But we’re not done! Viper has loaded the whole environment.yaml file, including configuration for test, production, and any other group you might have stored in there. That’s why the first thing loadEnvVars does is to look for the target environment with os.Getenv("FB05_ENV"). We pass this value to viper.Sub to return a *viper.Viper that contains only the subset of environment variables that match our target environment.

但是我们还没有完成！ Viper已加载了整个environment.yaml文件，包括test ， production配置，以及您可能存储在其中的任何其他组。这就是为什么loadEnvVars的第一件事是使用os.Getenv("FB05_ENV")查找目标环境。我们将此值传递给viper.Sub以返回*viper.Viper ，其中仅包含与目标环境匹配的环境变量的子集。

What happens to this *viper.Viper when loadEnvVars returns? You’ll notice that I don’t declare any new variables in when I call env, err = loadEnvVars(*configName, *configPath) on main:25. env is actually a global variable declared outside of main: var env *viper.Viper. That way, it’s accessible anywhere in the script, just like os.Getenv.

当loadEnvVars返回时，此*viper.Viper会发生什么？您会注意到，在调用env, err = loadEnvVars(*configName, *configPath)在main:25上没有声明任何新变量env, err = loadEnvVars(*configName, *configPath) 。 env实际上是在main之外声明的全局变量： var env *viper.Viper 。这样，就可以在脚本中的任何位置访问它，就像os.Getenv一样。

'runMigration' (‘runMigration’)

This is a chunky one. Hold tight:

这是一个矮胖的人。抓紧：

func runMigration(command string, args []string) error {
	fmt.Printf("Running migrate %s %s...\n", command, strings.Join(args, " "))
	m, err := migrate.New("file://db/migrations", databaseURL())
	if err != nil {
		return migrationError(command, err)
	}


	// Ensure numeric arguments can be parsed.
	var n int
	if len(args) > 0 {
		n, err = strconv.Atoi(args[0])
		if err != nil {
			return migrationError(command, err)
		}
	}


	// Run command.
	switch command {
	case "down":
		err = m.Down()
	case "drop":
		err = m.Drop()
	case "force":
		if len(args) == 0 {
			return migrationError(command, errors.New("a migration version number is required"))
		}
		err = m.Force(n)
	case "steps":
		if len(args) == 0 {
			return migrationError(command, errors.New("the number of steps to migrate is required"))
		}
		err = m.Steps(n)
	case "toVersion":
		if len(args) == 0 {
			return migrationError(command, errors.New("a migration version number is required"))
		}
		err = m.Migrate(uint(n))
	case "up":
		err = m.Up()
	case "version":
		version, dirty, err := m.Version()
		if err != nil {
			return migrationError(command, err)
		}
		fmt.Printf("\tVersion: %d\n\tDirty: %t\n", version, dirty)
	default:
		return migrationError(command, errors.New("unknown command"))
	}


	if err != nil {
		return migrationError(command, err)
	}
	return nil
}


func migrationError(command string, err error) error {
	return fmt.Errorf("migrate %s: %v", command, err)
}

First up, we use the Migration package to create a new *migrate.Migration. There’s some complexity in the setup here because you have to register different subpackages of Migrate by blank importing them depending on where your migrations are stored and which DBMS you’re using.

首先，我们使用Migration包创建一个新的*migrate.Migration 。这里的设置有些复杂，因为您必须根据迁移的存储位置和使用的DBMS，通过空白导入来注册不同的Migrate子程序包。

I’m storing my migrations locally, and my drug of choice is PostgreSQL. So my imports look like this:

我将迁移存储在本地，而我选择的药物是PostgreSQL。所以我的导入看起来像这样：

import (
	"errors"
	"flag"
	"fmt"
	"os"
	"strconv"
	"strings"


	"github.com/golang-migrate/migrate/v4"
	_ "github.com/golang-migrate/migrate/v4/database/postgres"
	_ "github.com/golang-migrate/migrate/v4/source/file"
	"github.com/spf13/viper"
)

The first argument to migrate.New is the source: the location your migration files are stored in. Mine are at db/migrations under my project root.

第一个参数migrate.New是source ：迁移文件存储在矿的位置是在db/migrations在我的项目的根。

The second is the database URL, and this is the reason why loading environment variables before running your migrations is so valuable:

第二个是数据库URL，这就是为什么在运行迁移之前加载环境变量如此有价值的原因：

func databaseURL() string {
	return fmt.Sprintf(
		"postgres://%s:%s@%s:%d/%s?sslmode=disable",
		env.Get("FB05_DB_USER"),
		env.Get("FB05_DB_PASSWORD"),
		env.Get("FB05_DB_HOST"),
		env.Get("FB05_DB_PORT"),
		env.Get("FB05_DB_NAME"),
	)
}

Imagine having to type this out every time you wanted to run a migration. Imagine having to run your migrations by hand in production. These are the problems this miniproject sets out to solve.

想象一下，每次您要运行迁移时都必须输入此内容。想象一下，必须在生产中手动运行迁移。这些是这个小型项目着手解决的问题。

After creating the *migrate.Migration and validating that any numeric arguments provided are indeed numbers, we switch using the specified command and trigger the corresponding Migrate method.

在创建*migrate.Migration并验证提供的任何数字参数确实是数字之后，我们使用指定的命令进行switch并触发相应的Migrate方法。

I won’t repeat the documentation here, but as an example, (m *Migration) Up() migrates the database forward through every up migration in your migrations folder, while (m *Migration) Down() does the reverse. That’s something to bear in mind if you’re coming from Rails — there’s no single change method here. Each migration must have separate up and down SQL files to apply and reverse the change, respectively.

我不会在这里重复说明文档，但是作为示例， (m *Migration) Up()通过迁移文件夹中的每个up迁移向前迁移数据库，而(m *Migration) Down()则相反。如果您来自Rails，这是要记住的-这里没有单一的change方法。每个迁移必须具有单独的向上和向下SQL文件，才能分别应用和撤消更改。

A simple up migration might look like:

一个简单的向上迁移可能类似于：

-- Create the ENUM types on which tables depend.


CREATE TYPE "looking_for" AS ENUM (
  'friendship',
  'dating',
  'relationship',
  'random_play',
  'whatever'
);


...

While the corresponding down migration would look like:

虽然相应的向下迁移看起来像：

-- Drop the ENUM types on which tables depend.


DROP TYPE IF EXISTS looking_for;


...

(If you hadn’t already guessed, I’m building an old-school Facebook clone).

(如果您还没有猜到的话，我正在构建一个老式的Facebook克隆)。

运行迁移 (Running Migrations)

We now have the code to run migrations, but typing out commands go run cmd/migrate/migrate.go steps 5 isn’t the picture of convenience.

现在我们有了运行迁移的代码，但是输入命令go run cmd/migrate/migrate.go steps 5并不是很方便。

To finish off, we’ll create a simple Makefile with convenient shorthand for our migration tasks. As a bonus, we’ll automate the vitally important task of dumping the database schema whenever we update it.

最后，我们将创建一个简单的Makefile，并为我们的迁移任务提供方便的简写。另外，每当更新数据库架构时，我们将自动完成至关重要的任务，即转储数据库架构。

db_migrate_down:
	go run cmd/migrate/migrate.go down
	@$(MAKE) db_dump_schema


db_drop:
	go run cmd/migrate/migrate.go drop


db_force_version:
	go run cmd/migrate/migrate.go force $(VERSION)


db_show_version:
	go run cmd/migrate/migrate.go version


db_migrate_steps:
	go run cmd/migrate/migrate.go steps $(STEPS)
	@$(MAKE) db_dump_schema


db_migrate_to_version:
	go run cmd/migrate/migrate.go toVersion $(VERSION)
	@$(MAKE) db_dump_schema


db_migrate_up:
	go run cmd/migrate/migrate.go up
	@$(MAKE) db_dump_schema

This is looking a little more Rails-like. The few tasks that require variables are triggered with, for example, make db_migrate_steps STEPS=3. That’s a rough edge, but good enough for my purposes.

这看起来有点像Rails。一些需要变量的任务是通过例如make db_migrate_steps STEPS=3触发的。这是一个粗糙的边缘，但对于我的目的来说已经足够了。

If we wanted to go the extra mile and make this a truly elegant migration runner, I’d explore ways to make migrate.go project-agnostic (DBMS-agnostic if we really wanted to push ourselves). Then we could install the binary and bring our migration runner with us into each new project.

如果我们想要去加倍努力，使这个真正的优雅迁移亚军，我想探索使migrate.go项目无关的(DBMS无关，如果我们真的想推动自己)。然后，我们可以安装二进制文件，并将我们的迁移运行程序带入每个新项目。

Taking the lazier approach, db_dump_schema is the last piece of the puzzle. You’ll rarely want to rely on migrations for the entire lifetime of your project, particularly if it’s going to be run in production. A year in, and you’ll have thousands of migration files cluttering up source control, tediously being applied one by one every time you rebuild a database.

采取懒惰的方法， db_dump_schema是难题的最后一部分。在项目的整个生命周期中，您几乎都不会希望依赖迁移，特别是如果要在生产环境中运行的话。一年后，您将拥有成千上万个迁移文件，这些文件杂乱了源代码管理，每次重建数据库时都会繁琐地逐一应用这些文件。

It’s cleaner to check a dump of the current database schema into source control. That way, instead of feeding step-by-step instructions into your DBMS to eventually arrive at the desired state, you can provide it with exactly what the database should look like in a single SQL file.

将当前数据库模式转储到源代码管理中更加干净。这样，您无需将逐步的说明输入DBMS最终达到所需的状态，而是可以为数据库提供在单个SQL文件中的数据库外观。

This is how Rails sets up test databases: It takes the dump of the development database and applies it in the test environment. That’s what I’ve chosen to do here with some quick additions to the Makefile:

这就是Rails设置测试数据库的方式：它提取开发数据库的转储并将其应用于测试环境。这就是我选择在此处对Makefile进行一些快速补充的方式：

# Avoid pg_dump version mismatch where multiple postgres versions are
# installed by specifying the absolute path.
PG_DUMP=/usr/local/opt/postgresql@12/bin/pg_dump
SCHEMA=./db/schema.sql
db_dump_schema:
	@if [ $(FB05_ENV) == "development" ]; then\
		echo "Dumping schema...";\
        	$(PG_DUMP) fb05_$(FB05_ENV) --file=$(SCHEMA) --schema-only;\
		echo "Schema dumped to $(SCHEMA)\n";\
	else\
		echo "Schema should only be dumped in development.";\
    	fi


db_test_prepare:
	psql --set ON_ERROR_STOP=on fb05_test < $(SCHEMA)

In db_dump_schema, we check that the current environment is development to make sure we don’t overwrite the existing dump with the structure of the test database. The @ at the start of the if block simply prevents make from echoing the command when it’s run.

在db_dump_schema ，我们检查当前环境是否正在development ，以确保我们不会用测试数据库的结构覆盖现有的转储。 if块开头的@只是阻止make在运行时回显该命令。

If the environment is development, we run Postgres’s pg_dump tool, specifying the output file in SCHEMA. Be sure to look up how to dump from your preferred DBMS if you’re not using Postgres.

如果环境是development环境，我们将运行Postgres的pg_dump工具，并在SCHEMA指定输出文件。如果您不使用Postgres，请务必查看如何从首选DBMS中转储。

Notice how each migrate action that changes the DB structure automatically calls our pg_dump recipe: @$(MAKE) db_dump_schema. We can rest easy knowing the schema we check into source control will always match the current state of the development database. (@$(MAKE) simply means “run the following task using make without echoing the command”).

请注意，每个更改数据库结构的迁移操作如何自动调用我们的pg_dump配方： @$(MAKE) db_dump_schema 。我们可以放心地知道我们签入源代码管理的模式将始终与开发数据库的当前状态匹配。 ( @$(MAKE)简单含义是“使用make不回显命令的情况下运行以下任务”)。

To quickly set up a test database, I’ve written the make task db_test_prepare, which is nothing more than the Postgres syntax for building a schema from dumped SQL.

为了快速设置测试数据库，我编写了make任务db_test_prepare ，它仅是Postgres语法，用于从转储SQL构建模式。

结论 (Conclusion)

There you have it: a super-fast build for a Rails-like migration runner for your Go projects. Find the source code here.

在那里，您可以找到：为您的Go项目提供类似于Rails的迁移运行器的超快速构建。在此处找到源代码。

The next step would be to generalise it into an installable binary with a per-project config file, giving you flexibility over the naming conventions and DBMS you choose to use. Over to you.

下一步将使用每个项目的配置文件将其通用化为可安装的二进制文件，从而使您可以灵活选择命名约定和选择使用的DBMS。交给你。