Skip to content

Blog

Adding a goose provider

Introduction

In this post, we'll explore the new Provider feature recently added to the core goose package. If you're new to goose, it's a tool for handling database migrations, available as a standalone CLI tool and a package that can be used in Go applications.

Requires version v3.16.0 and above.

Adding a provider to your application is easy, here's a quick example:

provider, err := goose.NewProvider(
  goose.DialectPostgres, // (1)!
  db, // (2)!
  os.DirFS("migrations"), // (3)!
)

results, err := provider.Up(ctx) // (4)!

  1. The first argument is the dialect. It is the type of database technology you're using. In this case, we're using Postgres. But goose also supports:

    clickhouse, mssql, mysql, postgres, redshift, sqlite3, tidb, vertica, ydb,

  2. The second argument is the database connection. You can use any database driver you want, as long as it implements the database/sql interface.

    A popular choice for Postgres is pgx/v5

  3. The last argument may be nil. Why? Because goose also supports the ability to register Go functions as migrations.

    However, in most cases, you'll be using SQL migrations and reading them from disk. In this case, you'll use os.DirFS or embed them into your binary and use embed.FS.

  4. The last step is to invoke one of the migration methods. In this case, we're running the Up method, which will run all the migrations that haven't been run yet. Here's a list of all the methods:

      (p *Provider) ApplyVersion
  (p *Provider) Close
  (p *Provider) Down
  (p *Provider) DownTo
  (p *Provider) GetDBVersion
  (p *Provider) ListSources
  (p *Provider) Ping
  (p *Provider) Status
  (p *Provider) Up
  (p *Provider) UpByOne
  (p *Provider) UpTo

SQL migration files and goose annotations

In this post we'll explore SQL migration files and +goose annotation comments, which are used to parse SQL statements and optionally modify how migrations are executed.

As of this writing there are five annotations:

-- +goose Up
-- +goose Down
-- +goose StatementBegin
-- +goose StatementEnd
-- +goose NO TRANSACTION

Improving ClickHouse support

ClickHouse is a an open-source column-oriented database that is well-suited for analytical workloads. Over the past few years we've seen more and more demand for improved ClickHouse support in goose.

To summarize:

The /v2 driver changed the DSN format, so be prepared for a breaking change. This is actually a good thing, because it brings the format in-line with other databases.

A tour of goose up and down commands

A while ago a co-op student, who happened to be a visual leaner, asked if it were possible to explain goose commands visually. At the time we were still at an office, so we gathered around the whiteboard and doodled some diagrams.

This post captures some of those whiteboard sketches, which seemed to help.

Better tests with containers

Managing state is hard. Managing database state is even harder. And coordinating state within a test suite is just always a bad time.

But it doesn't have to be this way!

There is a fantastic Go package called ory/dockertest that allows you to spin up ephemeral docker containers. It'll work both locally (assuming you have Docker installed) and in your Continuous Integration (CI) pipelines.

Embedding migrations

Embedding migrations

Go continues to be boring while sprinkling quality of life features. One of the recent additions was the ability to embed files at compile time. Click here for go1.16 release notes.

Sine many users compile goose themselves, this new embed feature paves the way for embedding SQL files directly into the goose binary. This was already possible with existing tools, however, now that embedding is part of the standard library it's never been easier to offer this feature.