Reduce. Reuse. Rehacktor.

One way to run rails migrations on GCP app engine


Running migrations during the release process for any reasonably complicated Rails application is usually a bit of a challenge. There are various strategies for writing migrations such that they don't require any downtime. But I'm more concerned here with the actual mechanics of how do you actually run the migrations. Whether they require downtime or not.

In our current setup we manually run migrations when necessary after deploying code for our new versions because that is the earliest opportunity to do so. This means there's a period where our new code is running without the schema changes needed to support it. Rails in a production mode will inspect the database and generate the various ActiveRecord getters and setters for you on app boot and keep those cached for the lifetime of the server process. We overcome this by rebooting the processes immediately after the migrations are complete. The process becomes: Deploy new version --> Run Migrations --> Reboot everything. And, again, this has worked ok so far.

The trouble is, at app engine, the "Reboot everything" step is a bit tricky. I think we can fake it a little bit by writing a script that will SSH into each instance and call docker restart on the gaeapp container. But we haven't actually tried that yet. One big downside is that it will leave all of our instances in debug mode. Which GCP calls out as somehow bad though they are a bit ambiguous on why that is.

In some evolved world, we would be able to run migrations before deploying the new version and avoid the restarts all together. Today it looks like we pulled that off.

To start with, the deployment process is broken into two distinct phases. The first one runs each time we make a new commit to source control. It outputs a new app engine build of our application but does not deploy it to anywhere that it impacts our users.

The second phase is manually triggered. It deploys all of our "real" app engine services using the --image-url option pointed at the image we built in phase one. But just before that happens, we run this step to run our database migrations:

- name:
  - 'c'
  - |
    /buildsteps/ \
      -i$PROJECT_ID/appengine/<service name>.<service version> \
      -- bundle exec rake db:migrate
  entrypoint: '/bin/bash'

Getting to this point involved several false starts and dead ends so here's a penny for the pennyjar of helpful tech blog posts. Hopefully it will save someone some time.