After painstakingly crafting a new change, Doomguy @jm-janzen maintainer of ratehub/rate-scrapers, is tasked with building the Docker image for the next release. Searching his zsh history, the build command can not be found. Without the knowledge from his past self, he must read through the Dockerfile to understand how to compile the docker build command.

After some important build args are removed, the crisp new image is ready to be released. JM must come up with a name that accurately reflects the way he feels about the change. The last release was 1.0.1-myhotfix so the new one, containing the “Total HTML and styling overhaul” is reasonably named v1.0.2.

He then searches the commit history to determine what notes to include with the release. Realizing that the commit messages don’t reflect the reason for the release, he must look through the files changed since the last tag and determine what is worth including. Resulting in:

  • Total HTML and styling overhaul.
  • Modernise it a bit. Make look nice(r).
  • Of course, this coincided with rate changes.

The ratehub/rate-scrapers repo is in desperate need of automation.

The Solution

There is a world where releases are done automatically, with clear version numbers, automatic changelogs, and images built consistently. But once we go down this road, there’s no going back.

Two Github Actions workflows are introduced to address these challenges, release.yml and build.yml. Along with a .releaserc config file.

.releaserc

The .releaserc file is added to the root of the repo to configure how semantic-release behaves. Here the plugins needed to analyze the commit history, create the Github Release, and generate the release notes are enabled. More importantly, the semantic-release/npm plugin (which is included by default), is excluded because the code base is Ruby, not NPM.

The tag format includes the v prefix by default (i.e., v${version}). To remove it, the tagFormat is also defined in the configuration file with the v removed.

The branches option is populated with master and main to improve portability of the workflow between repos.

{
  "branches": [
    "master",
    "main"
  ],
  "plugins": [
    "@semantic-release/commit-analyzer",
    "@semantic-release/github",
    "@semantic-release/release-notes-generator"
  ],
  "tagFormat": "${version}"
}

release.yml

On commit to the repo’s main branch, the release workflow will kickoff. Changes to the .github and charts directories are ignored to avoid new releases for workflow or Helm changes.

on:
  push:
    branches: ['master','main']
    paths-ignore:
      - '.github/**'
      - 'charts/**'

The tests are run before creating a release to serve as a quality gate.

- name: Set up Ruby.
  uses: ruby/setup-ruby@v1
  with:
    bundler-cache: true
    ruby-version: 2.7

- name: Run tests.
  run: bundle exec ruby ./test.rb

Next install semantic-release and the plugins for it. Then use npx to run semantic-release. It will determine what the next the version will be based on the commit messages since the previous release.

The GITHUB_TOKEN is set to the org secret for ratehub-machine so that it has permissions to push the tag and create the release.

- uses: actions/setup-node@v2
  with:
    node-version: 'lts/*'

- name: Create the next release.
  env:
    GITHUB_TOKEN: ${{ secrets.GH_RATEHUB_MACHINE_TOKEN }}
  run: |
    npm install -g semantic-release@18.x @semantic-release/commit-analyzer@9.x \
        @semantic-release/github@8.x @semantic-release/release-notes-generator@10.x
    npx semantic-release

Since the GITHUB_TOKEN needs to be defined in the semantic-release step, earlier in the workflow persist-credentials is disabled when checking out the repo.

- name: Checkout
  uses: actions/checkout@v3
  with:
    fetch-depth: 0
    # needs to be false, otherwise the generated GITHUB_TOKEN will interfere
    # with the global when used to write to the master (protected) branch
    # https://github.com/semantic-release/semantic-release/blob/master/docs/recipes/ci-configurations/github-actions.md#pushing-packagejson-changes-to-a-master-branch
    persist-credentials: false

Once the new version is determined, the Git tag and the Github Release, with the change notes, are pushed.

Screenshot of the `0.1.1` Github release for ratehub rate-scrapers repo with a Bug Fix for Dockerfile listed in the change notes.

The semantic-release/github plugin will then comment on any PRs that were included in the release.

Screenshot of semantic-release PR comment for the `0.1.1` release on the ratehub rate-scrapers repo.

build.yml

In the case of rate-scrapers, the build is a Docker image used to run the Ruby scripts as a CronJob in Kubernetes.

The build workflow is then triggered by the creation of the release tag.

on:
  push:
    tags: ['[0-9]+.[0-9]+.[0-9]+']

The workflow starts by getting the tag from the GITHUB_REF env var and setting it as an output so that it’s available for the with argument of the Docker action.

- name: Get tag from ref
  id: set-tag
  run: |
    echo "::set-output name=tag::${GITHUB_REF/refs\/tags\//}"

The authentication for GCR is set up using the GCP_SA_KEY_JSON org secret. Then the Docker image is built from the Dockerfile at the root of the repo. The resuling Docker image, with the release and the latest tags, is pushed to GCR.

- name: Login to GCR
  uses: docker/login-action@v2
  with:
    registry: gcr.io
    username: _json_key
    password: ${{ secrets.GCP_SA_KEY_JSON }}

- name: Build and push
  uses: docker/build-push-action@v3
  env:
    DOCKER_IMAGE: 'gcr.io/platform-235214/rate-scrapers'
  with:
    context: .
    push: true
    tags: ${{ env.DOCKER_IMAGE }}:${{ steps.set-tag.outputs.docker_tag }},${{ env.DOCKER_IMAGE }}:latest

The First Release

The rate-scrapers repo is under active development and has no previous versions. Based on the Semver 2.0.0 spec, the initial release for the repo should be 0.1.0 but when semantic-release can’t find a previous release, it will default to 1.0.0.

To overcome this, the 0.1.0 tag was created manually. This way semantic-release will start with the 0 major version. The creation of the tag also triggered the build workflow, producing the rate-scrapers:0.1.0 and rate-scrapers:latest Docker images.

Room for Improvement: Labels

Remember those important labels that were removed in the intro? Those are from the Label Schema Convention which aimed to:

  • Avoid duplication in cases where the same information is needed in multiple labels
  • Encourage the use of labels, both by image creators and by tool builders which might consume them
  • Codify good community practice in a way that is easy to consume and keep up-to-date with

It was deprecated for the mainstream OCI Image Spec, which includes back-compatibility with Label Schema. Though use of the new namespace, org.opencontainers.image, is encouraged. To avoid confusion though, in the OCI spec they are annotations, not labels. However, annotations are implemented with the LABEL Dockerfile instruction.

All of this is to say that the workflow should be improved by adding conventional labels to the images to improve compatibility with image tools and the usability of the images in large complex systems.

Call to Action 📢

The workflows implemented in ratehub/rate-scrapers are agnostic to the code base and all of the secrets used are available to every repo. So if you have a repo that could benefit from automatic releases and uses a Docker image as the build artifact, go checkout these workflows!

Do you have a pull request with workflow changes and need another set of eyes? Post it to #code-reviews to get more visibility.


This is the first post on the Ratehub Engineering blog. Consider this the PoC for using blog posts as tool to share domain knowledge across the organization. If you are interested in writing a blog post about a problem you solved, reach out to your manager/team lead and let them know! Also have a look at the pull request for this post.