From time to time, I catch myself trying to over-engineer a solution to a simple problem. It’s nothing new, and this seems to happen to everyone on occasion.

Recently, I had a request raised from one of the developers that I work with to run additional Karma tests on Pull Requests made to a specific branch (lets call it $branch) in one of our repositories. Karma is a JavaScript testing framework. These tests would only need to run and report a pass or a fail. The intent was to keep these tests completely separate from existing build pipelines and deployments.

Finding the Right CI Tool For the Job

Originally, I had started thinking about doing this via a Jenkins job. While working on this project there were some challenges that I faced with Jenkins. At first, I wanted to see if I could use GitHub Webhooks as a trigger for Jenkins and script everything else out. I had created something that functionally worked, but the comments on the Pull Requests weren’t quite working how I had hoped. I figured there are plugins that can accomplish this, so why reinvent the wheel?

I had already looked into using the GitHub Pull Request Builder plugin. Initially, I had passed on this because there is a notification that this plugin can be insecure (which is why I had tried building something myself first.) Taking a second look, the newest version had addressed and resolved this issue. I gave it a shot. It ended up actually being much more challenging to get it working than building something from scratch myself. The documentation was pretty good, but I couldn’t get it to work for our use case.

“Perhaps I’m overthinking this,” I thought to myself. Having used a variety of different CI tools in the past, this is a simple task compared to some other jobs that I’ve created. Jenkins is sort of overkill for what I was trying to accomplish. I recognized that I was trying to over-engineer a solution to this problem and started looking elsewhere. I’ve been happy with GitLab CI in the past and hadn’t had a chance to work with GitHub Actions yet. Being curious, I figured why not give it a shot?

GitHub Actions fit the scope of the problem perfectly. I had noticed we had allocated build minutes included in our GitHub plan that we have been paying for but haven’t used. I was able to get a working solution up and running in a matter of minutes. The configuration is simple and straight forward.

For this project all I needed was to install the node packages via npm and run the Karma tests, nothing fancy. The yaml file was easy to write and is equally easy to follow as a reader.

name: Karma Tests

# Run on Pull Request to $branch only
on:
  pull_request:
    branches: [ $branch ]

# Job Setup
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2 #Checkout Repo
    - uses: actions/setup-node@v1 #Setup Node
      with:
        node-version: '12'
    - name: Run Karma Tests
      run: |
        npm install
        npm run test        

The jobs have been running much faster than I was expecting as an added bonus. The tests themselves take very little time. If you are familiar with GitLab CI or other CI tools, you won’t feel out of place using GitHub Actions.

Additional Bonuses of Using GitHub Actions

If you have something like a typical VM running Jenkins, you’ll know that you have to keep up on things like updates, installing packages, etc. This has its own set of risks. If you run updates on packages that are installed or if you need to do something like upgrade the kernel on the host, you run the risk of unexpected downtime or having some sort of conflict/issue with packages (rare, but still possible). Something like multiple versions of npm can also be installed on the host requiring nvm to be used to specify the correct version of npm to be used during a build, or when using the CLI. This may get confusing for someone who doesn’t access the server frequently or for someone new to the environment.

GitHub Actions seems to solve this quite well. Rather than dealing with installed software and dependencies you simply specify what you need the GitHub Actions runner to have via the config and GitHub does the rest.

Additionally, you have the headache of security with Jenkins. If you are using GitHub Webhooks as a trigger for Jenkins, you have to make sure your Jenkins server is publicly accessible. Sure, you can solve this by setting whitelists for the GitHub Webhook IP Addresses, but it is still additional provisioning that needs to be done that you may or may not think you can handle.

With GitHub Actions you don’t need to do any whitelisting or additional provisioning outside of storing secrets (if needed, and cannot be viewed) from the GitHub repo.

Another big plus is cost. If you are already paying for GitHub Teams or Enterprise, you likely already have build minutes available that you aren’t using.

Comparatively, Jenkins itself is free, but you are likely paying a cloud provider for the instance/storage or your own equipment in a datacenter, data transfer for the traffic in and out, and probably someone to manage it.

Finally, a minor bonus comes if you already have the GitHub App installed to something like a Slack channel. It works perfectly out of the box, notifying you and your team if a test passes or fails without any additional configuration.

Closing Out My Initial First Impressions

I really liked working with GitHub Actions for this small task that was requested. It was very simple to setup and gets rid of a lot of the additional headaches of something like hosting your own Jenkins server.

For an environment that requires a massive amount of build minutes, GitHub Actions could potentially get quite expensive. While it was fun and easy to work with, I am not sure that I would feel comfortable quite yet with running something like a full pipeline without doing some additional testing. It does appear that you can do a majority of what Jenkins can do with GitHub Actions. Having not made something that complicated with it yet, I am still a bit hesitant to give it my stamp of approval. That doesn’t mean I think it’s bad, I actually really like it a lot. I just haven’t built anything mission critical with it yet. For me Jenkins is sort of tried and true and has really good plugins and community behind it.

I would really like to continue using GitHub Actions to run tests and perhaps down the road we can start using this tool for more complicated and important builds.