Speed up your CI pipeline by up to 75% with layer caching

June 30, 2022

Introduction

We streamlined our build and deploy process with Gitlab’s CI pipeline. It took care of the usually tedious process of waiting for a Docker build. (passing in environment variables, pushing to the registry, etc) This freed up our hands to continue development as these were working in the background.

As we added more code and packages, we experienced build times growing longer. We grew hesitant to make frequent commits and merges. There was a fear of burning through our CI minutes or queueing up too many overlapping jobs.

Here’s how we cut a 10+ minute build to 2 minutes. This saved time and allowed us to ship more and deliver value faster.

TL;DR: Organizing our Dockerfiles saved hours of build time and productivity!

Order Matters

Don’t overlook the importance of the steps in your Dockerfile. Docker is smart so we don’t have to be. But making use of it requires a little bit of planning.

Docker can to reuse information from previous builds in the form of a cached layer. So if you haven’t touched you package.json, then there’s no need to rebuild the npm i step, right?

This cut out a huge part of redundant rebuilding from our pipeline and saved the bulk of the time. If you notice your CI jobs take a significant amount of time, then consider which layers you could cached.

The Node packages laeyr is only rebuilt when package.json file changes. This is the same technique from the reducing Docker size blog which led to a reduction of 50% to 70% build time on average.

Future Improvement

It’s quick enough as is but we will never say no to further optimization. It just may take a bit of research to figure out whether other stages can be smarter about being rebuilt. Or maybe how we can optimize the build step itself so it’ll be fast without the cache.

Recap

We started with 10+ minute Docker builds with a bulk of that time coming from installing packages. This made us hesitant to make changes with the concern of overlapping, redundant, and long-span CI jobs. It also made it very slow and difficult to deploy updates and bug fixes for each environment.

  • On average, we have reduced the build time by 75% and deployment time by 90%. Saving hours of time spent waiting for a change to go out to our environments.
  • We have been able to make smaller changes more frequently. This led to incremental, consistent, and safer improvements to our app. We also gained finer-grained control over commits and changes in case of rollbacks.