My Personal Monorepo & Pipeline

A monorepo is a software development strategy where code for many projects is stored in the same repository. The code doesn't necessarily have to be related.

Okay, but why use a monorepo? Gathering all of my personal projects into a single repository makes it easier for me to manage and maintain the code. One repository is easier to deal with than a dozen. I can develop a common interface for building and deploying projects in the monorepo. One build tool to rule them all. Versioning is not a concern for me here, but if it were, we also get the advantage of atomic commits.

Another advantage of the monorepo to consider would be the ease of re-using code between projects. For example, you could have a docker image that acts as the base of several projects. That base image is just another project managed in the monorepo, but that other projects in the monorepo are themselves built upon. Or, maybe you have shared 3rd party dependencies that you could simply download once and the other projects would then consume. Simpler dependency management is pretty great!

So what is my monorepo's directory structure?

/bin
/build
/config
/infrastructure
/src
  /lunchboxradio
  /roylart
  /roycom
  /matomo

Let's break this down.

/bin

Contains utilities for working with the monorepo (eg: setup and teardown utilities).

/build

This contains code related to building and deploying projects in the monorepo, the common cli.

/config

Global configuration for the monorepo (eg: env files for docker-compose could go here)

/infrastructure

Code related to building and provisioning infrastructure. Think terraform and ansible.

/src

Home to the individual projects

It's great to have everything under one roof, but for me the real power in the monorepo is being able to write tooling to build, deploy, and develop each of the projects contained within. To build this system I have chosen Ruby and Rake. I like writing code with Ruby, it's very pleasant and fun, and Rake is a nice interface for writing the sort of tool I am thinking of. You could do this with any tool and language you prefer!

The glue for my monorepo is this build system. Build system is expressed as Rake tasks. Very simply it works like this: I load project data from a file called build.yml then I generate build & deploy tasks for each project from that data. There is a builder class for each type of project that has methods for building and deploying the project.

build.yml example:

all:
  startable: false
roylart:
start_by_default: true
build_by_default: true
after_build:
- “cmd to run after building”
before_build:
- “cmd to run before building”
build_type: capistrano
tags:
- wordpress
roycom:
build_type: middleman
lbr:
build_type: middleman
matomo:
build_type: capistrano

The generated rake tasks look like:

$ rake -T
rake buildsystem:build                      # Build everything
rake buildsystem:deploy                     # Deploy everything
rake buildsystem:lbr:build                  # Build the lbr project
rake buildsystem:lbr:deploy                 # Deploy the lbr project
rake buildsystem:matomo:build               # Build the matomo project
rake buildsystem:matomo:deploy              # Deploy the matomo project
rake buildsystem:roycom:build               # Build the roycom project
rake buildsystem:roycom:deploy              # Deploy the roycom project
rake buildsystem:roylart:build              # Build the roylart project
rake buildsystem:roylart:deploy             # Deploy the roylart project

Adding a new project is easy: add the src files, add the project to build.yml, incldue a builder class if it does not already exist.

Now to build and deploy I simply run:

rake buildsystem:roycom:build buildsystem:roycom:deploy

I run a more complex version of this on a more important monorepo that builds docker images so there are additional tasks that tag and push images to a private docker repository with support for multiple docker clusters and environments. But I could also expand this to tar up a build and "tag" it with a version number, then "push" to an s3 bucket, where then deploy is to pull that tar onto some servers someplace. Really, it's a super flexible system that allows me to do anything I want.

The builder class looks something like:

module BuildSystem
  class Capistrano < Base
    def build
      BuildSystem.logger.info "Bundle Install #{@params[:service_name]}"
      BuildSystem.system! "cd src/#{@params[:service_name]}; bundle install"
    end
def deploy
  BuildSystem.logger.info "Deploying #{@params[:service_name]}"
  BuildSystem.system! "cd src/#{@params[:service_name]}; #{@build_args.join(' ') if @build_args} bundle exec cap #{BuildSystem.branch} deploy"
end

end
end

With all of this in place it's now trivial to hook up to something like CircleCI, or GitHub actions, or Gitlab to build and deploy when you commit your changes, it's just running the commands you have designed.

So that's my monorepo in a nutshell. I think this is a super flexible system and a pretty decent way to manage a bunch of projects.