Build utility for monorepos

Monorepo Build Tool (mbt) is a utility that supports differential builds, dependency tracking and metadata management for monorepos stored in git.

In the context of mbt, term 'Module' is used to refer to a part of source tree that can be developed, built and released independently. Modules can be built with different programming languages and their native build tools.

For example, a repository could have .NET modules built with msbuild and NodeJS modules built with npm scripts. Module developers should be able to use the build tools native to their tool-chain.

Each module in a repository is stored in its own directory with a spec file called .mbt.yml. Presence of the spec file indicates mbt that the directory contains module and how to build it.

.mbt.yml file must be in yaml and has the following structure.

name: module-a      # name of the module
build:              # list of build commands to execute in each platform
  darwin:
    cmd: npm        # build command
    args: [run, build]    # optional list of arguments to be passed when invoking the build command
  linux:
    cmd: npm
    args: [run, build]
  windows:
    cmd: npm
    args: [run, build]

In the preceding spec file, we are basically telling mbt that, module-a can be built by running npm run build on OSX, Linux and Windows. With this committed to the repository, we can start using mbt cli to build in several ways as shown below.

# Build the current branch 
mbt build branch --in .

# Build a specific branch
mbt build branch feature --in .

# Build only the modules changed in a specific branch relative to another
mbt build pr --src feature --dst master --in .

# Build only the modules changed between two commits
mbt build diff --from <commit-sha> --to <commit-sha> --in .

Currently, all mbt commands interrogates the git repository for source files required by the operation. Therefore, side effects of un-committed changes will not be visible.

Sometimes a change to a module could require the build of the modules that depend on it. We can define these dependencies in .mbt.yml files and mbt takes care of triggering the build in the topological order.

For example, module-a could define a dependency on module-b, so that any time module-b is changed, build command for module-a is also executed.

name: module-a 
dependencies: [module-b]

One example of where this could be useful is shared libraries. A shared library could be developed independently of its consumers. However, all consumers gets automatically built whenever the shared library is modified.

Another situation could be where one module is a single page web app and the other module is the back-end API and the host application of that. It's a prevalent model to develop such modules in different programming languages and usually host app build contains the steps to pack the web app into the deployment package. Using the dependency feature, host module could depend on the client app module so that anytime the web app changes the host module is also built.

When mbt executes the build command specified for a module, it sets up several important attributes as environment variables. These variables can be used by the actual build tools in the process.

In addition to the variables listed above, module properties (arbitrary list of key/value pairs described below) is also populated in the form of MBT_MODULE_PROPERTY_XXX where XXX denotes the key.

One useful scenario of these variables would be, a build command that produces a docker image. In that case, we could tag it with MBT_MODULE_VERSION so that the image produced as of a particular Git commit SHA can be identified accurately. (We will also discuss how this information can be used to automatically produce deployment artifacts later in this document)

When working in a dense, modular codebase it is sometimes important to assess the impact of your changes to the overall system. Each mbt build command variation has a mbt describe counterpart to see what modules are going to to be built. For example, to list modules affected between two git branches, we could run:

mbt describe pr --src <source-branch-name> --dst <destination-branch-name> --in .

Furthermore, you can specify --json flag to get the output of describe formatted in json so that it can be easily parsed and used by tools in the rest of your CD pipeline.

mbt produces a data structure based on the information presented in .mbt.yml files. That is what basically drives the build behind scenes. Since it contains information about all modules in a repository, it could also be useful for producing other assets such as deployment scripts. This can be achieved with mbt apply command. It gives us the ability to apply module information discovered by mbt to a go template stored within the repository.

As a simple but rather useless example of this can be illustrated with a template as shown below.

{{ $module := range .Modules }}
{{ $module.Name }}
{{ end }}

With this template committed into repo, we could run mbt apply in several useful ways to produce the list of module names in the repository.

# Apply the state of master branch
mbt apply branch --to <path to the template> --in <path to repo>

# Apply the state of another branch
mbt apply branch <branch-name> --to <path to the template> --in <path to repo>

# Apply the state as of a particular commit
mbt apply commit <git-commit-sha> --to <path to the template> --in <path to repo>

Output of above commands written to stdout by default but can be directed to a file using --out argument.

It's hard to imagine useful template transformation scenarios with just the basic information about modules. To make it little bit more flexible, we add a couple user-defined properties the data structure used in template transformation. First one of them is called, .Env. This is a map of key/value pairs containing arbitrary environment variables provisioned for mbt command.

For example, running mbt command with an environment variable as shown below would make the key TARGET available with value PRODUCTION in .Env property.

TARGET=PRODUCTION mbt apply branch --in .

Second property is available in each module and can be specified in .mbt.yml file.

name: module-a
properties:
  a: foo
  b: bar 

module-a shown in above .mbt.yml snippet would make properties a and b available to templates via .Properties property as illustrated below.

{{ $module := range .Modules }}
{{ $module.Properties["a"] }} {{ $module.Properties["b"] }}
{{ end }}

More realistic example of this capability is demonstrated in this example repo. It generates docker images for two web applications hosted in nginx, pushes them to specified docker registry and generates a Kubernetes deployment manifest using mbt apply command.

Complete documentation

curl -L -o /usr/local/bin/mbt [get the url for your target from the links below]
chmod +x /usr/local/bin/mbt

OS	Download
darwin x86_64
linux x86_64
windows

This blog post covers some initial thinking behind the tool.

mbt is powered by these awesome libraries

• git2go
• libgit2
• yaml
• cobra
• logrus

Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY