• 1. A Basic Example
• 2. Code Layout Rules
  • 2.1. Multi-module library packages
• Note that a library package with multiple modules still has only one binary .lib (or .a) file.
  • 2.2. Weak Dependencies
  • 2.3. Compatibility with other code layout schemes
• 3. Installation
• 4. Usage
• 5. Semantics of CPM.JSON file
• 6. Operation
  • 6.1 Clone/Fetch
  • 6.2 Create Symlinks
  • 6.3 Build
  • 6.4 Post-build Commands
• 7. Proving Ground
• 8. Integration with GitHub actions

CPM is a tool that helps coordinate work between multiple repositories. All you need is a simple JSON file to describe dependencies between repositories. This, together with the magic of symbolic links helps maintain a consistent environment for multiple libraries.

You have two libraries cool_A and cool_B that need to be used in an application super_app. Both cool_A and cool_B use code from another library utils. Each one for these has its own Git repository.

You need to create a JSON file called cpm.json in each repository:

In super_app:

{ "name": "super_app", "git": "[email protected]:user/super_App.git",
  "depends": [
      {"name": "cool_A", "git": "[email protected]:user/cool_A.git"},
      {"name": "cool_B", "git": "[email protected]:user/cool_B.git"}
  ],
  "build" : [
      {"os": "windows", "command": "msbuild", "args": ["super_app.proj"]},
      {"os": "linux", "command": "cmake"}
  ]
}

In cool_A:

{ "name": "cool_A", "git": "[email protected]:user/cool_A.git",
  "depends": [
      {"name": "utils", "git": "[email protected]:user/utils.git"},
  ],
  "build": [
      {"os": "windows", "command": "msbuild", "args": ["cool_a.proj"]},
      {"os": "linux", "command": "cmake"}
  ]
}

In cool_B:

{ "name": "cool_B", "git": "[email protected]:user/cool_B.git",
  "depends": [
      {"name": "utils", "git": "[email protected]:user/utils.git"},
  ],
  "build": [
      {"os": "windows", "command": "msbuild", "args": ["cool_b.proj"]},
      {"os": "linux", "command": "cmake"}
  ]
}

In utils:

{ "name": "utils", "git": "[email protected]:user/utils.git",
  "build": [
      {"os": "windows", "command": "msbuild", "args": ["utils.proj"]},
      {"os": "linux", "command": "cmake"}
  ]
}

After that, you just have to fetch the super_app repository and invoke the CPM utility:

cpm super_app

It will take care of pulling all the other repositories and issuing the build commands for the current operating system.

To be able to use this magic you have to adhere to a set of basic rules:

RULE 1 - All projects have their own folder and all project folders are in one parent folder. The environment variable DEV_ROOT points to this root of development tree.

Here is a diagram showing the general code layout:

RULE 2 - Include files that need to be visible to users are placed in a subfolder of the include folder. The subfolder has the same name as the library.

Users of cool_A can refer to hdr1.h file like this:

#include <cool_A/hdr1.h>

An additional advantage of this organization is that it prevents name clashes between different libraries. In this case, if a program uses both cool_A and cool_B, the corresponding include directives will be:

#include <cool_A/hdr1.h>
#include <cool_B/hdr1.h>

Note: There is an enhancement to this rule: if the library package contains separate groups of files, they can be grouped together in modules. See below about multi-module libraries. Usually however, a library package has only one module.

RULE 3 - Include folders of dependent modules are made visible through symbolic links

In the structure shown before, the application that uses cool_A and cool_B will have an include folder but in this folder there are symbolic links to cool_A and cool_B include folders. The folder structure will look something like this (blue items in angle brackets denote symbolic links):

RULE 4 - All binary library packages reside in a lib folder at the root of development tree. Each package contains a symbolic link to this folder.

Without repeating the parts already shown of the files layout, here is the part related to lib folder (again, blue items denote symbolic links):

If there are different flavors of link libraries (debug, release, 32-bit, 64-bit) they can be accommodated as subfolders of the lib folder.

Sometimes, a library may contain more than one group of files. For instance a communication library may contain a group of functions for serial communication, another for Bluetooth communication, and so on. We call these groups of files modules^[1]. In this case, the header files can be divided in different folders, one for each module in the library:

Inside the library, the headers can be referenced as:

#include <serial/stuff.h>
#include <bluetooh/other_stuff.h>

If a dependent package wants to include only one module of the library, it can use the modules attribute in the dependency descriptor.

Example:

"depends": [
    {"name": "libcom", "modules": ["serial"], "git": "[email protected]:user/mml.git"}]

This will produce the following folder structure (again, blue denotes symbolic links):

It is OK to refer more than one module:

  "depends": [
      {
        "name": "libcom",  
        "git": "[email protected]:user/mml.git", 
        "modules": ["serial", "bluetooth"]
      }]

¹ The word module is heavily overused in the software arena; adding one more use is not going to make much difference.

Sometimes it may happen that two modules are interdependent. For instance cool_A needs a type definition that is provided by cool_B. Symbolic links can take care of this situation like shown below:

In such cases, CPM has to fetch the packages and create the symbolic links but should not initiate the build process of cool_B as part of the build process for cool_A. These situations are called weak dependencies and are flagged by the fetchOnly flag in the CPM.JSON file.

The layout required by CPM is simple and, as such, very compatible with other layout recommendations. My personal favorite is The Pitchfork Layout. Note however the following differences:

• PFL does not describe any mechanism for cooperation between different packages. The symbolic links mechanism described in this document is specific to CPM.
• PFL does not use a shared lib/ directory. The PFL libs/ folder is used for a different purpose.

CPM is written in Go. You can download a prebuilt version for Windows or Ubuntu. Alternatively, you can build it from source. To build it, you need to have the Go compiler installed. Use the following command to build the executable:

go build cpm.go

There are no other dependencies and you just have to place the CPM executable somewhere on the path.

cpm [options] [package]

or

cpm version

If package is not specified, it is assumed to be in the current directory.

Valid options are:

• -b <branch_name> switches to a specific branch
• -F discards local changes when switching branches (issues a git switch -f ... command)
• -f fetch-only (no build)
• -l local-only (no pull)
• --proto [git | https] preferred protocol for package cloning
• --root <folder> or -r <folder> set root of development tree, overriding DEV_ROOT environment variable
• --uri <uri> or -u <uri> set URI for fetching root package
• --version show program version
• -v verbose
• --help or -h show usage information

Following is a list of attributes that are recognized in the JSON file. Unknown attributes are silently ignored.

CPM reads the `CPM.JSON`` file in the selected folder and follows these steps.

For each dependent package, CPM checks if the project folder exists under the DEV_ROOT tree. If not, it issues a git clone command to bring the latest version. If you have selected a specific branch, CPM issues a git switch ... command to switch to that branch and then a git pull ... command to bring in the latest version of that branch.

If CPM has been invoked with the -l command line switch, it skips this step.

CPM creates symlink to include directories of all dependent packages and to the main lib folder. If the symlinks already exist, it verifies they point to proper target.

The next step is to build each package by issuing the build commands appropriate for the OS environment. The build attribute contains an array of commands used to build the package. Each command has the following structure:

{"os": "<windows|linux|any>", "cmd": "command name", "args": ["arg1", "arg2", ...]}

All commands that have an os attribute matching the current OS or without any os attribute are issued in order. Arguments that contain an environment variable using the syntax ${variable} or $variable will be expanded.

If CPM has been invoked with the -f command line switch, it skips this step.

Each dependency descriptor may contain an array of commands to be executed after a dependent package was built. Commands have the same structure as the build commands.

If CPM has been invoked with the -f command line switch, it skips this step.

CPM can be tested using a sample project. To use it, follow these steps:

For Windows

1. Make sure you have installed Visual Studio 2017 or higher (preferably VS2022).
2. Download CPM program and place it somewhere in the path
3. Create a devroot folder:

c:\temp>mkdir devroot
c:\temp>set DEV_ROOT=c:temp\devroot
c:\temp>cd devroot

4. Run the CPM program:

c:\temp\devroot>cpm -u [email protected]:neacsum/example_super_app.git super_app

If all goes well, you should have a file c:\temp\devroot\super_app\build\bin\x64\Debug\super_app.exe. Also the directory c:\temp\devroot\lib\x64\Debug should contain the files

• cool_A.lib
• cool_B.lib
• multi_mod.lib

For Linux

1. Make sure you have a C/C++ compiler installed. If not, sudo apt install build-essential should take care of that.
2. Download CPM program and place it somewhere in the path. This CPM program has no relation with the with the Console Password Manager.
3. Create a devroot folder:

$mkdir devroot
$export DEV_ROOT=devroot
$cd devroot

4. Run the CPM program:

$cpm -u [email protected]:neacsum/example_super_app.git super_app

CPM can be integrated with GitHub actions. You only need to fetch the CPM program and run it. Below is an example pulled from the same proving ground application:

name: Build
on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]

permissions:
  contents: read

env:
  # This is the destination directory for CPM tool
  USERPROFILE: .
  
jobs:
  build:
    runs-on: windows-latest
    
    steps:      
      - name: Get CPM
        uses: engineerd/[email protected]
        with:
          name: cpm.exe
          url: https://github.com/neacsum/cpm/releases/latest/download/cpm.exe
      
      - name: Clone
        run: git clone https://github.com/neacsum/example_super_app.git .\super_app

      - name: Build
        run: cpm -v --proto https -r . super_app
        
      - name: Run app
        run: .\super_app\build\app\x64\Debug\super_app.exe

It uses an action to fetch the CPM executable, clones the repo to be built and invokes CPM to build it.

Note that the CPM command specifies the --proto https option. Using HTTPS protocol bypasses user authentication problems.