NSO in Docker is a project created by Cisco to enable users of NSO to easily run NSO in Docker.

This repository contains all you need to build Docker images out of Cisco NSO. It produces two Docker images;

• a production image
  • stripped of documentation and similar to make it small
  • use this as a base image in the Dockerfile for your production container image
    • add your own packages on top
• a development image
  • contains Java compiler and other useful tools
  • can be used directly to compile packages and similar

The development image can be used immediately, for example as the image for a CI docker container runner to use for running CI jobs that involve compilation of NSO packages and similar. The production image is intended to be used as a base image on which you add your own packages, like NEDs and your service packages, to produce a final image for your environment.

There are many reasons for why Docker and containers in general might be good for you. The main drivers for using Docker with NSO lies around packaging, ensuring consistency in testing and production as well as making it simple and convenient to create test environments.

• build a docker image out of a specific version of NSO and your packages
  • distributed as one unit!
  • you test the combination of NSO version X and version Y of your packages
    • think of it as a “version set”
  • the same version set that is tested in CI is deployed in production
    • guarantees you tested same thing you deploy
  • conversely, using other distribution methods, you increase the risk of testing one thing and ending up deploying something else - i.e. you didn’t really test what you use in production
• having NSO in a container makes it easy to start
  • simple to test
  • simple to run in CI
  • simple to use for development
• benefits of NSO in Docker ecosystem
  • easy to run netsim or virtual routers for testing
  • creating testing and development environments (testenv) that are shareable
    • it is key for an efficient development team to have organized and shareable environments for development
• you do NOT need Kubernetes, Docker swarm or other fancy orchestration
  • run Docker engine on a single machine

It’s also worth noting that using Docker does not mean you have to replace all of your current infrastructure. If you are currently using OpenStack or some system that primarily deals with virtual machines you don’t have to rip this out. On the particular VM that runs NSO you can simply install Docker and have a single machine Docker environment that runs NSO. You are using Docker for the packaging features!

Yet another alternative is to use Docker for development and CI and when it’s time to deploy to production you use something entirely different. Docker images are glorified tar files so it is possible to extract the relevant files from them and deploy by other means.

NSO in Docker is not just two Docker container images but rather an ecosystem - the NID (Nso In Docker) ecosystem, which is about a way of working and approaching problems. The NID ecosystem defines the concept of common development and test environments. It uses the base NSO images as the foundation to allow building:

• NED repository
  • including running the NED as a netsim
  • having a standardized test and development environment
• package repository
  • service or other package repository
  • including testing of the repository
• NSO system repository
  • a repository encompassing an entire NSO system
  • use NSO service packages
  • include NEDs and other packages built on other repositories
  • write tests in a simple way
  • having a standardized test and development environment
• common for all NID skeletons
  • easily implement tests for
  • test across multiple NSO versions
  • share the definition of a development environment among colleagues
    • provide a simple to use and familiar starting point and ergonomics
    • drastically shorten time from branching feature branch to writing and testing first lines of code
    • leverage same topology for development and testing

See the NID skeletons for how to get started developing in the NID ecosystem.

NSO in Docker runs on:

• Linux
• Mac OS X, see Mac OS X support for more information
• Windows, see Windows support for more information

To build these images, you need:

• Docker
• Make
• realpath

Install with:

• Debian: apt install coreutils make
• Mac OS X: brew install coreutils
• see https://docs.docker.com/get-docker/ for installation instructions for Docker

If you want to run the test suite you also need:

• expect
• sshpass

The ideal scenario would be to ship prebuilt Docker images containing NSO but as legal requirements prevent that, this is the second best option. This repository contains recipes that you can use to produce Docker images yourself. Just add water Cisco NSO ;)

• Clone this repository to your local machine
  • git clone https://gitlab.com/nso-developer/nso-docker.git
• Download Cisco NSO
  • go to https://developer.cisco.com/docs/nso/#!getting-nso/getting-nso and click the “NSO 5.x Linux” link to download NSO
• If the file ends with .signed.bin, it is a self-extracting archive that verifies a signature, execute it to produce the installer
  • for example running bash nso-5.3.linux.x86_64.signed.bin will produce a number of files, among them the install nso-5.3.linux.x86_64.installer.bin
• Place the nso-5.x.linux.x86_64.installer.bin file in nso-install-files/ in this repository
• run make in repository root directory, which will build Docker images out of all the NSO install files found
  • NOTE: running docker commands, which are invoked by make, typically require root privileges or membership in the docker group
  • this runs make build-all which will build images for all found NSO versions
  • use NSO_VERSION=5.3 make build to build for a specific version
• verify your new images are built with docker images which should look something like the following
  • NOTE: the docker images are tagged with a suffix
    • the suffix will be your username, for example cisco-nso-base:5.3-kll if your username is kll
    • the suffix is to avoid overwriting a version tag, like cisco-nso-base:5.3, before the image has been tested and determined to be a good build
    • run make tag-release to also add a docker tag without the suffix, like cisco-nso-base:5.3

docker images

#+RESULTS[3ff05549238600a48eb05c5a59c1d17b6d78738b]:

REPOSITORY                   TAG                 IMAGE ID            CREATED             SIZE
kll-test-cisco-nso-5.3-kll   latest              999b88b099ed        16 hours ago        550MB
<none>                       <none>              14806a997e24        16 hours ago        1.15GB
cisco-nso-base               5.3-kll             8ed0cb9decad        16 hours ago        550MB
<none>                       <none>              1c332a6ffb25        16 hours ago        505MB
cisco-nso-dev                5.3-kll             d94c42ccd65f        16 hours ago        1.15GB
debian                       buster              b5d2d9b1597b        11 days ago         114MB

Run make tag-release and provide the version to tag using the variable NSO_VERSION:

make NSO_VERSION=5.3 tag-release
docker images

#+RESULTS[e4db0280029c988c9d279cab68425484f77831c0]:

docker tag cisco-nso-dev:5.3-kll cisco-nso-dev:5.3
docker tag cisco-nso-base:5.3-kll cisco-nso-base:5.3
REPOSITORY                   TAG                 IMAGE ID            CREATED             SIZE
kll-test-cisco-nso-5.3-kll   latest              999b88b099ed        16 hours ago        550MB
<none>                       <none>              14806a997e24        16 hours ago        1.15GB
cisco-nso-base               5.3                 8ed0cb9decad        16 hours ago        550MB
cisco-nso-base               5.3-kll             8ed0cb9decad        16 hours ago        550MB
<none>                       <none>              1c332a6ffb25        16 hours ago        505MB
cisco-nso-dev                5.3                 d94c42ccd65f        16 hours ago        1.15GB
cisco-nso-dev                5.3-kll             d94c42ccd65f        16 hours ago        1.15GB
debian                       buster              b5d2d9b1597b        11 days ago         114MB

• Clone this repository to your local machine
  • git clone https://gitlab.com/nso-developer/nso-docker.git
• Download Cisco NSO
  • go to https://developer.cisco.com/docs/nso/#!getting-nso/getting-nso and click the “NSO 5.x Linux” link to download NSO
• If the file ends with .signed.bin, it is a self-extracting archive that verifies a signature, execute it to produce the installer
  • for example running bash nso-5.3.linux.x86_64.signed.bin will produce a number of files, among them the install nso-5.3.linux.x86_64.installer.bin
• Place the nso-5.x.linux.x86_64.installer.bin file in nso-install-files/ in this repository
• commit file(s) in nso-install-files/ using git LFS and push
  • git add nso-install-files/*
  • ~git commit nso-install-files -m “Add NSO install files”~
    • it is a good practice to add the files one by one and write the version you added in the commit message, like Add NSO install file for v4.7.5
  • git push -u origin master
  • CI will now build the docker images for you
    • naturally provided you first setup CI
• verify your new images are built by going to the container repository in Gitlab viewing the list of container images
  • the docker tag for built images consists of the NSO version number and the CI pipeline id, for example cisco-nso-base:5.3-7583729 for NSO version 5.3 and pipeline id 7583729
  • CI builds on the master branch will in addition be tagged with just the NSO version, that is cisco-nso-base:5.3, after passing tests

The above method involves committing the NSO install files to this git repository (your clone of it). This means the repository must be private so that you don’t leak the NSO install files nor the produced Docker images. There are a number of reasons for why this setup might not be ideal;

• you have an open source public repo and wish to run CI publicly
• LFS doesn’t work with your choice of code hosting
• NSO install files are too big or you just don’t like LFS

There is an alternative. The path in which the build process looks for the NSO install file(s) is specified by NSO_INSTALL_FILES_DIR. The default value is nso-install-files/, i.e. a directory relative to the root of the repository. The standard way of delivering the NSO install files, as outlined in the process above, is to place the NSO files in that directory. The alternative is to change the NSO_INSTALL_FILES_DIR variable. Note how you can set this environment variable through the GitLab CI settings page under variables. You do not need to commit anything. In case you are running Gitlab CI with the docker runner, add the path to the list of volumes, for example:

[[runners]]
  name = "my-runner"
  url = "https://gitlab.com/"
  token = "s3cr3t"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "debian:buster"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock", "/data/nso-install-files:/nso-install-files"]
    shm_size = 0

The path /data/nso-install-files on the host machine becomes available as /nso-install-files/ in the CI build docker containers and by specifying that path (/nso-install-files) using the CI variable settings, the job will now pick up the NSO images from there. This is how the public repo at https://gitlab.com/nso-developer/nso-docker works. It allows us to host all code in public, run CI tests in public yet not reveal the NSO install file as required by its EULA.

• if you built a production image, i.e. using base image from this repo and adding in your own packages
• run a standalone container
• no persistent volume - since we are doing testing we don’t need to survive a restart
• use docker networking - connect to other things running in docker, like netsim etc

docker run -itd --name nso-dev1 my-prod-image:12345

• mount the source code directory into the container
• makes it possible to use compiler etc in the container
• avoid installing compilers and other tools directly on your computer

docker run -itd --name nso-dev1 -v $(pwd):/src cisco-nso-dev:5.2

• with a production image, i.e. using the base image from this repo and adding in your own packages
• use shared volume to persist data across restarts
  • CDB (NSO database)
  • SSH & SSL keys
  • NETCONF notification replay
  • rollbacks
  • backups
  • optionally NSO logs
    • if remote (syslog) logging is used there is little need to persist logs
    • if local logging, then persisting logs is a good idea
• possibly use –net=host to share IP address with host machine
  • makes it easier to handle connectivity

This uses the --net=host option to let the container live in the hosts networking namespace. This means that it binds to the IP address of the (virtual) machine it is running on. NSO is configured to expose the CLI over SSH on port 22. If you have SSH running on the VM, there will be a collision when using --net=host. To avoid port collision you can reconfigure NSO to listen on a different port by setting the SSH_PORT environment variable. Also note that we use a shared volume for logs. /log inside the container contains the logs and you can access them outside the container in /data/nso-logs.

docker run -itd --name nso -v /data/nso:/nso -v /data/nso-logs:/log --net=host -e SSH_PORT=2024 my-prod-image:12345

The ncs.conf configuration file for NSO is written at run time of the container. It is generated (modified to be precise) at startup of the container. It is possible to influence the configuration through a number of variables. These can be fed into the container through environment variables.

The Docker images produced by this repo per default carry a unique tag based on the CI_JOB_ID variable set by Gitlab CI, for example registry.gitlab.com/nso-developer/nso-docker/cisco-nso-dev:31337 where 31337 is the value from CI_JOB_ID.

In addition, if the job is built on the master branch, it will also receive a tag based on the NSO version it contains, for example if the previously mentioned image is based on NSO 5.2.1, if it was built from the master branch it would also get the tag registry.gitlab.com/nso-developer/nso-docker/cisco-nso-dev:5.2.1. This makes it possible for other repositories to use the 5.2.1 tag to always refer to the latest build of 5.2.1.

Do note that the example image URLs used above would be the result of the default configuration for the official origin repository for the nso-docker project. However, as the official repo CI builds happen in a public environment, the resulting images can’t be pushed as it would effectively publish this is per the default configuration and although the example URL follows that for the official origin repo for the nso-docker project.

It is recommended to use a nightly job to produce new images every night that include the latest security patches and similar to the base images. Do note however that this also means that updates to packages will happen and that could have negative consequences if they are not fully backwards compatible. These images are based on Debian stable but for example, pylint has been known to include additional lints in newer version and so new version of the image could include change like this which lead to unintended results.

For a truly deterministic environment, downstream repositories that rely on these Docker images should be based on the unique tag and consequently be updated with the same cadence as new images are built.

It is possible to reconfigure the port that SSH uses by setting the SSH_PORT variable to the wanted value.

An admin user can be created on startup by the run script in the container. There are three environment variables that control the addition of an admin user;

• ADMIN_USERNAME: username of the admin user to add, default is admin
• ADMIN_PASSWORD: password of the admin user to add
• ADMIN_SSHKEY: private SSH key of the admin user to add

As ADMIN_USERNAME already has a default value, only ADMIN_PASSWORD or ADMIN_SSHKEY need to be set in order to create an admin user. For example:

docker run -itd --name nso -e ADMIN_PASSWORD=foobar my-prod-image:12345

This can be very useful when starting up a container in CI for testing or when doing development. It is typically not required in a production environment where there is a permanent CDB that already contains the required user accounts.

Also note how this only adds a user. If you are using a permanent volume for CDB etc and start the NSO container multiple times with different ADMIN_PASSWORD then the last run will effectively overwrite the older password. However, if you change ADMIN_USERNAME between invocations then you will create multiple users! An admin user account created during the last run of NSO will not be removed just because ADMIN_USERNAME is set to a different value.

These docker images default to using python3.

In NSO v5.3 and later, the python VM to use is probed by first looking for python3, if not found python2 will be tried and finally it will fall back to running python. In earlier versions of NSO, python is executed, which on most systems means python2. As python2 is soon end of life, these docker images default to using python3.

NOTE: SSH keys and SSL certificates are not included in backups produced by ncs-backup. Backup and restore largely behaves as it normally does with ncs-backup as run outside of Docker, with some exceptions.

Normally, the ncs-backup script includes the NCS_CONFIG_DIR (defaults to /etc/ncs). SSH keys and SSL certificates are normally placed in /etc/ncs/ssh and /etc/ncs/ssl respectively. This means that the SSH keys and SSL certificates are part of the produced backup file. This is NOT the case for when NSO is run in a container as SSH keys and SSL certificates are not in the default configuration path.

To take a backup, simply run ncs-backup. The backup file will be written to /nso/run/backups.

To restore a backup, NSO must not be running. As you likely only have access to the ncs-backup tool and the volume containing CDB and other run time state from inside of the NSO container, this poses a slight challenge. Additionally, shutting down NSO will terminate the NSO container.

What you need to do is shut down the NSO container and start a new one with the same persistent shared volume mounted but with a different command. Instead of running the /run-ncs.sh which is the normal command of the NSO container, you should run something that keeps the container alive but doesn’t start NSO, for example read DUMMY (it’s a bash builtin command so still have to run bash). A full docker command could look like:

docker run -itd --name nso -v /data/nso:/nso -v /data/nso-logs:/log --net=host my-prod-image:12345 bash -lc 'read DUMMY'

You now have the NSO container running but without NSO itself. Get a shell in the container with

docker exec -it nso bash -l

Then run the ncs-backup restore command, for example:

ncs-backup restore /nso/run/backups/ncs-4.7.5@2019-10-07T14:41:02.backup.gz

Or if you want to automate the whole process slightly you could do it all using docker exec and non-interactively:

docker exec -it nso bash -lc 'ncs-backup restore /nso/run/backups/ncs-4.7.5@2019-10-07T14:41:02.backup.gz --non-interactively'

Restoring a NSO backup should move the current run directory (/nso/run to /nso/run.old) and restore the run directory from the backup to the main run directory (/nso/run). After this is done, shut down your temporary container and start the normal NSO container again as usual.

NSO looks for the SSH host key in the directory /nso/ssh. The filename differs based on the configured host key algorithm. NSO in Docker will use the RSA algorithm for host keys.

If no SSH host key exists, one will be generated. As it is stored in /nso which is typically a persistent shared volume in production setups, it will remain the same across restarts or upgrades of NSO.

NSO version 5.3 and newer supports ed25519 and will in fact default to using ed25519 as server host key on new installations but this behavior is suppressed for NSO in Docker and instead RSA is used as it is supported by all currently existing versions of NSO.

NSO expects to find a TLS certificate and key at /nso/ssl/cert/host.cert and /nso/ssl/cert/host.key respectively. Since the /nso path is usually on persistent shared volume for production setups, the certificate remains the same across restarts or upgrades.

When no certificate is present, one will be generated. It is a self-signed certificate valid for 30 days making it possible to use both in development and staging environments. It is not meant for production. You should replace it with a proper signed certificate for production and it is encouraged to do so even for test and staging environments. Simply generate one and place at the provided path, for example using the following, which is the command used to generate the temporary self-signed certificate:

openssl req -new -newkey rsa:4096 -x509 -sha256 -days 30 -nodes \
        -out /nso/ssl/cert/host.cert -keyout /nso/ssl/cert/host.key \
        -subj "/C=SE/ST=NA/L=/O=NSO/OU=WebUI/CN=Mr. Self-Signed"

As the produced Docker image contains both NSO itself and a given version of all included packages, any changes to said components will result in a new Docker image. Deploying any change, however small, means building and deploying a new Docker image. Upgrading and downgrading of NSO itself, with the packages kept static, is also based on deploying another Docker image.

The database in NSO, called CDB, is using YANG models as the schema for the database. It is only possible to store data in CDB according to the YANG models that define the schema.

If the YANG models are changed, in particular if nodes are removed or renamed (rename is basically a removal of one leaf and an addition of another), any data in CDB for those leaves will be removed. NSO normally warns about this when you attempt to load the new packages, for example request packages reload will refuse to reload the packages if nodes in the YANG model have disappeared. You would have to add the force argument, e.g. request packages reload force.

NSO in Docker will automatically reload packages on startup, using the --with-packages-reload-force argument to ncs on startup. This means that destructive model changes will be accepted without warning. It is expected that NSO in Docker is developed in an environment where there are other safe guards, such as CI testing, to catch accidental destructive model changes.

The major new feature in NSO version 5 is what’s known as Common Data Models or CDM, which is based on the YANG schema-mount standard (RFC8528). With it, there are changes to the CDB database files on disk. The migration from a CDB written by NSO version 4 to NSO version 5 happens automatically but first the old CDB written by NSO version 4 must be compacted, which is a manual step. However, with NSO in Docker, the startup script takes care of this for you by automatically determining at startup if NSO version 5 is being started on a CDB written by NSO version 4. If this is the case, the CDB on disk is compacted.

NSO 5 requires that packages, in particular NEDs, be compiled for CDM. Thus, upgrading to NSO 5 typically also involves upgrading one or more NEDs. In the process of changing NEDs and upgrading NSO there is the risk of inadvertently making model changes that lead to data loss, in which case the upgrade process needs to be reattempted. The overall upgrade process is something along the lines of:

• take backup of CDB (in NSO 4 format)
• compact CDB
• take backup of CDB (in NSO 5 format)
• start NSO 5
  • verify data integrity
  • if model / data inconsistencies have lead to data loss
    • restore from backup that contains NSO 5 compacted CDB
    • rectify packages
    • start NSO 5 with new packages
    • repeat until done

Multiple attempts might be necessary to get everything to load and upgrade correctly. CDB compaction can take some time (depending on the size of CDB). By restoring from a backup of a compacted CDB, we avoid having to compact CDB for every retry.

In a production setting with a structured approach to development and operations, the recommendation would be to take a backup of CDB from production and move to a development machine where the above steps can be executed. Preferably also incorporating not just the NED / package changes into CI but also including testing of the CDB upgrade. The upgrade is thus tested in development & CI before being attempted on the production deployment machines. While we might use a compacted CDB to speed up the development and testing of the upgrade, as outlined above, the actual upgrade of the production system will only happen once inside of an NSO container in an unsupervised fashion, which is why startup script of NSO in Docker will automatically determine the CDB version + NSO version and, if deemed necessary, perform CDB compaction.

The ncs.conf used in the docker images produced by this repository is not checked into git but rather originates from the NSO install itself. This means that if we build a Docker image based on NSO 5.2 we will get the default ncs.conf that comes with 5.2. Any updates to the ncs.conf shipped with NSO will find its way into the Docker image. Since modifications are necessary, partly for NSO to fit into a Docker environment but also to apply instance specific configuration, like the administrator password, the ncs.conf configuration file is modified on startup. This is achieved by a startup script executed before NSO is started.

The standard Docker run script (run-nso.sh) looks for files that ends with .sh in /etc/ncs/pre-ncs-start.d/ and /etc/ncs/post-ncs-start.d/ and will run any scripts found before or after starting NSO. This facility is used to modify the ncs.conf configuration file before NSO is started. /etc/ncs/pre-ncs-start.d/50-mangle-config.sh performs the necessary modifications. Since ncs.conf is a structured XML document, it primarily uses xmlstarlet to perform modification operations on the configuration file.

You can further modify the ncs.conf configuration file by adding your own startup script in /etc/ncs.pre-ncs-start.d/. Since the configuration file is an XML document, modification is best done through an XML aware tool. The previously mentioned standard mangle-config.sh script modifies the ncs.conf configuration file using xmlstarlet which understands XML and offers XML aware editing capabilities.

There are multiple approaches to extending the functionality of the NSO docker image.

When NSO starts up with no pre-existing CDB, it will load the files placed in /nid/cdb-default/ in the container image. Simple place an XML file in /nid/cdb-default/ to have its content loaded on first startup.

The standard Docker run script (run-nso.sh) looks for files that ends with .sh in /etc/ncs/pre-ncs-start.d/ and /etc/ncs/post-ncs-start.d/ and will run any scripts found before or after starting NSO. ncs --wait-started is used to wait for NSO to start. If you want to modify the configuration file, produce some XML files to be read into CDB on startup or similar, you can write a script for that and place it in the relevant startup directory (typically before NSO is started).

In other situations you want to run scripts that load or modify some configuration in NSO (CDB) somehow, which might be better suited to be placed in /etc/ncs/post-ncs-start.d (though don’t mistake these capabilities for what CDB upgrade logic and similar offers). For example, it is possible to start another process in the same container and if that process is dependent upon NSO having started, placing the script in /etc/ncs/post-ncs-start-d/ is a convenient approach as those scripts are only started after NSO have started up (as determined by ncs --wait-started).

The production-base image comes with a basic Docker healthcheck. It is using ncs_cmd to get the phase that NCS is currently in. Only the result status, i.e. if ncs_cmd was able to communicate with the ncs process or not, is actually observed. This tells us whether the ncs process is responding to IPC requests.

As far as monitoring NSO goes, this is a very basic check. Just a tad above the basic process check, i.e. that the ncs process is actually alive, which is the most basic premise of production-base image.

More advanced and deeper looking healthchecks could be conceived, for example by observing locks and measuring the time a certain lock has been held, but it is difficult to find a completely generic set of conditions for flagging NSO as healthy or unhealthy based on that. For example, if a transaction lock has been held for 5 hours, is that healthy or not? In most situations, that would be an abnormally long transaction, but does it constitute an unhealthy state? In certain operational environments it could be normal with that long transactions (for example a batch import of some data). Marking the container as unhealthy and potentially restarting it as a consequence would only make things worse.

We really want to measure some form of progress, even if that progress is just internal. A five hours transaction is fine as long as we are continuously making progress. However, there are currently no such indicators available and so the healthcheck observes the rather basic operation of the IPC listener.

There are multiple make targets for building an NSO docker image.

Assuming the NSO install file has been placed in the NSO_INSTALL_FILES_DIR (per default nso-install-files/), you can run:

make NSO_VERSION=5.2.1 build

To produce a docker image based on NSO 5.2.1. It requires that the corresponding installer file is present, i.e. nso-install-files/nso-5.2.1.linux.x86_64.installer.bin.

You can use the build target to build a Docker image out of an NSO installer. It requires that you specify the complete path to the NSO installer file, for example:

make FILE=/home/foo/nso-docker/nso-install-files/nso-5.2.1.linux.x86_64.installer.bin build-file

To build docker images for all the NSO installer files present in the NSO installer directory, (specified by NSO_INSTALL_FILES_DIR), you can run:

make build-all

There are targets to run tests that correspond with the above;

• test-version
• test
• test-all

They require the same variables to be set as their corresponding build target described above.

NOTE: Using a Gitlab CI runner as described in this section has different security implications than what is normally associated with using containers for CI. See the Security sub-heading.

In order to build the CI pipeline as defined for this repository you need GitLab and a GitLab CI runner. It is possible to use the free and public gitlab.com in order to host the code but you have to provide your own Gitlab CI runner. While you have access to CI runners simply by using gitlab.com to host your code, their capabilities don’t match what is needed in order to build this project. Fortunately, Gitlab as a product makes it very simple to connect your own CI runner to any Gitlab instance, including the public gitlab.com one.

1. Get a VM or a physical machine to run your CI runner.
2. Install Debian on said machine.
3. Follow the guide on https://docs.gitlab.com/runner/install/linux-repository.html to install the Gitlab CI runner on your machine
4. Follow the guide at https://docs.gitlab.com/runner/register/ on how to register your runner with Gitlab
5. Expose the docker control socket in the gitlab runner configuration

Here’s a configuration file for gitlab ci runner. Note the volumes setting which includes /var/run/docker.sock - this exposes the Docker control socket to the containers run by the CI runner which enables the containers to start sibling containers.

[[runners]]
  name = "my-runner"
  url = "https://gitlab.com/"
  token = "s3cr3t"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "debian:buster"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
    shm_size = 0

You naturally need to use your token and not literally s3cr3t. The token is written when you do the runner registration per the guide referenced above.

Note that exposing the Docker control socket has security implications. Containers as run by the CI runner normally provide isolation such that CI jobs are contained within the container and are unable to access anything outside of the container. By exposing the docker control socket, the CI jobs can start new containers, including starting a privileged one, which means it has root access on the host machine and enables escaping the container entirely. Do not grant access to your project or CI runner to anyone you do not trust. For example, someone that is able to create a branch on your repository can write a Gitlab CI configuration file that instructs the CI runner to run a privileged container and then gain access to the CI runner machine itself.

The versions of NSO to build and test for will vary per environment. To handle this, the concept of “version sets” are used. A list of NSO versions is used to compute a number of CI configuration files that can be included from the main CI configuration (.gitlab-ci.yml) and different lists can be used for different environments.

For this repository in its online form at https://gitlab.com/nso-developer/nso-docker/, all currently supported versions of NSO are tested. This is useful to ensure that nso-docker itself is compatible with a wide range of NSO versions but also as other repositories in the NSO in Docker ecosystem can be checked against the same range of versions.

Ideally, a typical user will test against two versions of NSO;

• the current version used in production
• the latest version of NSO, as the potential target to move to

In practice, it is common to have a few versions, for example:

• 5.1.2 (current version)
• 5.1.4 (latest maintenance release in 5.1 train)
  • this is a smaller step than going to a newer train, like 5.2 or 5.3
• 5.3.1 (latest NSO version)

New NSO versions are released periodically and over time the list of NSO versions grows fairly long. For someone writing a NSO package, keeping the list of NSO versions to build and test for in CI up to date can be a time consuming task, in particular if there are many packages to be maintained. GitLab CI allows for inclusion of configuration files such that the project CI configuration can include another file that is external to the repo. By using this feature we can keep a central list of the NSO versions to test with across multiple repositories.

version-sets/supported-nso/versions.json is the root definition of the currently supported versions. version-sets/version-get is a Python script that is run from version-sets/supported-nso/Makefile and which uses version-sets/supported-nso/versions.json as input and produces a number of YAML files in version-sets/supported-nso/ that can be included in other repositories.

For example, we have build-all.yaml, which uses the standard CI job definition called build and defines jobs for all currently supported versions of NSO.

build-all4.yaml is similar but only includes NSO 4.x versions, whereas build-all5.yaml does the same for NSO 5.x. Since NSO 5 looks quite different with schema-mount, it could be reasonable for some packages to only target NSO 5.

build-tot.yaml only includes the “tip” of each train, where a train is the combination of a major and minor version number. Patch releases are not considered for tip-of-train as they are not supposed to be used by the wide masses. For example, if we have 4.7, 4.7.1, 4.7.2 and 4.7.2.1 as well as 5.2.1, the tip-of-train would include 4.7.2 and 5.2.1. Similarly, there’s also build-tot4.yaml and build-tot5.yaml for tip of train for NSO 4 or NSO 5 respectively.

To include a file, use the include directive in your .gitlab-ci.yml, for example:

include:
  - project: 'nso-developer/nso-docker'
    ref: master
    file: '/version-sets/supported-nso/build-tot5.yaml'

This will work for any repository hosted on the same GitLab instance as the nso-developer/nso-docker repo. Once you clone the nso-docker repository to your own environment, as you are encouraged to do, you are likely to place it in another namespace (not nso-developer) and so you must update the include statements for the dependent repositories accordingly. For example, if you place your clone of nso-docker in the foobar group on your gitlab.example.com instance, then the version-set referenced should be foobar/nso-docker. You can also reference a version-set on a remote Gitlab instance by specifying a complete URL instead, like https://gitlab.com/nso-developer/nso-docker/-/raw/master/version-sets/supported-nso/build-tot.yaml.

Merely copy an existing version set, modify the versions.json file and regenerate the files. For example;

cp -av version-sets/supported-nso version-sets/my-versions
cd version-sets/my-versions
vi versions.json # edit the file to list the NSO versions you want
make generate

Include it in your nso-docker build or build of other packages in the NSO in Docker ecosystem.

You are encouraged to mirror any components in the NSO-in-Docker (NID) ecosystem that you use.

While you can rely on binaries built upstream, including them in your NSO system means a build time risk as broken Internet connectivity or similar could mean you cannot download the packages you depend on. If you need to quickly rebuild your system to integrate a small hot fix, such a risk could mean you cannot deploy a new version. Mirroring the git source repositories of your dependencies not only mean you get to build them locally but also allows you to make minor (or major) modifications to the source. It could be to update the .gitlab-ci.yml file to add a build for a different NSO version or a minor patch to a NED. Mirroring was kept in mind while designing NID ecosystem.

We think it is important to keep a copy of your dependencies locally (in your own Gitlab instance) such that you can build it yourself if necessary. We also think it is important to keep dependencies up to date - in fact, we would like to encourage to “live-at-head”, i.e. follow and include the latest version of a dependency. This is why continuous mirroring of an upstream repository makes sense. However, you should not blindly accept new versions into your main NSO system build as it can break your downstream builds. A gating function is needed and we propose a explicit version pinning workflow to provide for that gating function.

While NSO in Docker isn’t specifically built for Gitlab (the intention is to make it more general than that), it is currently well suited to be hosted in Gitlab since the accompanying CI configuration file is for Gitlab CI. Gitlab features a mirroring functionality that can either push or pull in changes from a remote repository. For example, this functionality is used on this repository to keep it in sync (through pushing) with https://github.com/nso-developer/nso-docker/. You can use GitLab mirroring to continuously mirror this repository, however, it comes with a major constraint; only fast-forward merging is possible. This essentially prevents you from making even the most minute changes to the repository as continued mirroring will break. While you are encouraged to upstream any patches or changes you might have for this repository and others in the NID world, there are times when you want to make changes, for example if you need to apply a particular CI runner tag or limit the versions of NSO that you build for. To cater to such scenarios, an alternative mirror mechanism is provided: The CI configuration of this repository and the repo skeletons, are capable of mirroring itself from an upstream through a special CI job.

Enable mirroring from an upstream by scheduling a CI job and setting the CI_MODE variable to mirror. You create a CI schedule by going to CI / CD -> Schedules in Gitlab. In addition, you need to set a number of other variables for the mirroring functionality to work:

• CI_MODE: CI_MODE must be set to mirror which will skip running any of the normal build and test jobs and instead only run the mirror job
• GITLAB_HOSTKEY: the public hostkey(s) of the GitLab server
  • run ssh-keyscan URL-OF-YOUR-GITLAB-SERVER to get suitable output to include in the variable value
• GIT_SSH_PRIV_KEY: a private SSH key to use for cloning of its own repository and pushing the updates
  • create a deploy key that has write privileges
    • generate a key locally ssh-keygen -t ed25519 -f my-nso-docker-mirror
    • in GitLab for your repository, go to Settings -> CI / CD -> Deploy keys
    • create a new key, paste in the public part from what you generated
      • Check Write access allowed
  • enter the private key in the GIT_SSH_PRIV_KEY variable
• MIRROR_REMOTE: the URL of the upstream repository that you wish to mirror
  • for example, to mirror the authoritative repo for nso-docker, use https://gitlab.com/nso-developer/nso-docker.git
• MIRROR_PULL_MODE: can be set to rebase to do git pull --rebase instead of a normal git pull

Set CI_MODE=mirror in the CI schedule (since this should only apply for that job and not the normal CI jobs). Use the repo wide CI variable section to set at least GITLAB_HOSTKEY and GIT_SSH_PRIV_KEY, possibly MIRROR_REMOTE too (or set from CI schedule). These are multi-line values and it appears some GitLab versions cannot correctly set multi-line values in the CI schedule, instead using repo wide CI variables effectively works around this issue.

The mirroring functionality is quite simple. It will run git clone to get a copy of its own repository (which is why it needs SSH host keys and deploy keys), then add the upstream repository as a HTTP mirror (presuming it is a public repository and does not require any credentials). It will then pull in changes, allowing merge conflicts, and finally push the result to its own repository, thus functionally achieving a mirror. It uses the user name and email of the user who initiated the CI build as the git commit author (for merge commits).

A merge will be performed by the mirroring if necessary (when fast-forward isn’t possible). As only automatic conflict resolution is possible, it is important to write changes in such a way that we reduce the likelihood of conflicts arising in the first place.

For example, it is often easier to make small adjustments to a file. If we want to modify the CI configuration we can place then bulk of our addition in a new file, for example my-ci-config.yml and include this from the .gitlab-ci.yml through an include statement, like so:

include:
  - '/version-sets/supported-nso/nso-docker.yaml'
  - '/my-ci-config.yml'

Note how we are merely appending to the already existing include statement. It is a YAML dict and adding a new include: line would effectively overwrite the old one.

If you get a merge conflict, you will need to resolve it manually. Do this by cloning your repository, then adding the upstream repo as a git remote and pulling in from that:

git clone [email protected]:my-group/nso-docker.git
cd nso-docker
git remote add upstream https://gitlab.com/nso-developer/nso-docker.git
git pull upstream master

During the pull, if automatic merging is not possible, the merge will abort and give you the opportunity to sort out the conflicts. Do the needful and finally push back the result to your repo:

git push origin master

Contributions are welcome, however before you start writing code, please open an issue to discuss your idea or bug fix to make sure your ideas or intended solution align with the goals or ideals of the project.

New functionality should be covered by new test cases that proves the new functionality works.

The typical workflow for submitting code involves forking this git repository, creating a branch and committing some code which will then be tested in CI. However, this project has a specialized CI runner that carries the NSO install files required to successfully build this project and this CI runner is only available for the origin repository, i.e. gitlab.com/nso-developer/nso-docker. A branch on your own private fork of this repository will not have access to the CI runner and thus will not be able to successfully execute the CI tests.

In order to run the tests, a maintainer will need to do a coarse review of the changes to verify there is no hostile code, after which your private branch can be copied to the nso-developer/nso-docker repository, which then allows it to be tested with the specialized CI runner. A shadow MR can then be setup to merge the commits to master. The commits still maintain the author, preserving credit for the changes.

The intention is for Mac OS X to be a generally supported platform for NSO in Docker. Docker on Mac is using a Linux VM to run the Docker engine and as such, it is compatible with normal Docker images built for Linux. You don’t need to recompile your NSO in Docker images when moving between a Linux machine Docker on Mac as they are both really running Docker on Linux.

NSO in Docker has been primarily developed on Linux. Continued development and testing happens on Linux first but as OS X is a popular platform and the technical prerequisites are there, it should be supported and an effort is being made to make it compliant.

What works:

• building the NSO in Docker images cisco-nso-base and cisco-nso-dev
• using the various NID skeletons to build packages and run test environments

What doesn’t work:

• running the test suite of nso-docker itself
  • it relies on direct connectivity to the containers which isn’t provided by Docker on Mac
  • unless you are actually modifying this repo, you are unlikely to need to run the test suite

To build, make sure you have realpath installed, which comes with coreutils that you can install for example using brew install coreutils, in case you are using brew.

If you notice any issues, please open an issue.

Running NSO in Docker on Windows is supported, with some caveats. With a recent version of Windows 10 (May 2020 - version 2004), Docker is using a lightweight Linux VM using WSL2 (Windows Subsystem for Linux v2). As such, it is compatible with normal Docker images built for Linux. You don’t need to recompile your NSO in Docker images when moving between a Linux machine and Docker in WSL2 as they are both really running Docker on Linux.

Prerequisites:

• Windows 10, version 2004: https://docs.microsoft.com/en-us/windows/whats-new/whats-new-windows-10-version-2004
• WSL2 installed: https://docs.microsoft.com/en-us/windows/wsl/install-win10
• Windows Terminal is recommeneded: https://www.microsoft.com/en-us/p/windows-terminal/9n0dx20hk701

After installing the prerequisites, there are two methods of deploying Docker. Depending on the method, certain networking related aspects of NSO in Docker may not work:

1. (Easy) Install Docker Desktop and enable WSL2 integration: https://docs.docker.com/docker-for-windows/wsl/. The Docker engine runs as a separate lightweight WSL2 VM, next to your preferred Linux distro. After enabling WSL2 integration with your preferred distro in Docker Desktop settings, docker commands are made available without requiring changes to your Linux distro. As a consequence of Docker running in a separate VM, direct network access to containers from Windows and Linux is not possible, only through port mapping: https://docs.docker.com/docker-for-windows/networking. Other features (volumes, isolated networks, …) are not affected.
2. (A bit more work) Install Docker engine directly in your preferred WSL2 distro: https://docs.docker.com/engine/install/ubuntu/. This method has feature parity with a “bare” Linux installation, so all aspects of NSO in Docker are expected to work. Since there is no systemd in WSL2, Docker engine must be started manually the first time WSL2 starts up.

Note that picking one of the methods does not prevent you from switching to the other (and back). If Docker Desktop is installed, enabling WSL2 distro integration will configure your distro at runtime to prefer the engine provided by Docker Desktop. This can be disabled, allowing your distro to revert to using its own Docker engine.

What works:

• everything* (except when using Docker Desktop, where there is no direct network access)

To build, you need make and Docker executable available in your preferred WSL2 distro. To test, there are some additional dependencies (same as Linux).

If you notice any issues, please open an issue.

A: The larger the final container image is, the less impact the base image size typically has. Picking a 5MB or 50MB base image is not crucial when the final image is an order of magnitude larger.

Debian was chosen as it is a well working proven distribution with a long track record. It is supported by a considerably sized community.

minideb, which is a minimal build of a debian base image, was not only considered but actually used in early phases of this repository. It does provide a smaller image. Measured at the time of the switch from minideb to stock debian, the difference was about 10%. minideb weighed in at 471MB while debian:buster came in at 525MB. The proven track record of Debian ultimately made it the winner.

A: A delightful question with a less than delightful answer! It is a combination of multiple factors:

• we want to be able to run
  • docker run -it cisco-nso-dev:5.3 to get interactive shell
  • docker run -it cisco-nso-dev:5.3 echo foo to echo foo from within the container
  • docker run -it cisco-nso-dev:5.3 ncs_cli to get the NSO CLI
• sh, the Bourne shell, has a hard coded PATH
• ncs is not installed in PATH of sh
• we don’t want to modify the ncs install
  • likely error prone, in particular over time
• we can modify PATH of sh by configuring our profile
• sh only reads profile when started as interactive shell
• Docker runs sh as non-interactive shell
  • thus sh does not read profile

We solve this by effectively replacing Dockers standard use of sh by specifying our own entrypoint. It remains to be seen whether this is a good idea or a wildly bad one. Don’t hesitate to open an issue in case you have an issue. It is however tested (see the test-dev-entrypoint test case) including some more exotic scenarios.