See: https://docs.ansible.com/ansible/latest/user_guide/playbooks_filters.html#network-cli-filters

1, Query the actual router VRP database and compare the content to that of the Routinator.
2. Repeat the comparison after the data has changed in the Routinator. Does the router pick up the changes and does its database match the Routinator data?

This probably requires that I know the serial number of the Routinator data and the router data.

For the Routinator getting the serial number depends on NLnetLabs/routinator#73, though it might be possible to get the serial number from the Routinator logs.

For the router, at least Alcatel/Lucent/Nokia SROS 16.0.R6 exposes the serial via the console:

A:vSIM>show router origin-validation rpki-session detail
...
Serial ID          : 53149              Session ID         : 0

Use environment no more (otherwise the interface prompts for the next page) and show router origin-validation database to get the actual VRPs.

I'm not sure if I can get it via NETCONF or if doing so would be any more standard across routers than using the proprietary console.

Routinator 0.4+ now deploys but the playbook logic for obtaining the most recent Routinator serial number no longer works:

fatal: [routinator]: FAILED! => {"msg": "Unexpected templating type 
error occurred on ({{ result.stderr_lines | select('match', '^New serial is ([0-9]+)')
| list | last | regex_search('([0-9]+)', '\\\\1') | first }}): expected string or 
bytes-like object"}

E.g. sequence diagrams, network/deployment topology, visualize the relationship between the various tools and technologies, etc.

Default Ansible progress output is coloured and intended for humans. For integration with Travis CI it probably makes more sense to use a computer readable format such as Ansible JSON output.

Currently the router username and password are exposed in the Gantry output in the public Travis CI build logs. This wasn't thought to be an issue because the router is only temporarily deployed. However, if the same username and password are used for each build then the credentials from a previous build can be observed and used to connect to a newly deployed router in a subsequent build.

I think that the new support for parallel deployment of Routinator and routers can lead to a situation where the test checks the Routinator log for a serial number announcement before it exists in the log. This is I assume why Travis CI build #34 failed. The test should wait for the Routinator to announce the first serial number.

When running tests against multiple routers, and later potentially multiple RPKI caches and even extending to use publication servers such as NLnet Labs Krill, what should happen if a single deployment or a test against a single component fails? Should tests be run if one or more components fail to deploy?

Relates to #22.

See: https://travis-ci.com/NLnetLabs/gantry/builds/138725509#L1746

The cli Bash script has grown beyond the intended scope and is suffering from lack of proper types, e.g get_deployment_report returns a pipe separated string instead of an array of object, and should be test-driven/unit-tested. My suggestion would be to re-write it in Python in a test-driven manner.

Could do this using https://python-rtrlib.readthedocs.io/en/latest/.

This would be much quicker to deploy and minimally validate Routinator.

Fetching VRPs from Routinator via /json and the serial number from /status could result in the received ROAs being for a different serial number than was scraped from /status.

It might be better to use the RTR protocol instead as that communicates the serial number with the VRPs and notifies the client when new data is ready. See RFC-8210.

One way to use the RTR protocol could be to use Python RTRLib.

The current version was a first incomplete version with a TO DO placeholder inside it.

13:52 $ ./gantry deploy routinator vr-csr:16.09.02
...
<SNIP>
...
[vr-csr:16.09.02] PLAY RECAP *********************************************************************
[vr-csr:16.09.02] vr-csr-16.09.02            : ok=10   changed=3    unreachable=0    failed=0   

14:10 $ ./gantry docker vr-csr:16.09.02 ps
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS                    PORTS               NAMES
232245a3d41b        docker-reg.do.nlnetlabs.nl/vrnetlab/vr-csr:16.09.02   "/launch.py --userna…"   14 minutes ago      Up 14 minutes (healthy)                       router

14:10 $ ./gantry ssh vr-csr:16.09.02
2019-05-10 12:10:53 +0000: You are about to be connected to the proprietary interface of router vr-csr:16.09.02.
2019-05-10 12:10:53 +0000: When prompted enter password: ********
2019-05-10 12:10:53 +0000: Do you wish to proceed? [y/N] y

<HANGS>
<CTRL+C>
^C2019-05-10 12:11:14 +0000: Aborting..

14:11 $ ./gantry logs vr-csr:16.09.02
...
<SNIP>
...
2019-05-10 11:59:49,190: vrnetlab   DEBUG    writing to serial console: end
2019-05-10 11:59:49,191: vrnetlab   TRACE    waiting for '#' on serial console
2019-05-10 11:59:49,237: vrnetlab   TRACE    read from serial console: transport input all
            ^
% Invalid input detected at '^' marker.

csr1000v#
2019-05-10 11:59:49,237: vrnetlab   DEBUG    writing to serial console: copy running-config startup-config
2019-05-10 11:59:49,237: vrnetlab   DEBUG    writing to serial console: 
2019-05-10 11:59:49,238: launch     INFO     Startup complete in: 0:03:36.458781

Complete Cisco log attached: vr-csr:16.09.02.log

The router image was built and configured by vrnetlab, see:

• https://github.com/plajjan/vrnetlab/tree/master/csr
• https://github.com/plajjan/vrnetlab/blob/master/csr/docker/launch.py

Under https://rpki.readthedocs.io/en/latest/rpki/resources.html#supporting-tools.
And add docs on how to setup and use Gantry as its own independent section under “RPKI Tools” in the left hand menu.

E.g. after build #31 failed the Digital Ocean droplets were left running and the log shows no sign of the after_script undeploy step being invoked...

Transient Docker Machine failure when running ./gantry deploy routinator with the latest Gantry Docker image.

[routinator] Installing Docker...
[routinator] The default lines below are for a sh/bash shell, you can specify the shell you're using, with the --shell flag.
[routinator] 
[routinator] [routinator] Error creating machine: Error running provisioning: error installing docker: 

This problem cannot be fixed in-place by re-issuing'./gantry deploy routinator as that results in:

[routinator] [routinator] Error checking TLS connection: Error checking and/or regenerating the certs: There was an error validating certificates for host "188.166.36.130:2376": dial tcp 188.166.36.130:2376: connect: connection refused

The only solution with Gantry at present is undeploy followed by redeploy.

1. The issue should be worked around transparently by Gantry.2
2. Re-deployment should be able to proceed from the current status, not fail due to a previous error.
3. Undeployment should not be necessary as this incurs additional cloud costs to re-create the underlying VM, even though in this case the underlying VM is fine.

The issue happened again on a second attempt, but not on the third attempt.

Router tests will need to be added, e.g. as in #4, but ideally not written separately per router and not added to the base router post sanity check tests in the existing Ansible templates. How best to go about this? How portable are origin validation settings and outputs across routers? Would NETCONF help or are they bespoke extensions to NETCONF?

See: https://travis-ci.com/NLnetLabs/gantry/jobs/261734055#L1824

See: https://docs.ansible.com/ansible/latest/network/user_guide/faq.html#why-do-the-config-modules-always-return-changed-true-with-abbreviated-commands

Build #31.1 failed with this issue:

[vr-sros:16.0.R6] TASK [SHOW RPKI DATABASE SUMMARY] **********************************************
[vr-sros:16.0.R6] Tuesday 07 May 2019  14:27:52 +0000 (0:00:00.079)       0:06:46.407 *********** 
[vr-sros:16.0.R6] fatal: [vr-sros-16.0.R6]: FAILED! => {"changed": false, "msg": "Connection type ssh is not valid for this module"}

Error creating machine: Error checking the host: Error checking and/or regenerating the certs: There was an error validating certificates for host "178.62.211.191:2376": remote error: tls: bad certificate.

E.g. see build #32 line 1755.

I did not see this issue when testing locally,.

Due to vrnetlab/vrnetlab#183.

To get the official version of the Docker Machine module.

From build #35:

The command "./gantry deploy routinator vr-sros:16.0.R6 vr-vmx:18.2R1.9" exited with 1.

$ ./gantry --data-dir tests test all

There is no point running the tests if deployment fails.

Current status: I have the 18.2R1.9 image but have not built or deployed it.

See: https://docs.ansible.com/ansible/latest/network/user_guide/network_best_practices_2.5.html

Current status: I have the 16.09.02 image and it deploys healthily (according to Docker) but I can't connect to it.

Currently the Routinator playbook and the SROS router playbook have very similar logic for informing Ansible of Docker Machine hosts. This should be extracted and generalized. Alternatively it could be replaced by a pre-Ansible step that generates the Ansible inventory based on Docker Machine output, e.g. using something like https://github.com/nathanleclaire/dockerfiles/blob/master/ansible/machine.py. A dynamic inventory plugin might be the best way to go as it's the Ansible way.

Current status: I have the 6.5.1 image but have not built or deployed it.