MT-bulk is a toolset to help manage multiple Mikrotik/RouterOS devices by sending in parallel predefined or custom commands to multiple devices at once using Mikrotik SSL API, SSH and SFTP.

Version 2.x introduces a major breaking changes, please read Version 2 breaking changes section of this readme before upgrading.

MT-bulk toolset contains two tools:

CLI tool that process devices list and commands provided by command line arguments, loaded from file or external SQL database. Commands are distributed to several internal workers and processed parallel.

Simple REST API server that process HTTPS POST requests with specified pair of commands and hosts to asynchronously execute on.

• Command's options
• Download
• List of possible operations/modes
• Configuration description
• Quick Overview / Tutorial
• Troubleshooting

MT-bulk.

Usage:
  mt-bulk gen-api-certs [options]
  mt-bulk gen-ssh-keys [options]
  mt-bulk init-secure-api [options] [<hosts>...]
  mt-bulk init-publickey-ssh [options] [<hosts>...]
  mt-bulk change-password (--new=<newpass>) [--user=<login>] [options] [<hosts>...]
  mt-bulk system-backup (--name=<name>) (--backup-store=<backups>) [options] [<hosts>...]
  mt-bulk security-audit [options] [<hosts>...]
  mt-bulk sftp <source> <target> [options] [<hosts>...]

  mt-bulk custom-api [--commands-file=<commands>] [options] [<hosts>...]
  mt-bulk custom-ssh [--commands-file=<commands>] [options] [<hosts>...]
  mt-bulk -h | --help
  mt-bulk --version

Options:
  -C <config-file>         Use configuration file, e.g. keys/certs locations, ports, commands sequences, custom commands, etc...
  --source-db              Load hosts using database configured by -C <config-file>
  --source-file=<file-in>  Load hosts from file <file-in>

  <hosts>...               List of space separated hosts in format IP[:PORT]

MT-bulk REST API gateway.

Usage:
  mt-bulk-rest-gw [options]
  mt-bulk-rest-gw gen-https-certs [options]
  mt-bulk-rest-gw -h | --help
  mt-bulk-rest-gw --version

Options:
  -C <config-file>         Use configuration file

Current and historical releases of mt-bulk and mt-bulk-rest-api at https://github.com/migotom/mt-bulk/releases

List of possible operations to execute by CLI and REST API:

• Generate Mikrotik API SSL certificate
• Generate SSH RSA Private/Public keys
• Initialize device to use Mikrotik SSL API
• Initialize device to use Public key SSH authentication
• Change user's password
• System backup
• SFTP
• Scan for CVEs and security audit
• Execute sequence of custom commands

MT-bulk supports two formats of configuration files:

• TOML format (https://github.com/toml-lang/toml)
• YAML format (https://yaml.org/spec/)

Default configuration format since version 2.x is YAML.

• Application defaults
• System (/etc/mt-bulk/config.yml, /Library/Application Support/MT-bulk/config.yml)
• Home (~/.mt-bulk.yml, ~/Library/Application Support/MT-bulk/config.yml)
• Command line -C option

Host can be specified using:

• simple text format:

  • ip
  • ip:port
  • foo.bar.com:port

• using YAML:

host:
  - ip: "192.168.1.1"
    password: "secret"
  - ip: "192.168.1.2:22"
    user: "john"
    password: "secret"

• CSV export from The Dude:

  Hosts can be specified in CSV format as exported by The Dude.

  The Dude does not export username or password so these must be specified in the main configuration file.

  1. Open The Dude client and connect to your server.
  2. Select the List tab on the Devices pane.
  3. Click the "CSV" button to export all devices. All rows that are not "RouterOS" devices are ignored when the CSV file is read.

  Expected CSV format:

  Flag,Name,Addresses,MAC,Type,Maps,Services Down,Notes
  up,192.168.88.1,192.168.88.1,"7E:4D:28:00:00:00, 7E:4D:28:00:00:01, 7E:4D:28:00:00:04",RouterOS,Local,,
  up,192.168.88.104,192.168.88.104,BE:4E:26:00:00:00,Some Device,Local,,
  

  Note that in the above example, only 192.168.88.1 will be considered valid as it is the only row with the Type of "RouterOS". Rows of any other type will be ignored and not connected to.

• MT-bulk command line tool
• MT-bulk REST API

Detailed examples and sample configurations at examples/ directory of project.

mt-bulk gen-api-certs -C examples/configurations/mt-bulk.example.yml

Create new CA, host and device certificates.

mt-bulk init-secure-api -C examples/configurations/mt-bulk.example.yml 192.168.1.2 192.168.1.3:222 192.168.1.4:6654

Initialize 192.168.1.2, 192.168.1.3 and 192.168.1.4 (in this example each device has running SSH on different port) with user, SSL API and certificates pointed by mtbulk.cfg (section [service.clients.mikrotik_api.keys_store]).

mt-bulk change-password -w 16 -C examples/configurations/mt-bulk.example.yml --new=supersecret --user=admin --source-db

Change admin's password to supersecret for admin user on devices selected by SQL query using 16 workers. Connection details and query pointed by mtbulk.cfg (section [db])

mt-bulk-rest-api gen-https-certs -C examples/configurations/mt-bulk-rest-api.example.yml

Generates self signed SSL certificates and starts REST API daemon.

mt-bulk-rest-api -C examples/configurations/mt-bulk-rest-api.example.yml

Request authentication token:

$ curl -k -H "Content-Type: application/json" -d '{"key":"abc"}' -X POST https://localhost:8080/authenticate

{"token":"yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy.zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz.xxxxxxxxxxxxxxxxxxx"}

Request execute custom command:

$ curl -k -H "Authorization: some.token.value" -H "Content-Type: application/json" -d '{"host":{"ip":"10.0.0.1","user":"admin","password":"secret"},"kind":"CustomSSH","commands":[{"body":"/user print","expect":"LAST"}]}}' -X POST https://localhost:8080/job

{"results":[{"body":"/\u003cmt-bulk\u003eestablish connection","responses":["/\u003cmt-bulk\u003eestablish connection"," --\u003e attempt #0, password #0, job #bmnb10sllhcjk1tifn2g"]},{"body":"/user print","responses":["/user print\nFlags: X - disabled \n #   NAME                                 GROUP                                 ADDRESS            LAST-LOGGED-IN      \n 0   ;;; system default user\n     admin                                full                                                     jul/19/2019 07:38:54\n"]}]}

• POST https://localhost:8080/authenticate
  Authenticate and obtain auth token. "key" is one of access keys defined in configuration [authenticate.key], each "key" can have list of regexp rules defining list of allowed device IP addresses to use in requests.

{
  "key": "abc"
}

• POST https://localhost:8080/job
  MT-bulk API request. Run and execute specified job with optional additional commands on specified host. Each request must have valid token as Authorization header field. List of possible operations

{
  "host": {
    "ip": "10.0.0.1",
    "user": "admin",
    "password": "secret"
  },
  "kind": "CustomSSH",
  "commands": [{ "body": "/user print", "expect": "LAST-LOGGED-IN" }]
}

• GET https://localhost:8080/{root_directory}/{path_to_file}
  Download file (eg. uploaded earlier by SystemBackup operation to MT-bulk root_directory defined in configuration). Each request must have valid token as Authorization header field.

• POST https://localhost:8080/upload
  Upload a file as multipart/form-data request with field file. Uploaded file is accessible in root_directory to operations like SFTP or SystemBackup. Each request must have valid token as Authorization header field.

• Verify host with running MT-bulk have access to Mikrotik device
• Verify username, password, host and port are valid, double check using eg. OpenSSH (MT-bulk doesn't require any additional tools or libraries, uses builtin in runtime SSH implementation)
• Some older RouterOS allows old and insecure ciphers, SSH implementation builtin MT-bulk will not establish connection using such ciphers, please upgrade your Mikrotik/RouterOS device
• Use strong-crypto by setting /ip ssh set strong-crypto=yes on RouterOS
• If nothing helps please provide log of establishing connection using ssh command ssh -vvv <user>@<ip>:<port>

CLI in version 2.x is simplified, all switches and configuring options are moved into configuration file, CLI is used to specify operation mode, configuration file and source of hosts to parse.

Configuration structure is rewritten and divided into few sections. Some of options changed as well (eg. verify_check_sleep into verify_check_sleep_ms). Please compare your current configuration file with attached mt-bulk.example.cfg. To let know to MT-bulk that configuration file is compatible with version 2.x new entry in config file was added: version = 2.

Application was developed by Tomasz Kolaj and is licensed under Apache License Version 2.0. Please reports bugs at https://github.com/migotom/mt-bulk/issues.