Improve this workflow. A manager being able to log into and configure new servers (manager / worker nodes).

Ideas:

• Generate local SSH key specifically for communication with the swarm.
  • ssh-keygen -f ~/.ssh/id_rsa_swarm -t rsa -N '' -C "[email protected]"
• Copy the SSH key to swarm manager during/after installation.
  • ssh-copy-id -i ~/.ssh/id_rsa_swarm [email protected]
  • Set permissions, generate public key from private key (or just copy the id_rsa_swarm.pub file as well), etc.
  • Save in Consul(?)
• Add the swarm SSH public key to new resources / additional servers on creation. Create new servers with that key included.
• Use something like this to automatically configure a new server and to join it to the swarm.

  swarmlet join manager <ip-address>
  # or
  swarmlet join worker <ip-address>
  # e.g.
  swarmlet join manager 128.199.48.32

• Swarmlet will log into the new host, install dependencies such as Docker, and run the docker swarm join ... command to add the node to the swarm.

Does Swarmlet handle high availability gracefully when using with volume?
If you have worker and manager nodes, is it possible to setup volume such that a node (has app using volume) can go down without any issue?

Use an installation 'wizard' to interactively install Swarmlet

Currently the swarm registry can be used by anyone with push access to the swarm.
Decide how to implement using docker login instead of unauthenticated access by default.

For every Git commit, YAML files must be checked to ensure proper validation.

Continuous integration software suggestions are welcome. I only used GitLab's CI.

I spawned a new server on Vultr, ran the installer and it halted with an error:

-----> Selecting network interface
Device "eth1" does not exist.
eth0: error fetching interface information: Device not found

Hi there @woudsma - I love the framework but the git user is not auto-creating the repository as the sshcommand file above would suggest. First, I had to ssh to the server as root and set a password for the git user, because git push [swarm main] isn't happy with an empty password. Now at least I can authenticate when doing git push swarm main, but now I get xyz does not appear to be a git repository. Could not read from remote repository. And when I ssh swarm and su git, I only see deployer.git and registry.git in ~/repo.

Originally posted by @arseniybanayev in #46 (comment)

Update from v1.7 to v2+
This requires updating all examples and core modules using the loadbalancer.

Hi !

I've just find your project and I'm very glad to find it !
I have a suggestion, maybe, is it possible to you to make installation offline too ?

In some case, I work on intranet and It would be a great feature !

I have another question, but it's not the same subject.
For 2 nodes server, is glusterFS stable ?

Thanks for your work !

Currently the post-receive hook does not abort the git push if a deployment fails.
Problem: the deployments service doesn't return an error when a Dockerfile build step or registry push fails for example.

The current Traefik deployment labels can be somewhat tricky to understand, a simplified version would be nice.

Turning this:

labels:
  - traefik.enable=true
  - traefik.http.services.my-service.loadbalancer.server.port=9000
  - traefik.http.routers.my-service.rule=Host(`mysite.com`) || Host(`www.mysite.com`)
  - traefik.http.routers.my-service.entrypoints=http,https
  - traefik.http.routers.my-service.middlewares=redirect@file
  - traefik.http.routers.my-service.tls=true
  - traefik.http.routers.my-service.tls.certresolver=letsencrypt
  - traefik.http.routers.my-service.tls.domains[0].main=mysite.com
  - traefik.http.routers.my-service.tls.domains[0].sans=www.mysite.com

Into this:

labels:
  - swarmlet.port=9000
  - swarmlet.domains=mysite.com,www.mysite.com
  - [email protected]

When Traefik labels are detected next to Swarmlet labels, the file won't be parsed and only the Traefik labels will apply.

The ability to backup and restore a Swarmlet cluster is essential for production systems, particularly when dealing with databases.

A Bash script coupled with something like gobackup and/or rclone would be nice.

In order to ensure backup consistency on databases I suggest the following additional service labels:

labels:
  - swarmlet.backup.before: "./stop_disk_writes.bash"
  - swarmlet.backup.after: "./start_disk_writes.bash"

Before a backup procedure starts database containers should stop writing to disk to prevent the creation of an inconsistent backup. The limitation is lifted as soon as the backup procedure finishes. The disk write stand-by is handled by the database application itself through an application-specific command.

Currently there is no support for WireGuard, let's look into this soon.

Links:

• https://github.com/erikespinoza/v4raider
• https://www.edc4it.com/en/blog/access-docker-over-wireguard-vpn-part1-tutorial

Notes:

• https://docs.traefik.io
• https://medium.com/@containeroo/traefik-2-0-wildcard-lets-encrypt-certificates-1658370adc68
• https://frankw.net/traefik-2-0-upgrade-with-docker-compose/
• https://moritzvd.com/upgrade-traefik-2/

The documentation site is quite messy at the moment.
Todo:

• remove documentation for commands / features that don't work yet
• explain how to configure the built-in services such as Swarmpit
• make the website repo public and enable the 'edit page' button so people can contribute
• start thinking about how to explain the examples, maybe add a README.md in each example and load that into Docusaurus
• refactor and cleanup
• add Passionate People badge
• ...

I installed swarmlet on a domain, but can't seem to get anything but 404s. Should swarmpit be accessible from swarmpit.domain.tld?

Update the website and docs, how to set up a redundant swarm using multi-cloud deployment strategies.
Add some emphasis on the benefits of self-hosted software and vendor lock-in prevention by using a personal server cluster.

Let's add some functionality to easily uninstall / reinstall Swarmlet.
Something like:

uninstall-swarmlet() {
  rm -rf /opt/swarmlet
  rm -rf /var/repo
  rm -rf /home/git
  deluser git
}

reinstall-swarmlet() {
  uninstall-swarmlet
  curl -fsSL https://get.swarmlet.dev | bash
  # or install the 'develop' branch using:
  curl -fsSL https://get.staging.swarmlet.dev | bash -s INSTALL_BRANCH=develop
}

Idea: using oh-my-zsh on the server could speed things up. Function name completion, history search etc. instead of pure Bash.
Think about installing zsh optionally(?):

apt-get install -y zsh
sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

Before upgrading to Traefik v2.0+ we need to find a way to store the Let's Encrypt certificates, since Traefik doesn't support Consul as a distributed key/value store anymore.

I'm looking for a good solution to store certificates across the swarm so that they're always available and up-to-date.

Currently looking at:
https://awesome-docker.netlify.app/#volume-management--data
https://github.com/rexray/rexray
https://github.com/minio/minio
https://github.com/MatchbookLab/local-persist
https://sysadmins.co.za/docker-swarm-persistent-storage-with-nfs/

For development purposes, allowing the re-deployment of modules (e.g.: router) without doing a git push to swarmlet (deployer) is essential. A simple docker stack deploy is not enough because the deployer does some additional magic I believe.

What would be the best?

• current config file to store all options
• .env file
• user prompts during the installation?

Hey, awesome project!

I already have configuration management to take care of my swarm setup and I don't need some of the applications like consul (I run a single host cluster) or matomo (not always needed).

So, I am wondering, is it possible to configure a "bare" installation of swarmlet that just assumes that the swarm is up and just has to manage the SSH connections and deploy the stacks after that?

I am not sure if that would just cause issues within the bash scripts by creating two execution paths for "bare" and "full" install but I thought I'd open a ticket to see if it is possible to do this here in the project.

Think about extensibility of the platform.
Creating a 'plugin' could be as simple as writing and deploying an app to the swarm.
Swarmlet is completely agnostic to the language or frameworks used. If it works with Docker / Docker Compose, it should run fine with Swarmlet.
It might be good to think about a preferred format or template for these plugins/extensions/services. <- think of naming as well.

I'd like to be able to store my cluster configuration in a central location, I'm thinking of using a config service, which you can clone/pull/edit/push like any other service you've deployed. If the config service is updated, all apps using this configuration have to be updated as well.

App-specific secrets should not be stored in the config service, I'd like to store them in a safe location by creating a vault service for example. I'm currently looking into HashiCorp Vault + Consul, since Traefik integrates nicely with Consul already. Vault would provide a web UI, this is where application secrets (S3/API/SSH keys) can be stored on the swarm itself.

The deployer service will need access to the vault to use secrets in the stack build phase, that should be possible.

add-vault branch.

Notes:
https://www.marcolancini.it/2017/blog-vault/
https://learn.hashicorp.com/vault/getting-started/ui
https://testdriven.io/blog/managing-secrets-with-vault-and-consul/

Default root user SSH authorized keys:

cat /root/.ssh/authorized_keys
no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command="echo 'Please login as the user \"ubuntu\" rather than the user \"root\".';echo;sleep 10" $SSH_KEYS

Default ubuntu user SSH authorized keys:

cat /home/ubuntu/.ssh/authorized_keys
$SSH_KEYS

Issue

If the user doesn't select /home/ubuntu/.ssh/authorized_keys during installation, the generated SSH authorized keys for the git user will look like this and prevent git pushes:

cat /home/git/.ssh/authorized_keys
no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command="/opt/swarmlet/src/lib/sshcommand /var/repo /opt/swarmlet/src/hooks" no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command="echo 'Please login as the user \"ubuntu\" rather than the user \"root\".';echo;sleep 10" $SSH_KEYS

When pushing a local repository (which will fail), the output looks like:

git push swarm master
git@ec2-$IP_ADDRESS.eu-west-2.compute.amazonaws.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Solution

• Either detect and notify user if the wrong SSH keys are selected during installation
• Or: when generating the git user SSH keys, remove:

no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command="echo 'Please login as the user \"ubuntu\" rather than the user \"root\".';echo;sleep 10"

(not the best option IMO)

https://buildpacks.io/

Edit: Please disregard this comment and jump directly to #91 (comment)

TASK [install pip modules] *****************************************************
fatal: [localhost]: FAILED! => changed=false
  cmd:
  - /usr/bin/pip3
  - install
  - netaddr
  - pyvmomi
  - docker
  - requests
  msg: |-
    stdout: Collecting netaddr
      Downloading https://files.pythonhosted.org/packages/ff/cd/9cdfea8fc45c56680b798db6a55fa60a22e2d3d3ccf54fc729d083b50ce4/netaddr-0.8.0-py2.py3-none-any.whl (1.9MB)
    Collecting pyvmomi
      Downloading https://files.pythonhosted.org/packages/ba/69/4e8bfd6b0aae49382e1ab9e3ce7de9ea6318eac007b3076e6006dbe5a7cd/pyvmomi-7.0.1.tar.gz (584kB)
        Complete output from command python setup.py egg_info:
        Traceback (most recent call last):
          File "<string>", line 1, in <module>
        ModuleNotFoundError: No module named 'setuptools'

        ----------------------------------------

    :stderr: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-1dmbsd8u/pyvmomi/

PLAY RECAP *********************************************************************
localhost                  : ok=8    changed=3    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Doing pip3 install setuptools is half-solution, later:

TASK [docker-install : Install docker-compose] *********************************
fatal: [localhost]: FAILED! => changed=true
  cmd:
  - pip3
  - install
  - docker-compose
  delta: '0:00:05.654953'
  end: '2021-02-12 04:57:05.483047'
  msg: non-zero return code
  rc: 1
  start: '2021-02-12 04:56:59.828094'
  stderr: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-koscktxg/cryptography/
  stderr_lines: <omitted>
  stdout: |-
    Collecting docker-compose
      Downloading https://files.pythonhosted.org/packages/74/63/8593580d950bad554e631f60c7d09e9de684b9cb7128d0cc53aa8dccbc42/docker_compose-1.28.2-py2.py3-none-any.whl (114kB)
    Collecting docker[ssh]<5,>=4.4.0 (from docker-compose)
      Using cached https://files.pythonhosted.org/packages/9f/a5/eec74d8d1016e6c2042ba31ca6fba3bba520e27d8a061e82bccd36bd64ef/docker-4.4.1-py2.py3-none-any.whl
    Collecting distro<2,>=1.5.0 (from docker-compose)
      Downloading https://files.pythonhosted.org/packages/25/b7/b3c4270a11414cb22c6352ebc7a83aaa3712043be29daa05018fd5a5c956/distro-1.5.0-py2.py3-none-any.whl
    Collecting docopt<1,>=0.6.1 (from docker-compose)
      Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz
    Requirement already satisfied: jsonschema<4,>=2.5.1 in /usr/lib/python3/dist-packages (from docker-compose)
    Collecting dockerpty<1,>=0.4.1 (from docker-compose)
      Downloading https://files.pythonhosted.org/packages/8d/ee/e9ecce4c32204a6738e0a5d5883d3413794d7498fe8b06f44becc028d3ba/dockerpty-0.4.1.tar.gz
    Collecting python-dotenv<1,>=0.13.0 (from docker-compose)
      Downloading https://files.pythonhosted.org/packages/32/2e/e4585559237787966aad0f8fd0fc31df1c4c9eb0e62de458c5b6cde954eb/python_dotenv-0.15.0-py2.py3-none-any.whl
    Requirement already satisfied: websocket-client<1,>=0.32.0 in /usr/local/lib/python3.6/dist-packages (from docker-compose)
    Collecting texttable<2,>=0.9.0 (from docker-compose)
      Downloading https://files.pythonhosted.org/packages/06/f5/46201c428aebe0eecfa83df66bf3e6caa29659dbac5a56ddfd83cae0d4a4/texttable-1.6.3-py2.py3-none-any.whl
    Collecting cached-property<2,>=1.2.0 (from docker-compose)
      Downloading https://files.pythonhosted.org/packages/48/19/f2090f7dad41e225c7f2326e4cfe6fff49e57dedb5b53636c9551f86b069/cached_property-1.5.2-py2.py3-none-any.whl
    Requirement already satisfied: PyYAML<6,>=3.10 in /usr/lib/python3/dist-packages (from docker-compose)
    Collecting requests<3,>=2.20.0 (from docker-compose)
      Using cached https://files.pythonhosted.org/packages/29/c1/24814557f1d22c56d50280771a17307e6bf87b70727d975fd6b2ce6b014a/requests-2.25.1-py2.py3-none-any.whl
    Requirement already satisfied: six>=1.4.0 in /usr/lib/python3/dist-packages (from docker[ssh]<5,>=4.4.0->docker-compose)
    Collecting paramiko>=2.4.2; extra == "ssh" (from docker[ssh]<5,>=4.4.0->docker-compose)
      Downloading https://files.pythonhosted.org/packages/95/19/124e9287b43e6ff3ebb9cdea3e5e8e88475a873c05ccdf8b7e20d2c4201e/paramiko-2.7.2-py2.py3-none-any.whl (206kB)
    Requirement already satisfied: idna<3,>=2.5 in /usr/lib/python3/dist-packages (from requests<3,>=2.20.0->docker-compose)
    Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3/dist-packages (from requests<3,>=2.20.0->docker-compose)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests<3,>=2.20.0->docker-compose)
    Requirement already satisfied: chardet<5,>=3.0.2 in /usr/lib/python3/dist-packages (from requests<3,>=2.20.0->docker-compose)
    Collecting bcrypt>=3.1.3 (from paramiko>=2.4.2; extra == "ssh"->docker[ssh]<5,>=4.4.0->docker-compose)
      Downloading https://files.pythonhosted.org/packages/52/a7/51ab6481ac355517696477889d8ab232106a0ddadda642c54e47a2ab40b9/bcrypt-3.2.0-cp36-abi3-manylinux1_x86_64.whl (63kB)
    Collecting cryptography>=2.5 (from paramiko>=2.4.2; extra == "ssh"->docker[ssh]<5,>=4.4.0->docker-compose)
      Downloading https://files.pythonhosted.org/packages/27/5a/007acee0243186123a55423d49cbb5c15cb02d76dd1b6a27659a894b13a2/cryptography-3.4.4.tar.gz (545kB)
        Complete output from command python setup.py egg_info:

                =============================DEBUG ASSISTANCE==========================
                If you are seeing an error here please try the following to
                successfully install cryptography:

                Upgrade to the latest pip and try again. This will fix errors for most
                users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
                =============================DEBUG ASSISTANCE==========================

        Traceback (most recent call last):
          File "<string>", line 1, in <module>
          File "/tmp/pip-build-koscktxg/cryptography/setup.py", line 14, in <module>
            from setuptools_rust import RustExtension
        ModuleNotFoundError: No module named 'setuptools_rust'

        ----------------------------------------
  stdout_lines: <omitted>

PLAY RECAP *********************************************************************
localhost                  : ok=16   changed=9    unreachable=0    failed=1    skipped=12   rescued=0    ignored=0

As per the title, I’m curious to know if this project supports horizontal auto-scaling out of the box i.e. when all physical servers couldn’t hold new replicas demand by the master/controller.

If it’s supported, where is the doc/reference for it?

Decide on a style guide and implement / refactor.
Nice to have for both maintainers and contributors.

Notes:

• Use kebab-case for function names. Example: set-username(){ ... }
• Use full caps for environment variables (use camelcase for local / declare variables?)
• Use double square brackets in if statements everywhere
• Prefer double-quotes
• ...

Quoting a user from the Swarmlet Slack channel:

I'm on the latest release of swarmlet
when I push to existing repositories, the swarm stacks / services are not redeployed for some reason
I always have to delete the stack before pushing
I solved this by adding --with-registry-auth --resolve-image always to docker stack deploy in the deployer entrypoint file

cat /root/.ssh/authorized_keys | ts "$SSH_OPTIONS" > $USER_ROOT/.ssh/authorized_keys

This isn't always the case?

Perhaps it would be better to detect the current users and take that authorized_keys file? In the dokku world, we have that initial install screen where you can add a public key. Furthermore, there is the dokku ssh-key add command-line interface. I am not sure which suits Swarmlet better to do ;)

Attempted to run the interactive installation on an RPi 3 running DietPi v6.32.2 and after finishing confirming all the options the installation failed with the following output:

=====> Installing Swarmlet
-----> Installing required packages
-----> Cloning https://github.com/swarmlet/swarmlet.git
-----> Swarmlet is already installed, updating
-----> Linking executable
-----> Initializing Swarmlet
-----> Starting interactive installation
=====> Configuration confirmed
-----> The installation will take a few minutes to complete
-----> Checking apt-get for upgraded packages
-----> System up to date, continuing installation
-----> Installing Docker and Docker Compose
2020-09-03 17:02:49 URL:https://get.docker.com/ [13857/13857] -> "-" [1]
curl: (22) The requested URL returned error: 404 Not Found

Currently all Swarmlet services included (Swarmpit, Swarmprom, Matomo, Traefik, Consul, ..) are configured to be hosted on their own subdomains, under a root domain such as https://swarmlet.dev

Need to find a good solution for configuring the root domain before and/or after installation. Or maybe not even use a root domain at all, and leaving it up to the user to deploy an included service instead of deploying them anyway during installation.

FYI: placement key is duplicate in docker-compose.yaml

https://www.conventionalcommits.org/en/v1.0.0/

I'm thinking about using cz-cli.
The downside is that it requires adding a package.json and package-lock.json to the project, and requires npm or npx to be installed in order to consistently use Conventional Commits.

I haven't found good lightweight alternatives which are more language-agnostic.
Let me know if you have suggestions!

The following task fails when installing Swarmlet on Ubuntu 18.04:

TASK [system : Add the git user] ***********************************************
fatal: [localhost]: FAILED! => changed=false 
  msg: |-
    useradd: UID 1000 is not unique
  name: git
  rc: 4

I think the UID of the new git user should be set to something else other than 1000. It is very common other users might already exist in a fresh Ubuntu installation, and usually the first user created gets set to 1000.

To fix this issue, I first ran the following command to see which user has UID 1000.

id -nu 1000

Output:

opc

I then set the user opc to a different UID

usermod -u 2005 opc

Now UID 1000 should be free to use.

https://go-acme.github.io/lego/

WIP: https://github.com/swarmlet/swarmlet/tree/feature/glusterfs/modules/core/router

Awesome project, am really looking forward to the future improvements!

Currently, when pushing a deployment with a Dockerfile, I systematically get a
"Get http://127.0.0.1:5000/v2/: dial tcp 127.0.0.1:5000: connect: connection refused"

Same message if I do a "docker-compose push" (following a successful build) on the server itself.

Haven't touched consul's configuration - what am I missing?

Many thanks and keep up the great work!

Will this appear in the #swarmlet-github channel?

Todo:

• Write a document on how to initialize Matomo.
  Steps:
  • Visit the Matomo UI at https://matomo.<yourdomain.com>
  • As database url, provide db instead of 127.0.0.1
  • Use matomo as database name, user and password (unless you've configured this yourself)
• Add a README.md document to each service and example where necessary.
• Update Swarmlet docs to include readme's from the examples.

Hey, I've setup Swarmlet 3 days ago but since yesterday I'm getting an error when deploying with git or stopping a service through Swarmpit.

On Swarmpit I get : 'Service stopping failed. No matching registry ( 127.0.0.1:5000 ) linked with Swarmpit'

On git push : 'remote: Get http://127.0.0.1:5000/v2/: dial tcp 127.0.0.1:5000: connect: connection refused'

I tried restarting the server/containers but it didn't solved anything.

Did I make something wrong ? Is there a fix ?

Thanks,

Using this amazing article as reference for example:
https://knowledgepill.it/posts/docker_swarm_compendium/

Note: take a look at https://github.com/kwk/docker-registry-frontend

Self-explanatory.

TBD

I am following the demo video.
I noticed the installation created a new user "git"
However, the installation video is configuring ssh's "swarm" as "root"

When I am performing git push, am I supposed to push to user "root" or "git"?

Thanks.

Do you have any pointers for updating an existing installation? My version is pre-v0.1

The install script requires whiptail to run. The documentation does not mention this and the script fails.

Tested the following command on Ubuntu 20.04.

root@swarm-manager-1:~# curl -fsSL https://get.swarmlet.dev | bash
=====> Installing Swarmlet
-----> Installing required packages
-----> Cloning https://github.com/swarmlet/swarmlet.git
-----> Initializing Swarmlet
-----> Starting interactive installation
/tmp/swarmlet/src/commands/dialog: line 30: whiptail: command not found

Perhaps whiptail should be installed as part of the required packages?

Use Ansible to manage GlusterFS repliacted volumes

Todo:

• Implement solution to remove an app and clean up after. Something like

  swarmlet app:<remove|destroy|delete>

  Should execute:

  rm -rf /var/repo/<app-name>.git
  rm -rf /home/git/<app-name>
  docker stack rm <app-name>