The current 0.9.12 mainnet image lisk/mainnet:latest on DockerHub does not have pgsql-client installed and is configured for testnet. This can also be seen because the image size is just half the size of the last image.

Therefore the restore scripts and guides from the Documentation are not working.

@karmacoma @fchavant @Nazgolze This breaks nodes running on Docker.

Proposal

Additionally I propose that the Docker images should be built and pushed on Jenkins from the scripts in this repo.

As far as I could see in the image there are different scripts than the ones in this repository.

Title says it all.

The current README.md needs to be updated to reflect the following changes:

• All contributors should be linked to instead of listing only authors.
• Review grammar and spelling.
• Add build status badge.

In order to maintain time against the Host OS which can be untrustworthy, we should implement ntp and configure ntpdate on a cronjob to resolve potential time drift issues.

Instead of using two images mainnet and testnet, we should be able to pass one parameter to the node which allows to connect to tesnest (--testnet). It would be easy to maintain.

Also We should include more tags as nightly ,beta and stable.

Lisky should be preinstalled on every docker image we offer, this way its even easier to get access to it.

Goal:

• build lisk/testnet and lisk/mainnet docker images from released tarballs (i.e. https://downloads.lisk.io/lisk/test/lisk-Linux-x86_64.tar.gz for lisk/testnet) instead of building everything from sources
• separate the build of the images from the examples docker-compose.yml files
• improve the README.md file to include commands for common use cases

Expected behavior

The default user to be used on PGADMIN container should be read-only. Blockchain database is immutable from its basis.

Actual behavior

We allow to write into the database if we install PGADMIN container and we follow the instructions.

Which version(s) does this affect? (Environment, OS, etc...)

0.9.11+

Currently you can only whitelist one IP for forging, would be nice if you could provide multiple. I can pick this up after my holiday or any one else is welcome to submit a PR.

This is not an urgent change and can be picked up after release.

This will simplify repeating a build for a given version (which might not be the latest) and will allow us to updated our images when the upsteam one (e.g. ubuntu:16.04) gets updated.

The test make target in images/lisk-base/Makefile is confusing. It should be renamed to "base".

The docker-compose.yml files should not expose any port for their redis containers; it is unnecessary.

Several issues make the image to change a lot:

• Update dependencies LiskHQ/lisk-sdk#594

• lisk-js as dependency LiskHQ/lisk-sdk#622

• Decoupling lisk-ui LiskHQ/lisk-sdk#458

• Remove selected dapps functionalties LiskHQ/lisk-sdk#560

  • LiskArchive/lisk-build#160 synch
  • LiskArchive/lisk-elements#106 synch
  • LiskArchive/lisk-sandbox#6 archivation
  • LiskHQ/lisk-node#9 archivation

From this commit LiskHQ/lisk-sdk#484, we use Redis in the core to improve API caching. Besides, it is planned to manage in the future more functionalities with it.

I propose then to add an isolated REDIS container to the docker-compose as we have done with PSQL.

https://hub.docker.com/r/_/redis/

Background

I was playing around with the container, and spun up a few different versions, which resulted in large docker disk consumption.
After a short time, I found the main running container had consumed all disk space on my root partition.

 $ docker system prune

was able to recover the space and continue.

Making db live outside container

I have a dedicated partition to store blockchain data, as it is can be regenerated and does not require backup. In this example, we store the data the previously created /blockchain/lisk

Running container with:

docker run -v /blockchain/lisk:/tmp/db --entrypoint /bin/bash -it lisk/mainnet

will provide an interactive shell in the container.
in the shell, transfer the contents of /var/lib/postgresql to host, and fix perm to match postgres user in container.

sudo cp -r /var/lib/postgresql/ /tmp/db/
sudo chown -R postgres:postgres /tmp/db/
exit

Run the container with the /blockchain/lisk mounted over /var/lib/postgresql folder, saving the database to the host.

docker run -v /blockchain/lisk:/var/lib/postgresql -it lisk/mainnet

If you stop and start the container, it will resume from last execution point.
The entrypoint script detects an already bootstrapped db and continues p2p and web interface.

Ideas for the docker container:

• Mount the /var/lib/postgresql folder on the host by default. Having to use container snapshots for persistence can be inefficient.
• Tar the db skeleton and extract to the mounted folder on bootstrap. This would remove the need for the run and copy step

This issues appears twice, under heading 3 an 5 on https://lisk.io/documentation?i=lisk-docs/DockerInstall

How to reproduce: do not use docker-machine, build as a user with UID other than 1001; then make testnet: restoring the database will fail when ~/.pgpass cannot be read.

Proposed solution:

• use fixed UID and GID for lisk
• make sure .pgpass always belongs to lisk
  • use a volume

In order to be aligned to the rest of products, we should use the same command or directly the script itself:

https://github.com/LiskHQ/lisk-scripts/blob/db174280162fa4b607defeb797614ef3c190225c/packaged/lisk.sh#L148

Desired behaviour

We offer several kinds of docker-compose with the typical scenarios used by delegates. Some examples are listed below:

• several nodes from the same network : useful to keep a delegate always forging avoiding forks or failover
• 1 node from mainnet + 1 node from testnet : useful for data mining or scouting

Ideally we should have a container per Lisk node but just one postgres container, one redis and one pgadmin for all the nodes running on the machine. That way the postgres container would have several databases and it can connect the pgadmin container to all of them.

Actual behaviour

Right now our docker-compose runs one single node.

In most circumstances Docker containers will be firewalled by default, therefor having open API is acceptable so that users can access the wallet functions without needing to do any advanced configuration.

There is no point to have empty databases in the container. We should give them a new value as it might be for example becoming them fresh snapshot backups for development purposes.

If this sound to complicated, I recommend to get rid of them finding instead a new simpler way to check if recovering from snapshot in the bootstrap process.

As of lisk-docker version 1.3.4, we are changing the licensing scheme from MIT to to GPLv3.

The main reason for this change is to provide protection against the future use of lisk-docker in a proprietary form, while at the same time maintaining all of the freedoms to use, change or share the project which are currently in-place.

For more information on GPLv3 please read: https://www.gnu.org/licenses/quick-guide-gplv3.html

Tasks:

• Change license footer, with amendment to reflect change of license
• Add license badge to README.md
• Include license file within project

Local build requires ssh keys to build from source. Solution is to automate generation of sshkeys specific to the end user.

There is an issue with psql hanging due to not logging to /dev/null on multi container implementations.

We should allow for a setup where the DATABASE_NAME is different from DATABASE_USER.

This allows for scenarios where people can set up one database and have multiple testnet nodes and multiple mainnet nodes all running from the same database server.

Expected Behavior

The lisk-dev docker image builds successfully in Jenkins.

Actual Behavior

The lisk-dev docker image fails to build successfully in Jenkins, with the following error returned:

cp: cannot stat '/home/lisk/lisk/test/config.json': No such file or directory
The command '/bin/sh -c cd /home/lisk/lisk &&     npm --quiet install --production &&     mkdir /home/lisk/config &&     cp -f /home/lisk/lisk/test/config.json /home/lisk/config/ &&     ln -s ../config/config.json /home/lisk/lisk/config.json &&     cp -f /home/lisk/lisk/test/genesisBlock.json 	  /home/lisk/lisk/test/genesisDelegates.json /home/lisk/lisk/ &&     jq -c ".db.database = \"lisk_local\"" config.json |sponge config.json &&     jq -c ".dapp.masterpassword = \"local\"" config.json |sponge config.json' returned a non-zero code: 1
Makefile:11: recipe for target 'local' failed
make[2]: *** [local] Error 1
make[2]: Leaving directory '/home/lisk/workspace/lisk-docker_2.0.0-RJ7NQAFTHFLSRL3XV3FE6KJDSCV27I2UTMEX47ZXW5JHXHRYT5HQ/images/lisk-dev'
Makefile:13: recipe for target 'local' failed
make[1]: *** [local] Error 2
make[1]: Leaving directory '/home/lisk/workspace/lisk-docker_2.0.0-RJ7NQAFTHFLSRL3XV3FE6KJDSCV27I2UTMEX47ZXW5JHXHRYT5HQ/images'
Makefile:13: recipe for target 'image-local' failed
make: *** [image-local] Error 2

Steps to reproduce

Run the Jenkins build on 2.0.0.

Changes since current 6.9.5:

• https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_V6.md#6.10.0
• https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_V6.md#6.10.1

Affected files:

• https://github.com/LiskHQ/lisk-docker/blob/development/Dockerfile.local
• https://github.com/LiskHQ/lisk-docker/blob/development/Dockerfile.main
• https://github.com/LiskHQ/lisk-docker/blob/development/Dockerfile.test

The Docker image should allow users to specify all of the config variables using environment variables.

Like e.g. LISK_FORGING_SECRET_<X> please consult #85 as a reference implementation.

Baking in a config.json file is not a best-practice and should not be done in order to avoid leaking secrets.

This will also make #78 much easier since ports, seed nodes etc. could be changed from e.g. a docker-compose file.

Changes since current 6.11.1:

• https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_V6.md#6.11.2
• https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_V6.md#6.11.3
• https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_V6.md#6.11.4
• https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_V6.md#6.11.5
• https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_V6.md#6.12.0
• https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_V6.md#6.12.1
• https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_V6.md#6.12.2

Lisk core has gone up a version of nodejs, docker must follow.

Expected behavior

A delegate is able to change database credentials from default.

Actual behavior

The Lisk Docker image overwrites the work done by the entrypoint. Even if changing by hand config.json, db.env and docker-compose.yml, db password gets overwritten when we restart the container.

Which version(s) does this affect? (Environment, OS, etc...)

0.9.11+

Goal:

• build lisk/local image
  • from release tarball, just like in #77
• provide example docker-compose.yml file suitable for CI usage

Expected behavior

It has to be easy to understand how to enable forging on your node while using Docker containers. One possibility would be to offer an easy interactive mode. Lisk-commander https://github.com/LiskHQ/lisky might help?

Actual behavior

There are no guidelines on which parameters to modify.

Which version(s) does this affect? (Environment, OS, etc...)

0.9.11+

We need a lisk/betanet image since betanet is open for anyone to use and test against.

Since we've switched to using Jenkins the hooks/ directory can be removed.

Now that LiskHQ/lisk-sdk#2056 has been merged the images directory can be removed. Jenkinsfile and README.md should be updated accordingly.
This repository will contain only example docker-compose.yml files and some Makefile for convenience once lisk 1.0 is released.

• link to the official documentation instead of advising to use our outdated setup_docker.Linux script
• do not recommend using the "docker" group without warning about the security implications
• clarify that creating an account and running docker login is only needed to publish one's own image on the docker hub
• explain why one would want to "archive the image"
• (in examples) use less-than and greater-than symbols to denote required options and positional arguments and square brackets for optional arguments
• add a warning to the "Useful Commands" section; reconsider its usefulness

Currently, Lisk and Postgresql go into the same container, this causes performance and data permanency issues. Therefore we should split the two into their own containers and switch to using Docker-Compose to manage the files.

Most of lisk-docker.sh reimplements the functionality of Compose which is the de facto standard for deploying multi-container applications in docker.

Compose has the following advantages:

• it's been officially supported by Docker Inc. since they took over the project;
• its yaml syntax is easy to read and modify (it's well documented);
• docker users know how to use it already;
• creates containers with unique names so that several version of the same applications (or unrelated containers) can coexist.

To test docker files changes always build and launch correctly.

Currently, we support two Docker launch mechanisms (Dockerfile and docker-compose) which makes building process much more complicated. Enforcing docker-compose will help to isolated several containers as PSQL and REDIS, which in turn improves maintenance, security, and performance.

Upgrade the pgadmin dockerfile.

Steps to repro

docker pull lisk/mainnet
docker run -d --restart=always -p 0.0.0.0:7000:7000 lisk/mainnet

docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                              NAMES
068a429f1f2a        lisk/mainnet        "/bin/sh -c './sta..."   2 seconds ago       Up 2 seconds        0.0.0.0:7000->7000/tcp, 8000/tcp   goofy_bhabha

docker logs -f 068a429f1f2a
 * Starting PostgreSQL 9.6 database server
   ...done.
psql: FATAL:  the database system is starting up
[FTL] 2017-06-10 21:13:42 | Error: FATAL:  the database system is starting up
FATAL:  the database system is starting up

    at Error (native)
^C

docker info

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 17.03.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.27-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.952 GiB
Name: moby
ID: URJF:IG4W:PU33:6H7U:AOO5:KXZT:YK36:NBB7:F6GE:BQ3Z:COUF:LJ7T
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 23
 Goroutines: 32
 System Time: 2017-06-10T21:16:51.50067995Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Switch to postgres:9.6-alpine and redis:alpine for smaller images.

Currently the link for instructions on installing the official docker image points to the archive.

Instead of building lisk in the way we do now for Docker. We should create a docker image that provides application management scripts in conjunction with docker compose

Try to stick to the best practices, especially:

• avoiding the installation of unnecessary packages;
• sorting multi-line arguments;
• making good use of the build cache;
• using ENV and EXPOSE properly;
• and preferring COPY over ADD

Running

docker build -t lisk -f Dockerfile.main .

Has this error when installing postgresql:

...
+ sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
+ wget -q https://www.postgresql.org/media/keys/ACCC4CF8.asc -O - | sudo apt-key add -
gpg: no valid OpenPGP data found.
Error executing command, exiting
The command '/bin/sh -c bash ./setup_postgresql.Linux' returned a non-zero code: 1