liskarchive / lisk-docker Goto Github PK
View Code? Open in Web Editor NEW:package: Lisk docker-compose examples
Home Page: https://lisk.io/
License: GNU General Public License v3.0
:package: Lisk docker-compose examples
Home Page: https://lisk.io/
License: GNU General Public License v3.0
The default user to be used on PGADMIN container should be read-only. Blockchain database is immutable from its basis.
We allow to write into the database if we install PGADMIN container and we follow the instructions.
0.9.11+
Switch to postgres:9.6-alpine
and redis:alpine
for smaller images.
This issues appears twice, under heading 3 an 5 on https://lisk.io/documentation?i=lisk-docs/DockerInstall
Try to stick to the best practices, especially:
Lisky should be preinstalled on every docker image we offer, this way its even easier to get access to it.
I was playing around with the container, and spun up a few different versions, which resulted in large docker disk consumption.
After a short time, I found the main running container had consumed all disk space on my root partition.
$ docker system prune
was able to recover the space and continue.
I have a dedicated partition to store blockchain data, as it is can be regenerated and does not require backup. In this example, we store the data the previously created /blockchain/lisk
Running container with:
docker run -v /blockchain/lisk:/tmp/db --entrypoint /bin/bash -it lisk/mainnet
will provide an interactive shell in the container.
in the shell, transfer the contents of /var/lib/postgresql to host, and fix perm to match postgres user in container.
sudo cp -r /var/lib/postgresql/ /tmp/db/
sudo chown -R postgres:postgres /tmp/db/
exit
Run the container with the /blockchain/lisk mounted over /var/lib/postgresql folder, saving the database to the host.
docker run -v /blockchain/lisk:/var/lib/postgresql -it lisk/mainnet
If you stop and start the container, it will resume from last execution point.
The entrypoint script detects an already bootstrapped db and continues p2p and web interface.
The Docker image should allow users to specify all of the config variables using environment variables.
Like e.g. LISK_FORGING_SECRET_<X>
please consult #85 as a reference implementation.
Baking in a config.json
file is not a best-practice and should not be done in order to avoid leaking secrets.
This will also make #78 much easier since ports, seed nodes etc. could be changed from e.g. a docker-compose file.
Goal:
lisk/testnet
and lisk/mainnet
docker images from released tarballs (i.e. https://downloads.lisk.io/lisk/test/lisk-Linux-x86_64.tar.gz for lisk/testnet
) instead of building everything from sourcesdocker-compose.yml
filesTitle says it all.
Several issues make the image to change a lot:
Update dependencies LiskArchive/lisk-sdk#594
lisk-js as dependency LiskArchive/lisk-sdk#622
Decoupling lisk-ui LiskArchive/lisk-sdk#458
Remove selected dapps functionalties LiskArchive/lisk-sdk#560
From this commit LiskArchive/lisk-sdk#484, we use Redis in the core to improve API caching. Besides, it is planned to manage in the future more functionalities with it.
I propose then to add an isolated REDIS container to the docker-compose as we have done with PSQL.
Currently you can only whitelist one IP for forging, would be nice if you could provide multiple. I can pick this up after my holiday or any one else is welcome to submit a PR.
This is not an urgent change and can be picked up after release.
Changes since current 6.11.1:
To test docker files changes always build and launch correctly.
In order to be aligned to the rest of products, we should use the same command or directly the script itself:
Currently, we support two Docker launch mechanisms (Dockerfile and docker-compose) which makes building process much more complicated. Enforcing docker-compose will help to isolated several containers as PSQL and REDIS, which in turn improves maintenance, security, and performance.
Upgrade the pgadmin dockerfile.
There is no point to have empty databases in the container. We should give them a new value as it might be for example becoming them fresh snapshot backups for development purposes.
If this sound to complicated, I recommend to get rid of them finding instead a new simpler way to check if recovering from snapshot in the bootstrap process.
Most of lisk-docker.sh
reimplements the functionality of Compose which is the de facto standard for deploying multi-container applications in docker.
Compose has the following advantages:
Running
docker build -t lisk -f Dockerfile.main .
Has this error when installing postgresql:
...
+ sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
+ wget -q https://www.postgresql.org/media/keys/ACCC4CF8.asc -O - | sudo apt-key add -
gpg: no valid OpenPGP data found.
Error executing command, exiting
The command '/bin/sh -c bash ./setup_postgresql.Linux' returned a non-zero code: 1
Steps to repro
docker pull lisk/mainnet
docker run -d --restart=always -p 0.0.0.0:7000:7000 lisk/mainnet
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
068a429f1f2a lisk/mainnet "/bin/sh -c './sta..." 2 seconds ago Up 2 seconds 0.0.0.0:7000->7000/tcp, 8000/tcp goofy_bhabha
docker logs -f 068a429f1f2a
* Starting PostgreSQL 9.6 database server
...done.
psql: FATAL: the database system is starting up
[FTL] 2017-06-10 21:13:42 | Error: FATAL: the database system is starting up
FATAL: the database system is starting up
at Error (native)
^C
docker info
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 2
Server Version: 17.03.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.27-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.952 GiB
Name: moby
ID: URJF:IG4W:PU33:6H7U:AOO5:KXZT:YK36:NBB7:F6GE:BQ3Z:COUF:LJ7T
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 23
Goroutines: 32
System Time: 2017-06-10T21:16:51.50067995Z
EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
We need a lisk/betanet
image since betanet is open for anyone to use and test against.
As of lisk-docker
version 1.3.4
, we are changing the licensing scheme from MIT to to GPLv3.
The main reason for this change is to provide protection against the future use of lisk-docker
in a proprietary form, while at the same time maintaining all of the freedoms to use, change or share the project which are currently in-place.
For more information on GPLv3 please read: https://www.gnu.org/licenses/quick-guide-gplv3.html
Tasks:
README.md
Currently, Lisk and Postgresql go into the same container, this causes performance and data permanency issues. Therefore we should split the two into their own containers and switch to using Docker-Compose to manage the files.
Currently the link for instructions on installing the official docker image points to the archive.
The lisk-dev
docker image builds successfully in Jenkins.
The lisk-dev
docker image fails to build successfully in Jenkins, with the following error returned:
cp: cannot stat '/home/lisk/lisk/test/config.json': No such file or directory
The command '/bin/sh -c cd /home/lisk/lisk && npm --quiet install --production && mkdir /home/lisk/config && cp -f /home/lisk/lisk/test/config.json /home/lisk/config/ && ln -s ../config/config.json /home/lisk/lisk/config.json && cp -f /home/lisk/lisk/test/genesisBlock.json /home/lisk/lisk/test/genesisDelegates.json /home/lisk/lisk/ && jq -c ".db.database = \"lisk_local\"" config.json |sponge config.json && jq -c ".dapp.masterpassword = \"local\"" config.json |sponge config.json' returned a non-zero code: 1
Makefile:11: recipe for target 'local' failed
make[2]: *** [local] Error 1
make[2]: Leaving directory '/home/lisk/workspace/lisk-docker_2.0.0-RJ7NQAFTHFLSRL3XV3FE6KJDSCV27I2UTMEX47ZXW5JHXHRYT5HQ/images/lisk-dev'
Makefile:13: recipe for target 'local' failed
make[1]: *** [local] Error 2
make[1]: Leaving directory '/home/lisk/workspace/lisk-docker_2.0.0-RJ7NQAFTHFLSRL3XV3FE6KJDSCV27I2UTMEX47ZXW5JHXHRYT5HQ/images'
Makefile:13: recipe for target 'image-local' failed
make: *** [image-local] Error 2
Run the Jenkins build on 2.0.0
.
It has to be easy to understand how to enable forging on your node while using Docker containers. One possibility would be to offer an easy interactive mode. Lisk-commander
https://github.com/LiskHQ/lisky might help?
There are no guidelines on which parameters to modify.
0.9.11+
There is an issue with psql hanging due to not logging to /dev/null on multi container implementations.
Instead of using two images mainnet and testnet, we should be able to pass one parameter to the node which allows to connect to tesnest (--testnet
). It would be easy to maintain.
Also We should include more tags as nightly
,beta
and stable
.
The current README.md
needs to be updated to reflect the following changes:
The test make target in images/lisk-base/Makefile
is confusing. It should be renamed to "base".
Instead of building lisk in the way we do now for Docker. We should create a docker image that provides application management scripts in conjunction with docker compose
Local build requires ssh keys to build from source. Solution is to automate generation of sshkeys specific to the end user.
We offer several kinds of docker-compose with the typical scenarios used by delegates. Some examples are listed below:
Ideally we should have a container per Lisk node but just one postgres container, one redis and one pgadmin for all the nodes running on the machine. That way the postgres container would have several databases and it can connect the pgadmin container to all of them.
Right now our docker-compose runs one single node.
A delegate is able to change database credentials from default.
The Lisk Docker image overwrites the work done by the entrypoint. Even if changing by hand config.json
, db.env
and docker-compose.yml
, db password gets overwritten when we restart the container.
0.9.11+
Goal:
lisk/local
image
docker-compose.yml
file suitable for CI usagesetup_docker.Linux
scriptdocker login
is only needed to publish one's own image on the docker hubHow to reproduce: do not use docker-machine, build as a user with UID other than 1001
; then make testnet
: restoring the database will fail when ~/.pgpass
cannot be read.
Proposed solution:
lisk
.pgpass
always belongs to lisk
This will simplify repeating a build for a given version (which might not be the latest) and will allow us to updated our images when the upsteam one (e.g. ubuntu:16.04
) gets updated.
In order to maintain time against the Host OS which can be untrustworthy, we should implement ntp and configure ntpdate on a cronjob to resolve potential time drift issues.
Changes since current 6.9.5:
Affected files:
Now that LiskArchive/lisk-sdk#2056 has been merged the images
directory can be removed. Jenkinsfile
and README.md
should be updated accordingly.
This repository will contain only example docker-compose.yml
files and some Makefile
for convenience once lisk 1.0 is released.
Since we've switched to using Jenkins the hooks/
directory can be removed.
The docker-compose.yml
files should not expose any port for their redis containers; it is unnecessary.
We should allow for a setup where the DATABASE_NAME is different from DATABASE_USER.
This allows for scenarios where people can set up one database and have multiple testnet nodes and multiple mainnet nodes all running from the same database server.
Lisk core has gone up a version of nodejs, docker must follow.
The current 0.9.12 mainnet image lisk/mainnet:latest
on DockerHub does not have pgsql-client installed and is configured for testnet. This can also be seen because the image size is just half the size of the last image.
Therefore the restore scripts and guides from the Documentation are not working.
@karmacoma @fchavant @Nazgolze This breaks nodes running on Docker.
Additionally I propose that the Docker images should be built and pushed on Jenkins from the scripts in this repo.
As far as I could see in the image there are different scripts than the ones in this repository.
In most circumstances Docker containers will be firewalled by default, therefor having open API is acceptable so that users can access the wallet functions without needing to do any advanced configuration.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.