Giter Club home page Giter Club logo

any-docker's Introduction

⚠️Important note from the devs⚠️

Hello, we have merged multi-spaces into main branch, we have some more tasks to do and we hope to start beta-testing later in October. Please, if you use app built from source, be aware that there are risks of potential damage to your accounts, so be careful. Right now we are testing spaces internally.

any-docker

Community-maintained dockerized instance of an anytype node

⚠️ Note: this project is "work in progress" and is not yet ready for production use. It's actually not yet even ready for development use ;) (but I'm working on that!)

⚠️ Disclaimer: I've tried to write the instructions below in such a way that as many people as possible can run their own Anytype node. This means it might be too verbose for your taste. In the case it is not clear enough, feel free to submit an Issue or submit a pull request 💌.

Aim: provide a simple to use yet customizable docker solution that runs an Anytype node and required infrastructure for development and testing.

Project Status

✅ Infrastructure

  • ✅ mongodb (database)
  • ✅ mongo_express (optional)
  • ✅ minio (s3 file storage)
  • ✅ redis (file cache)
  • ⏳ reverse proxy

✅ Any-sync

  • ✅ any-sync-coordinator
  • ✅ any-sync-node
  • ✅ any-sync-filenode

✅ Middleware libraries

  • ✅ Desktop
  • ✅ Android
  • ❌ iOS (can only be built on MacOS, which I currently don't have)

⏳ Clients

  • ⏳ Desktop
    • ⏳ Windows (build succeeds, but client won't run properly yet)
    • ❌ Linux (snapcraft error: snapcraft does not currently support building arm64 on amd64)
    • ❌ MacOS (can only be built on MacOS, which I currently don't have)
  • ⏳ Android
  • ❌ iOS (can only be built on MacOS, which I currently don't have)

Prerequisites

In order to follow this guide, you'll need to run a debian based linux distribution such as Debian or Ubuntu with git installed. You can install git running:

apt update && apt upgrade
apt-get install -y git

ℹ️ Tip: install portainer to easily manage your docker "stack" (portainer term for docker compose configuration) and see logs for troubleshooting.

Run node in docker

To get started, clone this repository, enter the directory, and run your node including the required infrastructure in Docker compose using the docker-compose.yml and .env files:

git clone https://github.com/SamBouwer/any-docker

Now enter the directory you just pulled from Github

cd any-docker/any-node/

Fire up 🔥 docker compose!

docker compose up -d

ℹ️ Tip: run git pull when you want to pull the latest version of the repository to your local machine.

You should now see something similar to this:

[+] Running 9/9
 ✔ Network any-docker_any-network  Created
 ✔ Volume "any-docker_s3"          Created
 ✔ Volume "any-docker_db"          Created
 ✔ Container mongo_anytype         Started
 ✔ Container any-sync-coordinator  Started
 ✔ Container any-sync-node         Started
 ✔ Container any-sync-filenode     Started
 ✔ Container minio_anytype         Started
 ✔ Container redis_anytype         Started

Congratulations! 🎉 You are now running your own Anytype Node!

To stop the node again, navigate to the any-node folder and bring the stack down:

docker compose down

Build clients

ℹ️ You can create your own network configuration using the any-sync-network tool as provided by Anytype. For the click-to-run experience I want to offer, you don't need to run this command and the default network addresses and configuration will be used. If you do want to setup your own network: make create-network-conf. Warning: this will break the docker setup!

To actually use the node, we need to build Anytype clients that can connect to this node. In the current version, the connection configuration and middleware libraries are baked into the clients when the clients are built from source. To build the middleware libraries and Anytype clients, run the following commands, where teh make build-client-all step can be replaced to only build a specific client. Checkout make help to find out the commands for the specific clients.

⚠️ Don't forget to navigate back to the folder where you already have cloned any-docker or where you want to clone the repo if not cloned yet!

You only need to run this part if you did not pull the any-docker repository in a previous step

git clone https://github.com/SamBouwer/any-docker
cd any-docker

Now setup the development environment by installing all required dependencies

make setup-dev-env

Finally, build the clients.

make build-client-all

ℹ️ Note: you only need to run make setup-dev-env once to install dependencies and create the required folder structure. If you (accidently) deleted or moved a folder or uninstalled packages required for building Anytype clients, you need to run this command again. You can always run it just to make sure everything is setup properly

ℹ️ Note: if you are running this in WSL, make sure to add the following lines to /etc/wsl.conf to ensure snap can be installed:

[boot]
systemd=true

Then run wsl.exe --shutdown

Clients will be in the "anytype" folder once done.

Build nodes

You can skip this step of you just want to run nodes as provided in the docker images without further customization. If you want to build the any-docker images yourself, clone this repo, enter the directory and make it!

git clone https://github.com/SamBouwer/any-docker
cd any-docker
make build-node-all

ℹ️ Tip: run make help to see all make options, for example to build individual nodes or clients only

This will build the following docker images:

  • any-sync-coordinator
  • any-sync-node
  • any-sync-filenode

Troubleshooting

In the case you run into an error, please remove the any-docker folder and start from scratch. There might be some leftover file from a previous run that is not compatible with a newer version of this repo, or a newer version of the Anytype source files. Either way, please submit a Github Issue so I can improve the script :).

Contribute

I'm building a Dockerfile and supporting scripts and files based on the instructions as posted here: https://tech.anytype.io/how-to/self-hosting

As building Anytype nodes and clients from source for selfhosting is new, and I am personally new to building docker images, any help is welcome!

⚠️ Note: You will find many Id's and Keys in this repo, but those are often changing and only exposed locally on my machine, so its of little use to you :)

Disclaimer

I am not part of the Anytype team.

any-docker's People

Contributors

sambouwer avatar developomp avatar bjcorder avatar shampra avatar

Stargazers

 avatar  avatar Marcos Pereira avatar Mary White avatar peroksid5 avatar zhoufoxcn avatar Artem Solovev avatar  avatar  avatar  avatar Aurel Weinhold avatar  avatar SpringHack avatar Kevin James O'Dea avatar Ben Force avatar José Muñoz avatar  avatar  avatar  avatar NayanTheSpaceGuy avatar Razor avatar  avatar Chris Clark avatar Alexey Dyumin avatar  avatar  avatar David avatar Benooye avatar Vladislav Sorokin avatar Annika avatar Till Tomczak avatar Benjamin D. Smith avatar Denis Ivannikov avatar  avatar LluísClaitec avatar Rin avatar Akshay Aggarwal avatar jiangzhiming avatar Henrik avatar Jeff West avatar Clément Michaud avatar عثمان محامدي avatar Nikolay Gniteev avatar  avatar  avatar Luke Wells avatar Moritz Kitzmann avatar Trevor Vance avatar TheSunCat avatar D S avatar 李旌 avatar  avatar  avatar  avatar tirefire avatar Sage Waters avatar Markus Stephanides avatar  avatar kerbless avatar Renaud Lifchitz avatar  avatar  avatar Didier Marin avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

any-docker's Issues

Ubuntu WLS2 : Package 'docker-ce' unavailable

setup_dev_env.sh has a problem with me.

Environment : VM Windows 11 (dev image by Microsoft), WSL2, Ubuntu 22.04.2 LTS

sudo apt-get install -y --no-install-recommends \ ... cannot find package

Error :

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package docker-ce is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'docker-ce' has no installation candidate
E: Unable to locate package docker-ce-cli
E: Unable to locate package containerd.io
E: Couldn't find any package by glob 'containerd.io'
E: Couldn't find any package by regex 'containerd.io'
E: Unable to locate package docker-buildx-plugin
E: Unable to locate package docker-compose-plugin

Impact : it doesn't seem to install any packages, generating errors afterwards (notably jq and non-existent unzip).
I had to manually install the missing packages

Android client

I don't see an option to select a custom server on the android app.
I don't see android client on the makefile.
I see the checkmark on the android Middleware library in the project status.
I'm not sure I undestand what that is.
Is there a way to use the android app and sync to my custom anytime node?

Thank you for bringing docker as an option for selfhosting anytype.

Keep up

WSL : cannot communicate with server snapcraft

Trying to build clients, following readme.

On make setup-dev-env, I have this error :

+ sudo snap install snapcraft --classic
error: cannot communicate with server: Post "http://localhost/v2/snaps/snapcraft": dial unix /run/snapd.socket: connect: no such file or directory
make: *** [Makefile:53: setup-dev-env] Error 1

Environment :

  • Windows 11
  • WSL Ubuntu 22.04.2 LTS
  • system.d active

(and after that, if I launch make build-client-win, it crash, I'm not sure if it's related to this issue or independent)
image

npx task remains stuck

npx [email protected] install-app-deps --platform=win32 --arch=x64 never finishes

Building Windows client...
  • electron-builder  version=24.6.3
  • loaded configuration  file=package.json ("build" field)
  • rebuilding native dependencies  [email protected] platform=win32 arch=x64

npx seems to have this problem sometimes. Using npm exec, it works

Split monolith Docker image

It's easy to get started with a single docker image for a single docker container, but some people probably want a bit more flexibility to run their own Mongodb, s3, and redis. Also requires separate docker run and docker-compose.yaml examples.

How to build, distribute and run clients

Try to script and add instructions how to build your own Anytype client from source to connect to your selfhosted node

  • Build anytype-heart middleware libraries for Desktop
  • Build anytype-heart middleware libraries for Android
  • Build anytype-heart middleware libraries for iOS
  • Build anytype-ts Desktop client for Windows
  • Build anytype-ts Desktop client for Linux
  • Build anytype-ts Desktop client for MacOS
  • Update docs

Inconsistent paths in dev setup script

Hi! Thank you for the effort in making this repo!

In the any-docker/scripts/setup_dev_env.sh, there's inconsistency in paths during the GO installation:

if [ "${GOLANG_VER_INSTALLED}" == "go version go${GOLANG_VER} linux/amd64" ] ; then
  echo "'go version go${GOLANG_VER} linux/amd64' is already installed, skipping..."
else
  echo "Installing 'go version go${GOLANG_VER} linux/amd64'"
  wget -N https://go.dev/dl/go${GOLANG_VER}.linux-amd64.tar.gz
  tar -C /usr/local -xzf $HOME/go${GOLANG_VER}.linux-amd64.tar.gz
  rm go${GOLANG_VER}.linux-amd64.tar.gz
fi

The .tar file is downloaded to the current directory but is expected to be in $HOME later.

Which gives this error:

tar (child): /home/<username>/go1.19.11.linux-amd64.tar.gz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now

Error with update.sh when used by build-client-win script

Strange error.

I launch build-client-win.sh, which launches the script update.sh windows-latest. (WSL2 Ubuntu)
But the script gets stuck moving folders protobuf/* to dist/lib, it can't find the directory.

renamed 'grpc-server.exe' -> 'dist/anytypeHelper.exe'
mv: target 'dist/lib/' is not a directory
renamed 'json/internalRelations.json' -> 'dist/lib/json/generated/internalRelations.json'
renamed 'json/internalTypes.json' -> 'dist/lib/json/generated/internalTypes.json'
renamed 'json/systemRelations.json' -> 'dist/lib/json/generated/systemRelations.json'
renamed 'json/systemTypes.json' -> 'dist/lib/json/generated/systemTypes.json'
Done

I thought there might be a problem with this script, but if I run it manually, it works fine (under Windows/Powershell as well as on WSL2/Ubuntu).

renamed 'grpc-server.exe' -> 'dist/anytypeHelper.exe'
renamed 'protobuf/pb' -> 'dist/lib/pb'
renamed 'protobuf/pkg' -> 'dist/lib/pkg'
renamed 'protobuf/protos' -> 'dist/lib/protos'
renamed 'json/internalRelations.json' -> 'dist/lib/json/generated/internalRelations.json'
renamed 'json/internalTypes.json' -> 'dist/lib/json/generated/internalTypes.json'
renamed 'json/systemRelations.json' -> 'dist/lib/json/generated/systemRelations.json'
renamed 'json/systemTypes.json' -> 'dist/lib/json/generated/systemTypes.json'
Done

No more build-heart in make file

Hi,

make build-heart does not exist (or no longer exists) in the make file. I'm assuming that since it's built into "build-client", it's been removed and all you have to do is update the readme file?

(no big deal compared to all the mistakes I've made, but hey...)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.