Giter Club home page Giter Club logo

hoprnet / hoprnet Goto Github PK

View Code? Open in Web Editor NEW
206.0 14.0 89.0 475.95 MB

HOPR is an open incentivized mixnet which enables privacy-preserving point-to-point data exchange. HOPR is similar to Tor but actually private, decentralized and economically sustainable.

Home Page: https://hoprnet.org

License: GNU General Public License v3.0

Dockerfile 0.01% Shell 3.03% Solidity 29.53% Nix 0.59% Makefile 1.28% Rust 63.10% Python 2.45%
privacy blockchain ethereum

hoprnet's Introduction

HOPR Logo

HOPR

A project by the HOPR Association

HOPR is a privacy-preserving messaging protocol which enables the creation of a secure communication network via relay nodes powered by economic incentives using digital tokens.

Table of Contents

About

The HOPR project produces multiple artifacts that allow running, maintaining and modiyfing the HOPR node. The most relevant components for production use cases are:

  1. hopr-lib
    • A fully self-contained referential implementation of the HOPR protocol over a libp2p based connection mechanism that can be incroporated into another projects as a transport layer.
  2. hoprd
    • Daemon application providing a higher level interface for creating a HOPR protocol compliant node that can use a dedicated REST API.
  3. hoprd-api-schema
    • Utility to generate the OpenAPI spec for the hoprd served REST API.
  4. hoprd-cfg
    • Utility for configuration management of the hoprd
  5. hopli
    • Utility designed to simplify and unify the management of on-chain and identity related tasks.

Unless stated otherwise, the following sections only apply to hoprd.

Install

For production purposes always run the latest stable release.

Multiple options for installation exist, the preferred choice for any production system should be to use the container image (e.g. using docker).

All releases and associated changelogs are located in the official releases section of the hoprnet repository.

Install via Docker

The following instructions show how any $RELEASE may be installed, to select the release, override the $RELEASE variable, e.g.:

  • export RELEASE=latest to track the latest changes on the repository's master branch
  • export RELEASE=saint-louis to track the latest changes on the repository's release/saint-louis branch (2.1.X)
  • export RELEASE=<version> to get a specific <version>

Container image has the format europe-west3-docker.pkg.dev/hoprassociation/docker-images/$PROJECT:$RELEASE. where:

  • $PROJECT can be either hopli or hoprd

Pull the container image with docker:

$ docker pull europe-west3-docker.pkg.dev/hoprassociation/docker-images/hoprd:saint-louis

It is recommended to setup an alias hoprd for the docker command invocation.

WARNING: This setup should only be used for development or advanced usage without any further support.

Clone and initialize the hoprnet repository:

$ git clone https://github.com/hoprnet/hoprnet
$ cd hoprnet

Build and install the hoprd binary, e.g. on a UNIX platform:

$ nix build
$ sudo cp result/bin/* /usr/local/bin/

Usage

hoprd provides various command-line switches to configure its behaviour. For reference these are documented here as well:

$ hoprd --help
HOPR node executable.

Usage: hoprd [OPTIONS]

Options:
      --network <NETWORK>
          ID of the network the node will attempt to connect to [env: HOPRD_NETWORK=]
      --identity <IDENTITY>
          The path to the identity file [env: HOPRD_IDENTITY=]
      --data <DATA>
          Specifies the directory to hold all the data [env: HOPRD_DATA=]
      --host <HOST>
          Host to listen on for P2P connections [env: HOPRD_HOST=]
      --announce...
          Announce the node on chain with a public address [env: HOPRD_ANNOUNCE=]
      --api...
          Expose the API on localhost:3001 [env: HOPRD_API=]
      --apiHost <HOST>
          Set host IP to which the API server will bind [env: HOPRD_API_HOST=]
      --apiPort <PORT>
          Set port to which the API server will bind [env: HOPRD_API_PORT=]
      --apiToken <TOKEN>
          A REST API token and for user authentication [env: HOPRD_API_TOKEN=]
      --password <PASSWORD>
          A password to encrypt your keys [env: HOPRD_PASSWORD=]
      --disableUnrealizedBalanceCheck...
          Disables checking of unrealized balance before validating unacknowledged tickets. [env: HOPRD_DISABLE_UNREALIZED_BALANCE_CHECK=]
      --maxBlockRange <MAX_BLOCK_RANGE>
          Maximum number of blocks that can be fetched in a batch request from the RPC provider. [env: HOPRD_MAX_BLOCK_RANGE=]
      --maxRequestsPerSec <MAX_RPC_REQUESTS_PER_SEC>
          Maximum number of RPC requestes that can be performed per second. [env: HOPRD_MAX_RPC_REQUESTS_PER_SEC=]
      --provider <PROVIDER>
          A custom RPC provider to be used for the node to connect to blockchain [env: HOPRD_PROVIDER=]
      --init...
          initialize a database if it doesn't already exist [env: HOPRD_INIT=]
      --forceInit...
          initialize a database, even if it already exists [env: HOPRD_FORCE_INIT=]
      --inbox-capacity <INBOX_CAPACITY>
          Set maximum capacity of the HOPRd inbox [env: HOPRD_INBOX_CAPACITY=]
      --heartbeatInterval <MILLISECONDS>
          Interval in milliseconds in which the availability of other nodes get measured [env: HOPRD_HEARTBEAT_INTERVAL=]
      --heartbeatThreshold <MILLISECONDS>
          Timeframe in milliseconds after which a heartbeat to another peer is performed, if it hasn't been seen since [env: HOPRD_HEARTBEAT_THRESHOLD=]
      --heartbeatVariance <MILLISECONDS>
          Upper bound for variance applied to heartbeat interval in milliseconds [env: HOPRD_HEARTBEAT_VARIANCE=]
      --networkQualityThreshold <THRESHOLD>
          Minimum quality of a peer connection to be considered usable [env: HOPRD_NETWORK_QUALITY_THRESHOLD=]
      --configurationFilePath <CONFIG_FILE_PATH>
          Path to a file containing the entire HOPRd configuration [env: HOPRD_CONFIGURATION_FILE_PATH=]
      --safeTransactionServiceProvider <HOPRD_SAFE_TX_SERVICE_PROVIDER>
          Base URL for safe transaction service [env: HOPRD_SAFE_TRANSACTION_SERVICE_PROVIDER=]
      --safeAddress <HOPRD_SAFE_ADDR>
          Address of Safe that safeguards tokens [env: HOPRD_SAFE_ADDRESS=]
      --moduleAddress <HOPRD_MODULE_ADDR>
          Address of the node mangement module [env: HOPRD_MODULE_ADDRESS=]
      --protocolConfig <HOPRD_PROTOCOL_CONFIG_PATH>
          Path to the protocol-config.json file [env: HOPRD_PROTOCOL_CONFIG_PATH=]
      --dryRun
          DEPRECATED [env: HOPRD_DRY_RUN=]
      --healthCheck
          DEPRECATED
      --healthCheckHost <HEALTH_CHECK_HOST>
          DEPRECATED
      --healthCheckPort <HEALTH_CHECK_PORT>
          DEPRECATED
      --defaultStrategy <DEFAULT_STRATEGY>
          DEPRECATED [env: HOPRD_DEFAULT_STRATEGY=] [possible values: promiscuous, aggregating, auto_redeeming, auto_funding, closure_finalizer, multi, passive]
      --maxAutoChannels <MAX_AUTO_CHANNELS>
          DEPRECATED [env: HOPRD_MAX_AUTO_CHANNELS=]
      --disableTicketAutoRedeem...
          DEPRECATED [env: HOPRD_DISABLE_AUTO_REDEEEM_TICKETS=]
  -h, --help
          Print help
  -V, --version
          Print version

Example execution

Running the node without any command-line argument might not work depending on the installation method used. Some command line arguments are required.

A basic reasonable setup is that uses a custom identity and enabels a REST API of the hoprd could look like:

hoprd --identity /app/hoprd-db/.hopr-identity --password switzerland --init --announce --host "0.0.0.0:9091" --apiToken <MY_TOKEN> --network doufur

Here is a short breakdown of each argument.

hoprd
  # store your node identity information in the persisted database folder
  --identity /app/hoprd-db/.hopr-identity
  # set the encryption password for your identity
  --password switzerland
  # initialize the database and identity if not present
  --init
  # announce the node to other nodes in the network and act as relay if publicly reachable
  --announce
  # set IP and port of the P2P API to the container's external IP so it can be reached on your host
  --host "0.0.0.0:9091"
  # specify password for accessing REST API
  --apiToken <MY_TOKEN>
  # an network is defined as a chain plus a number of deployed smart contract addresses to use on that chain
  --network doufur

Special care needs to given to the network argument, which defines the specific network hoprd node should join. Only nodes within the same network can communicate using the HOPR protocol.

Using Docker Compose with extended HOPR node monitoring

An optional docker compose setup can be used to run the above containerized hoprd along with extension to observe the node's metrics using Prometheus + Grafana dashboard:

docker compose --file scripts/compose/docker-compose.yml up -d

Copy the scripts/compose/default.env to scripts/compose/.env and change the variables as desired.

The composite setup will publish multiple additional services alongside the hoprd:

  • Admin UI at localhost:3000
  • Grafana with hoprd dashboards at localhost:3030 (default user: admin and pass hopr)

Testnet accessibility

To participate in a public network the node must be eligible. See Network Registry for details.

Node eligibility is not required in a local development cluster (see Develop section below).

Migrating between releases

There is NO backward compatibility between releases.

We attempt to provide instructions on how to migrate your tokens between releases.

  1. Set your automatic channel strategy to passive.
  2. Redeem all unredeemed tickets.
  3. Close all open payment channels.
  4. Once all payment channels have closed, withdraw your funds to an external wallet.
  5. Run info and take note of the network name.
  6. Once funds are confirmed to exist in a different wallet, backup .hopr-identity folder.
  7. Launch new HOPRd instance using latest release, observe the account address.
  8. Only transfer funds to new HOPRd instance if HOPRd operates on the same network as last release, you can compare the two networks using info.

Develop

Either setup nix and flake to use the nix environment, or install Rust toolchain from the rust-toolchain.toml, as well as foundry-rs binaries (forge, anvil).

Nix environment setup

Install `nix`` from the official website at https://nix.dev/install-nix.html.

Create a nix configuration file at ~/.config/nix/nix.conf with the following content:

experimental-features = nix-command flakes

Install the nix-direnv package to introduce the direnv:

$ nix-env -i nix-direnv

Append the following line to the shell rc file (depending on the shell used it can be ~\.zshrc, ~\.bashrc, ~\.cshrc, etc.). Modify the <shell> variable inside the below command with the currently used (zsh, bash, csh, etc.):

$ eval "$(direnv hook <shell>)"

From within the hoprnet repository's directory, execute the following command.

$ direnv allow .

Nix flake outputs

We provide a couple of packages, apps and shells to make building and development easier, to get the full list execute:. You may get the full list like so:

$ nix flake show

Code Formatting

All nix, rust, solidity and python code can be automatically formatted:

nix fmt

These formatters are also automatically run as a Git pre-commit check.

Code Linting

All linters can be executed via a Nix flake helper app:

nix run .#lint

This will in particular run clippy for the entire Rust codebase.

Local node with safe staking service (local network)

Running one node in test mode, with safe and module attached (in an anvil-localhost network)

# clean up, e.g.
# make kill-anvil
# make clean

# build deps and HOPRd code
make -j deps && make -j build

# starting network
make run-anvil args="-p"

# update protocol-config
scripts/update-protocol-config.sh -n anvil-localhost

# create identity files
make create-local-identity id_count=1

# create a safe and a node management module instance,
# and passing the created safe and module as argument to
# run a test node local (separate terminal)
# It also register the created pairs in network registry, and
# approve tokens for channels to move token.
# fund safe with 2k token and 1 native token
make run-local-with-safe id_file_path=/tmp
# or to restart a node and use the same id, safe and module
# run:
# make run-local id_path=$(find `pwd` -name ".identity-local*.id" | sort -r | head -n 1)

# fund all your nodes to get started
make fund-local-all id_dir=`pwd`

# start local HOPR admin in a container (and put into background)
make run-hopr-admin &

Local node with safe staking service (dufour network)

Running one node in test mode, with safe and module attached (in dufour network)

# build deps and HOPRd code
make -j deps && make -j build

# Fill out the `ethereum/contract/.env` from the `ethereum/contract/example.env`
#
# ensure a private key with enough xDAI is set as PRIVATE_KEY
# This PRIVATE_KEY is the "admin_key" (i.e. owner of the created safe and node management module)
#
# Please use the deployer private key as DEPLOYER_PRIVATE_KEY
# The Ethereum address to the DEPLOYER_PRIVATE_KEY should be a "manager" of the network registry.
# Role can be checked in the explorer:
# ```
# echo "https://gnosisscan.io/address/$(jq '.networks.dufour.addresses.network_registry' ./ethereum/contracts/contracts-addresses.json)\#readContract"
# ```
source ./ethereum/contracts/.env

export HOPR_NETWORK="dufour"
export IDENTITY_PASSWORD="SOmeranDOmPassHere-DefiniteLyChangeThis!"

# create identity files
bash scripts/generate-identity.sh

# start local HOPR admin in a container (and put into background)
make run-hopr-admin &

Local cluster

The best way to test with multiple HOPR nodes is by using a local cluster of interconnected nodes.

Test

Unit testing

Tests both the Rust and Solidity code.

make test

Github Actions CI

We run a fair amount of automation using Github Actions. To ease development of these workflows one can use act to run workflows locally in a Docker environment.

E.g. running the build workflow:

$ act -j build

For more information please refer to act's documentation.

End-to-End Testing

When using the nix environment, the test environment preparation and activation is automatic.

Tests are using the pytest infrastructure.

Running Tests Locally

Testing environment

If not using nix, setup the pytest environment:

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install -r tests/requirements.txt

To deactivate the activated testing environment if no longer needed:

deactivate
Test execution

With the environment activated, execute the tests locally:

make smoke-test-full

Contact

License

GPL v3 © HOPR Association

hoprnet's People

Contributors

0xjjpa avatar adharasi avatar andreykiryushkin avatar anicolaides avatar ausias-armesto avatar constantidis avatar dependabot[bot] avatar hopr-bot avatar hopr-version-bot avatar jaguaras avatar jeandemeusy avatar martinbenediktbusch avatar mathcrypto avatar mjadach-iv avatar mswezey23 avatar nionis avatar numberfour8 avatar peterbraden avatar qyuqianchen avatar reimo avatar renovate-bot avatar renovate[bot] avatar repo-ranger[bot] avatar scbuergel avatar sebastiantf avatar shresthagrawal avatar sponnet avatar teebor-choka avatar thewanderingeditor avatar tolbrino avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

hoprnet's Issues

add fathom tracker

Describe the use case

The feature
Add fathom tracker in docs.hoprnet.org

Archive old repositories

Now that we have all the code in the monorepo, let's go back through the old repositories and:

  • Make sure all issues that are still relevant are transferred to this repo.
  • Close all issues / PR's
  • Add a deprecation notice with a link to this repo in the readme.
  • Archive the repository.

Have health checks on the nodes via a ping bot and/or other mechanism

Describe the use case
There will be churn. Nodes will go offline and come back later with the same HOPR ID but a different connection.

Our bot should therefore ping the connected nodes every now and then and keep a list of nodes that are most likely online.

The feature
Suggestion:
We expose heartbeat as an API method and pass a list of "interesting nodes" that heartbeat will contact first.

Checkout why connection establishment fails during length-prefix parsing

  hopr-core:transport:listener:error inbound connection failed Error: Value is null
    at Object.readLP (/home/robert/Downloads/hopr-chat-nodebin-linux(3)/hopr-chat/index.js:86015:27)
    at processTicksAndRejections (internal/process/task_queues.js:85:5)
    at async exchange (/home/robert/Downloads/hopr-chat-nodebin-linux(3)/hopr-chat/index.js:111697:15)
    at async handshake (/home/robert/Downloads/hopr-chat-nodebin-linux(3)/hopr-chat/index.js:130264:3)
    at async Object.secure (/home/robert/Downloads/hopr-chat-nodebin-linux(3)/hopr-chat/index.js:220521:3)
    at async Upgrader._encryptInbound (/home/robert/Downloads/hopr-chat-nodebin-linux(3)/hopr-chat/index.js:25157:12)
    at async Upgrader.upgradeInbound (/home/robert/Downloads/hopr-chat-nodebin-linux(3)/hopr-chat/index.js:24915:11)
    at async Server.<anonymous> (/home/robert/Downloads/hopr-chat-nodebin-linux(3)/hopr-chat/index.js:1658:20) {
  code: 'ERR_ENCRYPTION_FAILED'

It seems that the connection establishment fails just before upgrading a connection.

Sync w/the avado team for ways to enhance technical documentation

Seems like some of the processes we are going through are very bleeding-edge and are yet to be documented (e.g. auto-update). It would be worth syncing with the AVADO team and see ways we can enhance their documentation as to benefit long-term in the project from both sides.

Create a smart contract migration workflow

Describe the use case
Automatically migrate smart contracts on major? release and update the addresses.ts file

The feature
Create a smart contract migration workflow

Node Crashes

Describe the bug
My node crashed

To Reproduce
Not sure, I will post the log. Maybe, it could help you.

Expected behavior
The node should have been running.

Screenshots
Attaching the log file

Desktop (please complete the following information):

  • OS: Ubuntu 18
  • Browser Not applicable
  • Version 18

Partial content of Log file below:

(node:14998) UnhandledPromiseRejectionWarning: Error: Value is null
at Object.readLP (/home/downloads/hopr-chat/index.js:82862:27)
at runMicrotasks ()
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async exchange (/home/downloads/hopr-chat/index.js:105539:15)
at async handshake (/home/downloads/hopr-chat/index.js:118387:3)
at async Object.secure (/home/downloads/hopr-chat/index.js:171647:3)
at async Upgrader._encryptInbound (/home/downloads/hopr-chat/index.js:26351:12)
at async Upgrader.upgradeInbound (/home/downloads/hopr-chat/index.js:26109:11)
at async TCP.handleDelivery (/home/downloads/hopr-chat/index.js:25002:24)
(node:14998) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 13493)
(node:14998) UnhandledPromiseRejectionWarning: Error: Value is null
at Object.readLP (/home/downloads/hopr-chat/index.js:82862:27)
at runMicrotasks ()
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async exchange (/home/downloads/hopr-chat/index.js:105539:15)
at async handshake (/home/downloads/hopr-chat/index.js:118387:3)
at async Object.secure (/home/downloads/hopr-chat/index.js:171647:3)
at async Upgrader._encryptInbound (/home/downloads/hopr-chat/index.js:26351:12)
at async Upgrader.upgradeInbound (/home/downloads/hopr-chat/index.js:26109:11)
at async TCP.handleDelivery (/home/downloads/hopr-chat/index.js:25002:24)
(node:14998) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 13494)
(node:14998) UnhandledPromiseRejectionWarning: Error: Value is null
at Object.readLP (/home/downloads/hopr-chat/index.js:82862:27)
at runMicrotasks ()
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async exchange (/home/downloads/hopr-chat/index.js:105539:15)
at async handshake (/home/downloads/hopr-chat/index.js:118387:3)
at async Object.secure (/home/downloads/hopr-chat/index.js:171647:3)
at async Upgrader._encryptInbound (/home/downloads/hopr-chat/index.js:26351:12)
at async Upgrader.upgradeInbound (/home/downloads/hopr-chat/index.js:26109:11)
at async TCP.handleDelivery (/home/downloads/hopr-chat/index.js:25002:24)
(node:14998) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 13495)

Enhance general documentation with latest updates

  • Review README and update it with the most up-to-date changes.
  • Review docs.hoprnet.org > Advanced Setup and update it with the most up-to-date changes.
  • Include the information about the hopr-chatbot and make it part of the quickstart.
  • Include the information about the hopr-mazebot and make it part of the quickstart.

xDai Chain HOPR Release (Säntis)

Introduction

As we are looking to move forward with our incentivized testnet, we are pushing to get HOPR deployed into a quick validation network that allows us to integrate with some of the work we have done with our bounties and community. Thus, HOPR will move to deploy an incentivized testnet over xDai Chain as a way to allow us to have a more realistic understanding of the nature of our network and the token economics behind it.

Architecture (Draft)

The following represents the initial draft of the xDai Chain HOPR release architecture:

image

  1. HOPR Association deploys 4 smart contracts on the xDai Chain network: xHOPR Token, HOPR Channel, HOPR Minter and HOPR Faucet. The owner of the HOPR Faucet has minting capabilities, granted by HOPR Minter upon migration. Manually, the HOPR Association “registers” a Cover Node, a HOPR Node w/minting capabilities which monitor the network and its peers as to see which nodes are stable and deserve some xHOPR Tokens.

  2. (3 in diagram) HOPR Peers create the HOPR Network by connecting to selected Bootstrap Nodes from the HOPR Association. These peers need to be reachable by the Cover Node to be detected and granted xHOPR tokens, which will level them up to the next level.

  3. (2 in diagram) Upon detecting a stable HOPR Peer (i.e. good uptime, connectivity), the Cover Node issues them an amount of xHOPR Tokens. This enhances the peer to HOPR Node status.

  4. HOPR Nodes need to have a minimal amount of xDAI stake to be able to execute operations. Upon a successful check of their nodes status and staked xDAI, the Cover Node transfers xHOPR tokens to the HOPR Node, and gives them capabilities to open payment channels among other HOPR Nodes.

  5. HOPR Nodes can use xHOPR to open payment channels, which allows them to become intermediate nodes in the network. HOPR Nodes first use a Payment Node to initialize the Relayer Network, which is powered by all HOPR Nodes connected to this Payment Node.

  6. To retain their HOPR Node status, HOPR Nodes need to retain a tweet with their HOPR Node address which will be queried multiple times by the Payment Nodes, allowing them to protect the network from spammy accounts and ensuring their status as “validators” of the network.

Refactor type specs for assert based tests over fail-safe checks

Introduction

Right now, the connector-interface relies on /src/index.spec.ts as a way to test the types within the file are consistent with the ones needed by the connector. However, this requires fail-safe checks (i.e. running the code and see if it doesn't throw an error) rather than assertion (i.e. ensuring that the Interface of a function is the one expected). Let's change that.

Task Description

  • Refactor our spec files to with asserts that extend the generic ReturnType<T> or use tsd instead.

password prompt in `hoprd` hard to see among other output

When starting hoprd it asks for a password in the terminal but there might be more (debug) output that is printed right away which makes it hard to see that I have to type the password to start my node.

It would be a lot nicer to handle all user interactions via the Web UI, including the password prompt.

Define role of bootstrap server

Right now, the bootstrap server acts in the same way as every other node, meaning that it registers with the smart contract and is allowed to participate in a payment channel - if someone else is opening a payment channel to that node.

Important points:

  • in case the bootstrap node is indistinguishable from a "normal" node, it needs to be equipped with some funds to pay for the gas fees for registering with the smart contract.
  • same applies for any tickets resp. channels that we receive

Why pinning Node and Yarn versions?

Describe the bug

While understanding reasons for pinning versions in general,
when building, package settings for hopr-chat and hopr-core and others force you to downgrade node and yarn to older versions (while e.g. contained packages are not pinned). Is there a breaking change requiring?

Otherwise I would recommend to either use LTS or loosen up the pinned versions to either ~ or >.

To Reproduce
pull repo
yarn

Expected behavior
build without complaint

Add favicon

Describe the use case
Sets a favicon for leaderboard.hoprnet.org

The feature
Add a favicon

Network Testing / Design doc

Problem

At the moment we have a small number of 'unit' tests in core that are actually more like integration tests as they require a network interface. These are flaky, due to socket release timeouts, and very limited in scope.

Because we are a network protocol, we also face a large number of non-deterministic issues due to things like network timing, network topology (NAT, firewalls etc.) and at the moment we rely entirely on manually testing these issues.

As a consequence, we have many hard to diagnose bugs, frequent regressions, and no real metric on our reliability.

We would like to address this with a proper automated testing plan.

We should distinguish between the goals for the tests:

  • Concurrency Testing - testing the interaction between multiple nodes
  • Configuration Testing - testing that nodes with different settings / platforms / versions can interoperate safely. This includes different network topologies - NAT etc.
  • Functional Testing - testing that we do what we claim, that we are 'correct'
  • Load Testing - testing the performance of a node with a high number of connected nodes
  • Volume Testing - testing the throughput of the protocol

For completeness, let's also define some testing schemes:

  • unit testing - testing only a small unit of code in isolation
  • integration testing - testing how a unit of code interacts with the rest of the codebase / network
  • smoke testing - probabilistically testing the complete application to find errors fast.

Let's also define some scenarios that we wish to prevent with these tests:

  • We develop a new feature in core, however changes here cause regressions in our downstream applications.
  • We deploy a new version, however nodes on different IP's can no longer connect to one another.
  • We deploy a new version, however due to a difference in protocol, older clients will no longer work now behave in an undefined manner.
  • A new version causes nodes on a particular operating system to crash as it exceeds the number of allowed open sockets.
  • A version doesn't work on a particular type of network, as the firewall drops all packets due to some pattern of use.

Tasks

  • Allow pure unit testing of as much of core as possible. This should enable us to speed up the suite too.

  • Find a way to initiate run of tests on multiple machines, by spinning up a cluster of nodes, and initiating suite.

  • Find a way to distribute pre-funded databases to test nodes so we can persist funding between test runs.

Tooling [WIP]

Add timeouts for Web3.js

We found out that some of the calls to the blockchain might not come back.

Therefore we should add timeouts to all our web3.js calls and make sure that we can recover in case of a timeout

  • add timeout to web3.js calls
  • add debug logs in case of timeouts
  • specify a recovery mechanism

Use our own STUN server in our transport module

Bootstrap nodes expose a STUN server functionality.

Right now, we are using external STUN servers to figure out our public IPv4 addresses.

  • enable USE_OWN_STUN_SERVERS in src/network/transport/constants.ts
  • change setup of bootstrap servers to allow UDP requests on port 3478
  • add a test case that checks whether the new bootstrap node behavior

Fixing Testing on Hopr Core

At the moment we don't have reliable tests on hopr core. This is due to a number of reasons:

  • Ports are not closed in time, so we get EADDRINUSE
  • Tests rely on timeouts, for example a blanket 30 seconds to let ganache do it's thing.

I think part of the reason is that we are essentially using the hopr-core unit tests as integration tests for the entire project.

I propose the following:

  • Separate hopr-core unit tests from hopr integration tests.
  • hopr-core unit tests will not use a real chain. Instead we implement a DummyCoreConnector that has no-ops for all operations.
  • This means that hopr-core can be tested without relying on ethereum, or ganache.
  • We instead add these as integration tests.
  • Integration tests are run nightly, and at the moment will remain broken, fixing this is phase 2.
  • hopr-core unit tests should only test code in the hopr-core repo
  • these tests should be passing and bulletproof.
  • This doesn't address the ports not being closed in time - for this we need a detailed sweep of the tests making sure all of them call destructors on sockets etc.
  • I have a nasty feeling that there will still be issues due to bugs in our libraries - I spent a while yesterday trying to track down the dangling connectors in SimplePeer in Jest.
  • this does get us closer though...

Debug chinese connection for bootstrap server

Seems like our Chinese bootstrap server is unable to show other users connected in the network. We have some suspicions about why this might be the case. For instance:

  • Google servers seem to be unreachable from Alibaba servers. Our STUN server call could be blocked due to this. An alternative would be to ensure the .de stun server goes before the Google one.

Create a way to allow discovery of bot addresses

Describe the use case
Automatically find the bot addresses by querying an API or DNS

The feature
Create a way to allow discovery of bot addresses, either via embedded API or discovery per DNS

Define token governance roadmap for HOPR within a DAO

Ping @SCBuergel. Pretty much let's come up with the requirements that are needed from a technical perspective to allow minor governance on $HOPR holders for a DAO. Let's discuss existing frameworks (e.g. Aragon), as well as token economics to execute actions within the DAO.

Choose addresses to connect to in a smarter way

Right now, we collect all addresses of other, e.g. 12.34.56.78:12345, 127.0.0.1:9091, 192.168.0.23:1234.

As we can easily deduct, dialing 127.0.0.1:9091 makes no sense our HORP address is not the same that we are calling.

We can also deduct from nodes with public IPv4 that are different ours that we cannot reach them neither under a local address nor a under 127.0.0.1.

  • filter addresses before trying to dial them.
  • add the decision to the debug log

Hoprd -- run a single command and exit.

Describe the use case
Allow hoprd to be used for one-off commands, such as 'ping' or 'myAddress' and exit afterwards.

This will enable us to run automated tests that query the node for information.

The feature
Enable:

hoprd --dry-run
Check other cli flags, and verify that enough is known to start a node.

hoprd --run "myAddress hopr"
Outputs hopr address.

hoprd --run "myAddress chain"
Output ethereum / xdai id.

hoprd --run "balance native"
hoprd --run "balance hopr"
Return the according balance.

hoprd --run "ping alice"
Ping alice and exit.

Error: Invalid input parameters.

Describe the bug

Sending message to 16Uiu2HAmMXzv2jnSs4VrX69C48fq9U1dwG7xBRWZ5YvBnhd96nxJ ...
sign undefined channelBalance undefined
Trace: Error: Invalid input parameters.
    at ChannelFactory.create (/Users/peterbraden/repos/hoprnet/packages/core-ethereum/lib/channel/index.js:135:19)
    at processTicksAndRejections (internal/process/task_queues.js:85:5)
    at async Function.create (/Users/peterbraden/repos/hoprnet/packages/core/lib/messages/packet/index.js:142:29)
    at async /Users/peterbraden/repos/hoprnet/packages/core/lib/index.js:212:30
    at Hopr.sendMessage (/Users/peterbraden/repos/hoprnet/packages/core/lib/index.js:237:21)
    at processTicksAndRejections (internal/process/task_queues.js:85:5)
Invalid input parameters.

To Reproduce
Send a message from a funded node to an unfunded one, single hop.

Moving to a Monorepo

EDIT: Adding a List to track progress

  • Core

    • Imported
    • Workflows migrated
    • Tests integrated
  • Chat

    • Imported
    • Workflows migrated
    • Tests integrated
  • Hoprd

    • Imported
    • Workflows migrated
    • Tests integrated
  • Server

    • Imported
    • Workflows migrated
    • Tests integrated
  • Utils

    • Imported
    • Workflows migrated
    • Tests integrated
  • Demo Seeds

    • Imported
    • Workflows migrated
    • Tests integrated

As we've moved forward with HOPR, our multirepo setup has become unwieldy, and versioning across feature branches has become a bit of a nightmare. Additionally workflows have diverged across repos etc.

It's time to address this, and one clear solution is to move back to a monorepo.

Goals

  • Maintain git history, especially for core modules.
  • Allow work to continue in other modules while this migration is attempted, and maintain branches.

Plan

  • In each repository, move all files to the end path for the structure described below.
  • Remove git submodule from this repository.
  • Merge the module into this repository with --allow-unrelated-histories
  • Try this with our lesser used modules first to iron out any process kinks.
  • Add workflows back into this repository so everything is run by the github actions again.

Structure

  • Modules should be under ./packages for example:

    • ./packages/core
    • ./packages/hoprd
    • ./packages/chat
  • Documentation repositories should move into a docs folder:

    • ./docs/hopr-documentation
    • ./docs/whitepaper

Prior Art / Research

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.