• Remember 99% of performance is lost if your switch from native Kotlin multithreads doing communication towards network-of-Dockers containers. Plus it won't scale to 10k instances.
• please laser focus on having 1 bash script on "bare metal" running compiled Kotlin (native)
• First get a example running 100x sockets
• Then bash to create 100x full IPv8 processes (or even try 1000!) (test when laptop is out of memory)
• No sharding, IPv8 should form a solid random network
• Final sprint: DAO code

from tribler.

• option to scale Android VM using QEMU. fun link: MegaDroid: 300,000 Androids clustered together to study network havoc
• https://github.com/Tribler/Bitcoin-Regtest-Server
• (preffered method) get JVM cmdline example running.
  • slowly make a isolated cmdline project which is compatible with Android Superapp
  • No GUI 💻
  • start making socket connections in IPv8/JVM
  • start joining shared wallet
  • join a given DAO, vote, etc.
  • support all features using code duplication 😅

Minimal sprint goal: all your laptops inside 1 IPv8 community x 4 (start multiple IPv8 client). try to do also crypto stuff, not just sockets.

from tribler.

Demo Week 4:

from tribler.

Pointers for next week:

• Focus on pure JVM clients / ignore phones --> look at serviceId
• Make code more flexible --> no code adjustments for changing clients
• Make single scripts that launches x amount of clients and performs default flow:
  • All clients make btc wallet and request btc from central node
  • First client makes shared wallet
  • All clients join shared wallet --> voting is automatically performed
• End goal: create graph that shows timings for the joining of all clients
  • Don't forget about connection limit parameter!

{added} Demo: Regtest block download operational. Get BTC inside superapp needs to work, reproduce stand-alone cmdline. Empty wallet means error creating DAO

from tribler.

12-03-2024 update:

• web-socket based server to control peers
• adding proposals and voting to the JVM environment
• make the JVM environment work for multiple peers
• (ongoing) remove the blocking-while loop that is waiting for votes when joining a DAO

from tribler.

Cleaning up the github repository

from tribler.

12-03-2024 node.js server to trigger events on instances:

from tribler.

This sprint's update:

• Created a visual interface for the coordinating server so that we can test functionalities
• JoinDAO works
• Voting is done automatically
• We gathered some measurements of how fast all the instances join the same DAO. We used 5 instances in our tests. The main problem is that the time (in ms) is dependent on how fast the voting happens (how fast the instances that have to vote realize that they have to vote)
  [{"id":0,"time":0},{"id":1,"time":5526}, {"id":2,"time":5500},{"id":3,"time":5550}, {"id":4,"time":5493}].

from tribler.

We are also working on the removal of the single coordinator from signature collection process, by removing the blocking while loop that waits for signatures on the client side with a new event based system:

Future plans

• Finish the removal of the single coordinator.
• Test against the current implementation.
• Networking dashboard: activity principle grid of status of each of the 25 connected IPv8 peers.
• Random gossip with all partial signatures in binary format.

from tribler.

• push the JVM instances to 100 possibly
• fully concurrent, but linear growth of the DAO. Join in order of sequence number. remove all magic (5 second) waits.
• current code the joining client waits for the required votes and is the blocking element to conduct the join transaction. Also the blocking element for any proposed transaction is the original proposing node.
• prior superapp code waits for threshold of DAO joining. Then it moves forward in protocol state.
• usage of unsafe custom multi-sig for both threshold-votes and threshold-transaction
• test with base line of everybody required to vote and another test with threshold.
• Mid-term evaluation
  • "bit on the edge", orange alert.
  • no figures, no running baseline test
  • no changes yet to protocol or performance boosting

from tribler.

Progress check 25-03-2024

• Removed single coordinator, the transaction is executed by the last peer who interacted with the proposal.
• Refactored JVM Project to run on docker and (not tested) kubernets.
• Basic netty-incubator-codec-http3 implementation + discussion about it.
• Run baseline with 30 peers, too slow, inconclusive results

from tribler.

Solid progress last sprint! 🎉

• Work on graph
• Fix synchronization bug
• PoC sharding concept
• Do a deep dive into IPv8 stack --> you can change anything you want to improve performance.

from tribler.

Progress check week 8:

Short description:
Focus on testing multiple techniques to achieve an increase in speed and/or number of connecting peers.
Detected possible improvements regarding the networking on trustchain ( preliminary results below )

• Sharding approach ( Join vs Transfer shards, to be discussed )
• Ran into hardware problems ( one laptop not enough even for 20 peers, possible solutions to expand the experimental setup under discussion) [ Docker Images vs Bash Script running x instances (running shadowJar to compile a jar and run it) ]
• Problems with the Bitcoin server

Findings:

• Bandwidth => the real bottleneck. Experimental setup: Peers broadcasting 1KB packages using the current broadcast algorithm (25-fanout broadcast) and using a "persistent gossip" ( TTL = untill every peer receives the message ). Run multiple trials with different number of peers and measure the number bandwidth usage of the docker network and the time for every peer to receive the messages.

Results (charts below: X-axis=number of peers, Y-axis=bandwidth usage in KB/ milliseconds):
As expected the bandwidth usage of the gossip mechanism is drastically lower than the current approach. The current algorithm for 15 peers peaked at around 1.2MB of band usage, which is a lot compared to only 228KB used by the gossip.

Unexpectedly, for low number of peers the gossip algorithm performed better speed-wise.
This result can be interpreted in two ways:

• By the nature of the gossip mechanism, we can expect this advantage to dissipate with the increase of the number of peers.
• The high bandwidth consumption of the 25-fanout broadcast would congest the network such that the peer messages would be delayed and the gossip would maintain its advantage with the increase of number of peers.

Future improvements proposal:

• The use of a DHT algorithm like Kadmelia for a more efficient topology.
• Archival mechanism for trustchain example (old nodes become "axioms" and can be archived so the chain size remains manageable in time.

GIT: https://github.com/Kheoss/CSE4110_jre

from tribler.

Update 12/04/2024

• managed to get to 100 peers
• tried compiling with kotlin-compiler-1.9.23 ( no IPV8 linkage)
• run existing experiments (on peers > 22)

Current setup:

• Export JAR
• Run multiple instances
• STOP all java instances

Observation:
Slow discovery, takes a while for each peer to discover other peers.
Performance increase on windows OS (?)

from tribler.

• running code is a requirement of this course
• Extension of 1 week to wrap up this course
  • "laser focus" on Kotlin failed sadly
  • still JVM route, less useful for our lab, lacks scalability
• Objective for running code
  • Infrastructure for DAO scalability research
  • 1 script to grow DAO until it breaks
  • Build upon completed task: concurrent voting on new member
    • that means faster join
  • DAO growing from 1 member, 2 members, 3 members... 100 members (but likely break somewhere, memory, UDP signature size)
  • 100 peer overlay, grow 1-by-one in Bash script
• no superapp modification, laser focus on DAO scalability test infra
• performance expectation: starting an IPv8 overlay or joining of single peer should only take 30 seconds.

Grading session --- 26/04

• Running code required for passing grade

  • JAR file (running natively)
• Finalize implementation
  • Create tests
• Make discussed graphs
• Write README-style report - check e.g. currencyii for inspiration.
• Make pull request with code
• Demo of your functionalities

from tribler.

Progress check 19-04-2024

• compiled the orchestrator into a CLI app.
• improved UI
• rollback to old voting
• fix shell script for linux ( 50 peers on Linux )
• add timers for "SYNC" ( when a peer joins a wallet all the other peers are "DESYNC" untill they receive the new data and become "SYNC")
  

from tribler.

Final notes

• First graph
  • 100 clients
  • X number of clients
  • Y the time
• Second graph
  • Connection graph
  • 
  • Tribler/py-ipv8#979 (comment)
  • Y #connections
  • X time
• Finish README
• Prepare demo
• Make PR before meeting
  • Put all code in a submodule of trustchain-superapp.

from tribler.

readme :https://github.com/Kheoss/trustchain-jvm-sandbox/blob/2244cd7a959a0046e33c68e4c77db8fa018ca22e/README.md

Fork: https://github.com/Kheoss/trustchain-superapp/tree/jvm_compatible_sandbox

from tribler.