So we can import once a jar with all these json files.

The distro needs different READMEs per package.

The plumtree algorithm specifies that if you receive a IHAVE message, you should be asking your eager peers for that message explicitly.

To do so, you send them a IWANT message with your info, and you hope to get back something in return (you run a timer, as implemented right now, and if you don't receive the full message, do a GRAFT request).

Publishing artefacts using publishToMavenLocal reveals an issue with dependency naming. While the artefacts are published using only the short module name some dependencies are declared using the Tuweni prefix.. I've been forced to hack a resolutionStrategy to deal with this but its far from ideal.

resolutionStrategy.eachDependency {details ->
  if(details.requested.group == "org.apache.tuweni" && details.requested.name.startsWith("tuweni")) {
        def shortName = details.requested.name.replaceFirst("tuweni-","")
        details.useTarget("org.apache.tuweni:"+ shortName +":"+ details.requested.version)
      }
}```

Run tests on Windows, making sure all tests pass on all major OSes.

The assemble step picks up the distsrc. Would be good to also pick up the binary distro, and make it available for folks to play.

Account-based blockchains store information in cryptographically-provable structures such as Patricia Merkle trees.

These structures don't lend well to indexing.

Motivation
Allow O(1) retrieval of information for block header data, account data of any account-based blockchain.

Scope of work

• Work has started in the module blockchain-indexing. Need to add accounts.

Issue in the implementation of the Bytes32.wrap(Bytes value, int offset) method:

    Bytes bytes = Bytes.fromHexString(
        "0x00112233445566778899aabbccddeeff00112233445566778899aabbccddeeff00112233445566778899aabbccddeeff");
    MutableBytes mutableBytes = MutableBytes.create(48);
    bytes.copyTo(mutableBytes);
    System.out.println(Bytes32.wrap(mutableBytes, 1));

Expected output:

0x112233445566778899aabbccddeeff00

Actual output:

0x00112233445566778899aabbccddeeff

From 1.0.0 release thread:
It a little unclear from the names if if gossip and relayer are binary or source artefacts, you might want to correct that.

Between version 0.9 and 0.10, the result of HashTreeUtil.hash_tree_root for VECTOR_OF_BASIC has changed. This has caused reference test failures in Teku suggesting the new value is incorrect.

Test case to demonstrate:

  @Test
  public void shouldHashTreeRootCorrectly() {
    final Bytes input =
        Bytes.fromHexString(
            "0xA99A76ED7796F7BE22D5B7E85DEEB7C5677E88E511E0B337618F8C4EB61349B4BF2D153F649F7B53359FE8B94A38E44C");
    final Bytes expected =
        Bytes.fromHexString("0x89E40BFF069E391CA393901DA3287BBE35DC429265927ABE3C3F06BEC8E0B9CD");
    final Bytes32 actual = HashTreeUtil.hash_tree_root(SSZTypes.VECTOR_OF_BASIC, input);
    assertEquals(expected, actual);
  }

The expected value is the one that 0.9 gives.

Tuweni version 1.1.0

Legitimate test case

    Bytes concatBytes = Bytes.wrap(
        Bytes.wrap(
            Bytes.wrap(new byte[32]),
            Bytes.wrap(new byte[6])
        ).slice(0, 37),
        Bytes.wrap(new byte[1]));
    byte[] bytes = concatBytes.toArrayUnsafe();

throws

java.lang.IllegalStateException: element sizes do not match total size
	at org.apache.tuweni.bytes.ConcatenatedBytes.copyToUnchecked(ConcatenatedBytes.java:206)
	at org.apache.tuweni.bytes.ConcatenatedBytes.toArray(ConcatenatedBytes.java:197)
	at org.apache.tuweni.bytes.Bytes.toArrayUnsafe(Bytes.java:1420)

EIP-1459 introduces a novel way to store information about records in a cryptographically verified tree of records as TXT DNS records, to be hosted under one or more domains.

Motivation
DNS is the most performant distributed system in the world. Right now, p2p systems use direct UDP connections to transmit metadata information to perform discovery.

By relying on DNS, the systems become more reliable, less open to eclipse attacks. This requires trusting the authority which updated the DNS record, and to know which DNS record to look up for. This can be considered an acceptable tradeoff.

Scope of work
Implement reading the ENRs from a DNS record.

Implement keeping up to date with DNS to update when the tree is modified.

Implement writing a set of DNS TXT records according to the logic.

The current plumtree implementation keeps track of the state of the last 1000000 messages it received.

This is rather memory intensive - and can be gamed by adversarial behaviors by recycling and running replay attacks with a bigger batch of messages.

We can use bloom filters to identify messages instead.

The work consists in:

-Find a bloom filter in java or implement one. (there is a bloom filter in the eth component that can be generalized under its own library, maybe?)

-Use it to identify messages as an option to the LRU map

-Test the tests still pass

-Allow that bloom filters need to still be rotated over time.

Would be also good to move the existing LRU map to a mechanism external to plumtree state, so it can be backed by a DB or a cache specified by the user.

Create a DSL written in Kotlin that calls to the OpcodeRegistry, allowing folks to create their own code that compiles down to EVM opcodes.

The current Bytes implementation uses direct buffer allocation.

This gets expensive when you perform a lot of small allocations quickly. A better way might be to approach a large direct buffer and allocate portions of it to smaller bytes objects.
An other set of approaches is to use a file-backed bytes object or simply the heap.

Offer options to use:

• Direct
• Heap
• File
• Large direct cache

This can also be specialized for 32 bytes objects vs general bytes buffer.

Reference:

https://www.fusion-reactor.com/evangelism/understanding-java-buffer-pool-memory-space/

Shamir secrets are used to store secrets and share them with multiple parties. Only by associating all or some of the pieces of the secret together can the recipient recreate the original message.

Motivation
Shamir secrets are useful tools for creating partial messages that can be sent over gossiping networks. They preserve a form of privacy.

Scope of work

• Review existing Shamir secret implementations out there. Some exist in Java but are no longer maintained.
• Implement an easy Shamir secret generation and recovery.

Tests that rely on Sodium being present are safely deactivated if it is detected to be missing.

Right now CI runs without Sodium being installed. It would be good to have Sodium installed during CI so it can run all the tests.

Migrated from https://issues.apache.org/jira/projects/TUWENI/issues/TUWENI-4

We have the ability to generate javadoc and dokka docs for Apache Tuweni, but it's not been maintained.

• Look at it and make sure this works well again.
• Take the latest bin release and make it part of the website, by unzipping it and adding it to the gh-pages branch of the repo.

Some of the components seam to have use cases outside this project. Can they be published separately. E.g.:

• low-level bytes library,
• serialization and deserialization codecs (e.g. RLP),
• various cryptography functions and primatives, and
• other helpful utilities

Create a very simple bootnode script that just runs discovery and has no rlpx tcp port open.

We have issues when running builds with the current setup.

A proposition is to add a Dockerfile to the source distribution that would allow folks to simply run the build without issues. This would facilitate the voting process.

Create an example of configuration that uses the Ethereum client to connect to mainnet.

It should use a toml file for configuration.

I just updated from 0.10.0 and one of the things I noticed was the pom file changes:

The packaging listed in the pom looks wrong, it's listed as pom and it likely should be jar (or not defined)
https://repo1.maven.org/maven2/org/apache/tuweni/tuweni-bytes/1.1.0/tuweni-bytes-1.1.0.pom

This is minor, but... the "name" field is wrong in the project poms, this field is commonly used in reports. for example usage of the maven-notice-plugin uses the name field. The "name" tuweni-bytes is now task ':bytes:jar' property 'archiveBaseName'

Generated output diff example from the maven-notice-plugin:

-  tuweni-bytes under The Apache License, Version 2.0
-  tuweni-crypto under The Apache License, Version 2.0
-  tuweni-io under The Apache License, Version 2.0
-  tuweni-units under The Apache License, Version 2.0
+  task ':bytes:jar' property 'archiveBaseName' under The Apache License, Version 2.0
+  task ':crypto:jar' property 'archiveBaseName' under The Apache License, Version 2.0
+  task ':io:jar' property 'archiveBaseName' under The Apache License, Version 2.0
+  task ':units:jar' property 'archiveBaseName' under The Apache License, Version 2.0

NOTE: I'm just providing the above diff as an example of how the name from a pom file may be used.

The current gossip implementation allows that attributes be passed into the message when it is published.

-We need to add attributes to the function listening for payloads.

-We can probably also add the sender peer to the function listening for payloads, so the user if they so choose can log that they received from that peer first.

Also, attributes are currently a string which is directly appended to the message headers. We could potentially make this a Map<String,Object> that can be mixed in to the message headers and serialized to BSON.

Genesis ceremonies involve having all participants take a JSON file which defines how the network will behave.

Motivation

Genesis files are easy to get wrong. They don't have a proper spec (question) and are written in JSON.

Scope of work

• Allow creation of Genesis files using a tool.
• Allow formatting of Genesis JSON contents using a specific formatter so it can be hashed and verified across peers.

Migrated from https://issues.apache.org/jira/projects/TUWENI/issues/TUWENI-32

TOML parsing does not work without adding extra ' quotes.

We get this org.apache.tuweni.toml.TomlInvalidTypeException: Value of 'plugin-kafka-log-filter-topics' is a array

This mean the following config does not work:

plugin-kafka-log-filter-topics = [
    [ "0xd3610b1c54575b7f4f0dc03d210b8ac55624ae007679b7a928a4f25a709331a8" ]
]

This is confusing for users since they don't have have to put extra quotes for the command line and they have to for the TOML config file.

Moreover this is a valid TOML configuration file without extra quotes.

More context here:

https://github.com/PegaSysEng/plus/pull/177

LES stands for Light Ethereum Subprotocol and is described at length here.

Motivation
LES allows to easily synchronize the chain, at the cost of running transactions in the EVM, and relying instead on merkle proofs.

LES is partially implemented by Apache Tuweni. Account data is not yet available.

Scope of work

• Implement merkle proofs for LES
• Implement, if that makes sense at all, the budget mechanism that allows a form of rate limiting for LES.

As noted during 1.0.0 release, the test artifacts used to test eth 2.0.0 are very large and encumber the release.
Also you might want consider removing the large amount of test resource form the source release, IMO users generally want to build the release not test it ands it on the rather large size with some test file being over 1Mb.

The ExpiringMap class can be used as a concurrent hash map with an explicit way to handle expiration.
It would be great to have a way to set a global default timeout for any puts.
It would be good to also have expiration callbacks that execute whenever a timeout is reached.

The encoding (serialisation) fo list of booleans seem to produce

1. a unit32 with the size of the list
2. a sequence of Bytes each one encoding a bit in the list.

This is according to the implementation

and test

If this is correct, it does not seem to correctly implement the format of the Eth2.0 spec https://github.com/ethereum/eth2.0-specs/blob/dev/ssz/simple-serialize.md#bitlistn

which specifies that the bits should be put together in Bytes (which seems to a good idea as produces more compact encodings).

Is that going to be problem when communicating with clients that do implement the spec?

For hashes some form of a short hash string method, for informative logs. Like this:

https://github.com/hyperledger/besu/blob/0bd50239f0f6e4a479c2d7be3804122f736bf9ce/ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/sync/BlockPropagationManager.java#L345

Trying to package an existing project that's curiously invested in Tuweni has met the regrettable obstacle due to there being no Auto-module-name or module-info.java in the project jars. I've had a look at existing issues and I don't see anything related.

Somehow the distribution archives contain several times the same file, which is a waste of space and produces warnings when folks expand the archives.

Some tests do not work well under Windows:

org.apache.tuweni.hobbits.UDPTest > testTwoEndpoints(Vertx) FAILED
   Caused by: java.net.BindException
org.apache.tuweni.hobbits.UDPTest > testTwoUDPConnections(Vertx) FAILED
   Caused by: java.net.BindException
org.apache.tuweni.hobbits.RelayerTest > testUDPRelay(Vertx) FAILED
   java.lang.NullPointerException at RelayerTest.kt:101
17 tests completed, 3 failed, 1 skipped
Task :io:test
org.apache.tuweni.io.ResourcesTest >
shouldIterateResourcesOnFileSystemAndInJars(Path) FAILED
   java.lang.RuntimeException at ResourcesTest.java:96
       Caused by: java.net.URISyntaxException at ResourcesTest.java:96
19 tests completed, 1 failed
Task :kv:test
org.apache.tuweni.kv.MapDBKeyValueStoreSpec > describe a MapDB-backed key
value store FAILED
   java.nio.file.FileSystemException at KeyValueStoreSpec.kt:202
55 tests completed, 1 failed```

Stratum is the venerable mining protocol the Ethereum community inherited from Bitcoin.

Stratum is a fragmented protocol with many small differences.

Motivation
There is no good way to manage miners from a client right now, as there is quite a bit of jitter and statefulness in the way Stratum is implemented.

Scope of work

• Write a Stratum specification for the existing support.
• Write a prospective Stratum specification (using protobuf) for prospective support.
• Implement a round-robin stratum proxy that can take connect to multiple Ethereum clients at once.
• Implement a way for multiple miners to connect to the proxy.
• Implement a round-robin or master/slave strategy to ensure uptime of miners and the client.

• expose network interface for service - whole enode
• expose the agreed capabilities for connection
• expose peer info from hello message
• subscribe to new connections
• subscribe to disconnections
• support maintaining connections to peer
• subscribe to capability
• keep peer connected
• remove peer from wireconnectionrepository when disconnected.
• enr get public key bytes

SimpleBlockchainInformation allows to create information about the eth subprotocol to other clients.

Use a fixed set for mainnet, goerli, rinkeby and ropsten.

An ETH proxy is a service running the Ethereum wire protocol, allowing one or more clients to run behind it while keeping connections open.

Motivation
The current peer-to-peer networking does not allow hot restarts of Ethereum clients.

We want to implement just enough logic we can keep connections open with existing peers, while we have the ability to create controller/worker or controller/controller replication schemes.

Scope of work

• Implement ETH proxy to capture and replay most important messages - status, in particular.
• Implement distribution of executable running ETH proxy

SecureScuttlebutt is a decentralized social network. ENR stands for Ethereum Node Record, and allows to specifically identify a node on the network.

Motivation
SecureScuttlebutt does not currently allow discovery through peer discovery protocols such as devp2p discv4 or discv5. Ethereum allows discovery by connecting to boot nodes, then allowing connections to neighbors to form a Kademlia distributed hash table.

We want to allow SecureScuttlebutt to permit discovery. ENRs allow to publish useful metadata information about a node, such as which network it would participate in.

Scope of work

• Identify the ENR attributes required for SSB
• Implement and test ENRs for SSB
• Propose to SSB community

EIP-152 introduced a new Blake2b precompile contract for Ethereum.

This blake2b compression is implemented by Sodium, but Ethereum chose to use a different sigma.

Motivation
Current implementations of blake2b are client-specific - there exists one for each client, so far. They also are implemented in Java where ideally they should be implemented in low-level C, and introduced via bindings.

Finally, this could be easily done by adopting and extending the libsodium primitives.

Scope of work

• Identify how to best implement a fast blake2b function, probably by reusing libsodium approaches.
• Bundle a C low-level primitive for this precompile with a simple Java wrapper - JNR, most likely.

The current hash functions use BouncyCastle, but Sodium is faster since it is native.

It's also possible to fallback to BouncyCastle if Sodium is detected to not be present in the system.

The EVM is written in Kotlin.

Write a simple test in Java to make sure it can be called from Java. Create a doc to show how to use in Java code.

The ETH subprotocol is the main subprotocol used by the wire protocol of Ethereum under devp2p.

Motivation
There is no independent ETH subprotocol implementation that can allow tools to perform connections and recover information from peers by calling status for example.

Scope of work

• Implement ETH subprotocol
• Implement each message
• Implement a test suite for the subprotocol.

EVMc stands for EVM (Ethereum Virtual Machine) connector, and allows multiple EVM implementations to be exposed behind a common interface.

EVMc has bindings in C, Go, Rust, Java.

Motivation
The EVMc bindings in Java are currently made available on the EVMc repository, but require to be distributed and made easily available with an existing EVM implementation such as evmone.

Scope of work
Build a distribution of the EVM using EVMc and EVMone.

To reproduce

    ByteBuf buf = Unpooled.buffer(100).writeByte(1);
    Bytes bytes = Bytes.wrapByteBuf(buf);
    System.out.println(bytes.size());
    System.out.println(bytes.toArray().length);

Expected behavior

1
1

The buf content is only 1 byte, other 99 reserved bytes are not the part of the buffer content

Observed behavior

100
100

Tuweni (1.2.0) has support up to Libsodium 1.0.16. Libsodium has newer versions available - 1.0.17 and higher.
https://github.com/jedisct1/libsodium/blob/master/ChangeLog

It would be nice to have support for newer versions of Libsodium also.

Related discussion thread is here:
https://mail-archives.apache.org/mod_mbox/tuweni-users/202011.mbox/%3cCAEShzP57Eu1U7EHSmUFHk+VfoMYc2sosVxvQ7x8bMcdXFQqUkQ@mail.gmail.com%3e

Follow instructions at https://www.apache.org/dev/crypto.html

When attempt to create a FileBackedFingerprintRepository, if the supplied path is a simple filename (eg Path.of("file.tmp"), it effectively has no parent directory, and thus the call to createDirectories() (on line 39) is invoked with a 'null' parameter, resulting in an NPE.

Current workaround is to ensure it the FileBackedFingerprintRepository is instantiated with an absolutePath

Apache Tuweni doesn't have a logo.

Tuweni is taken after the Lakota language word Tuweni which both means "nobody" and "never".

See our FAQ here: https://tuweni.apache.org/faq/

See also reference to tuweni here: https://www.wolakotaproject.org/lakota-pronunciation-glossary/

A logo would incorporate the ASF feather and the whole name "Apache Tuweni".
See press kit here: https://apache.org/foundation/press/kit/#:~:text=The%20%E2%80%9CPowered%20By%20Apache%E2%80%9D%20logo,the%20new%20ASF%20brand%20identity.

All ASF logos here for inspiration: https://apache.org/logos/