consensysmesh / cava Goto Github PK
View Code? Open in Web Editor NEWConsenSys core libraries for Java & Kotlin
License: Apache License 2.0
ConsenSys core libraries for Java & Kotlin
License: Apache License 2.0
When consuming a scuttlebutt stream, there are periodic sodium OutOfMemory
exception errors while decrypting the messages received as responses from the server for random messages received:
2019-04-02 09:37:20.086+0000 ERROR [localhost:8008] Sodium.sodium_malloc failed allocating 24
java.lang.OutOfMemoryError: Sodium.sodium_malloc failed allocating 24
at net.consensys.cava.crypto.sodium.Sodium.malloc(Sodium.java:224)
at net.consensys.cava.crypto.sodium.Sodium.dup(Sodium.java:262)
at net.consensys.cava.crypto.sodium.Sodium.dup(Sodium.java:273)
at net.consensys.cava.crypto.sodium.SecretBox$Nonce.fromBytes(SecretBox.java:230)
at net.consensys.cava.crypto.sodium.SecretBox$Nonce.fromBytes(SecretBox.java:213)
at net.consensys.cava.scuttlebutt.handshake.SecureScuttlebuttStream.decryptMessage(SecureScuttlebuttStream.java:107)
at net.consensys.cava.scuttlebutt.handshake.SecureScuttlebuttStream.decrypt(SecureScuttlebuttStream.java:85)
at net.consensys.cava.scuttlebutt.handshake.SecureScuttlebuttStream.readFromServer(SecureScuttlebuttStream.java:53)
at net.consensys.cava.scuttlebutt.handshake.vertx.SecureScuttlebuttVertxClient$NetSocketClientHandler.handle(SecureScuttlebuttVertxClient.java:98)
at io.vertx.core.net.impl.NetSocketImpl$DataMessageHandler.handle(NetSocketImpl.java:392)
at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:225)
at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:123)
at io.vertx.core.net.impl.NetSocketImpl.handleMessage(NetSocketImpl.java:370)
at io.vertx.core.net.impl.ConnectionBase.handleRead(ConnectionBase.java:386)
at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:320)
at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:43)
at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:188)
at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:174)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Steps to reproduce:
postMessageTest
test in the PatchworkIntegrationText
file in the mux test package to post 20,000 messages by changing the for loop to iterate more and run the test. I'm not sure the minimum number of items in a stream required to make the error happen.streamTest
test in the PatchworkIntegrationTest
file in the mux package.Hi there,
I hava added cava to my project, add then I used the SECP256K1 class to sign my data, but some errors happened. Is there anyone met the problem and tells me why? thx
The error message is like this:
Exception in thread "main" java.lang.ExceptionInInitializerError
at net.consensys.cava.crypto.SECP256K1$PublicKey.fromSecretKey(SECP256K1.java:527)
at net.consensys.cava.crypto.SECP256K1$KeyPair.fromSecretKey(SECP256K1.java:684)
at one.contentbox.boxd.samples.SignSample.main(SignSample.java:40)
Caused by: java.lang.IllegalStateException: BouncyCastleProvider is not available, see https://www.bouncycastle.org/wiki/display/JA1/Provider+Installation
at net.consensys.cava.crypto.SECP256K1$Parameters.<clinit>(SECP256K1.java:111)
... 3 more
Caused by: java.security.NoSuchProviderException: no such provider: BC
at sun.security.jca.GetInstance.getService(GetInstance.java:83)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:206)
at java.security.KeyPairGenerator.getInstance(KeyPairGenerator.java:279)
at net.consensys.cava.crypto.SECP256K1$Parameters.<clinit>(SECP256K1.java:109)
... 3 more
My code is like this:
String privKeyHex = "29fbf01166fc31c941cadc1659a5f684f81c22c1113e5aa5b0af28b7dd453269";
BigInteger privKeyBigInteger = new BigInteger(privKeyHex, 16);
String data = "46f7f67f515e7d053525459991c7a6b3e673950809af776324f46d91df9e600d";
SECP256K1.Signature signature = SECP256K1
.sign(Hex.decode(data), SECP256K1.KeyPair.
fromSecretKey(SECP256K1.SecretKey.fromInteger(privKeyBigInteger)));
byte[] data = signature.bytes().toArray();
Currently SSZWriter
supports serializing a number of single values as well as an [object]...
varags list. While this works for single elements and primitive arrays, a java.util.List must be converted into a primitive array first. Native support of java.util.List types would be a nice to have.
The implementation of the CoroutineLatch
(https://github.com/ConsenSys/cava/blob/master/concurrent-coroutines/src/main/kotlin/net/consensys/cava/concurrent/coroutines/experimental/CoroutineLatch.kt) is naive, synchronizing on most operations. It would be great to re-implement this using a more efficient algorithm, such as CLH queue locks.
addFingerprint()
writes to fingerprints
with synchronization, but reads it without synchronization through contains(String, Bytes)
. This seems to be a bug. If asymmetric read/writer locking is wanted, then probably a ReadWriterLock should be used.
I don't have steps to reproduce this, but I think it might be an issue (from reading the code.)
A scuttlebutt RPC message has a 34 bit header, followed by a variable length body. When bytes are received over the socket, the current implementation of scuttlebutt-handshake
assumes that it is a complete message in the byte buffer handed over by Vertx.
Depending on network conditions, application load, etc, this could theoretically be only part of a message in this buffer (or perhaps even 1 RPC response and a little bit of another.) Perhaps this could be re-implemented to make sure we buffer up full messages first before handing over the bytes to the receivedMessage
method.
We implemented detached signatures with Sodium.
On top of that, it would be nice to implement the combined and multi-part methods. Those are described here: https://download.libsodium.org/doc/public-key_cryptography/public-key_signatures
Signed Roots allow calculating the merkle hash of a subset of a SSZ container, specifically to allow signing of a container while excluding signature fields.
While this definition signed_root(container, "signature") = merkle_hash([hash_tree_root(sub_object_1), ..., hash_tree_root(sub_object_N)])
has been deprecated in further versions of the Eth2 SSZ spec, I see no harm in allowing public callers to use SSZ#merkleHash
directly for compatibility.
As reported by users, backwards-compatibility with Java 8 is broken.
We will fix this ASAP.
Makes it easier for the non-Gradle crowd to use your library.
I put this off a long time for my personal projects because I believed it was a hassle, but finally I went though the steps recently and it's really not that bad. It's basically just creating a Jira account and filing some tickets to claim ownership of the artifact ID, then pushing the sync button on the JCenter site.
Running ./gradlew
results in this warning:
> Configure project :
The DefaultSourceDirectorySet constructor has been deprecated. This is scheduled to be removed in Gradle 6.0. Please use the ObjectFactory service to create instances of SourceDirectorySet instead.
Looks like it will be fixed in an upcoming release:
There is no support for setting endianness at this time in the SSZ library.
The spec was modified to allow endianness when serializing individual values.
Hi,
I am wondering if I can I use these libraries to implement Enterprise Ethereum Client according v2 Spec or do I need to fork Ethereum code?
Thanks!
> Task :toml:test
net.consensys.cava.toml.TomlTest > shouldParseString(String, String)[3] FAILED
org.opentest4j.AssertionFailedError at TomlTest.java:81
net.consensys.cava.toml.TomlTest > shouldParseString(String, String)[8] FAILED
org.opentest4j.AssertionFailedError at TomlTest.java:81
net.consensys.cava.toml.TomlTest > shouldParseString(String, String)[23] FAILED
org.opentest4j.AssertionFailedError at TomlTest.java:81
net.consensys.cava.toml.TomlTest > testHardExampleUnicode() FAILED
org.opentest4j.AssertionFailedError at TomlTest.java:512
net.consensys.cava.toml.TomlTest > shouldParseArray(String, Object[])[8] FAILED
org.opentest4j.AssertionFailedError at TomlTest.java:555
212 tests completed, 5 failed
> Task :toml:test FAILED
FAILURE: Build failed with an exception.
Hi! Really nice TOML parsing library -- thanks for sharing it under a business-friendly license. I especially like the detailed messages for syntax validation errors. I thought I should report an issue I ran into while getting started.
In a project that uses net.consensys.cava:cava-toml:0.2.0
, parsing a TOML config file fails with the following exception unless the Antlr4 runtime is added as an explicit dependency:
Exception in thread "main" java.lang.NoClassDefFoundError: org/antlr/v4/runtime/CharStream
at com.couchbase.connector.elasticsearch.Config.main(Config.java:147)
Caused by: java.lang.ClassNotFoundException: org.antlr.v4.runtime.CharStream
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 1 more
The workaround is to manually add a dependency on the Antlr4 runtime. For gradle, this looks like
compile 'net.consensys.cava:cava-toml:0.2.0'
compile 'org.antlr:antlr4-runtime:4.7.1'
It would be nice if the Maven POM for the cava-toml
library expressed a transitive dependency on the Antlr4 runtime. Better yet, embed the runtime in the cava-toml
library as a shaded/shadowed dependency.
@atoulme and/or @cleishm, if you wouldn't mind giving a little thought to the below proposal, I'd be most appreciative. I've spent a little bit mulling this over and plan to take a first stab at a PR shortly, but in the meantime an extra set of eyes doesn't hurt.
Description
The SSZ spec was rewritten as part of the eth2-specs v0.5.1 release to include both fixed-length and variable-length arrays/lists. Following the python notation used in the eth2-specs, and for the sake of clarity, fixed-length arrays/lists will be referred to as "tuples" and variable-length array/lists will be referred to as "lists".
(Not so) Quick Background
uintN
(where N
is in 8, 16, 32, 64, 128, or 256)bool
byte
== uint8
tuple
, represented as [type, N]
list
, represented as [type]
bytes
== ["byte"]
bytesN
== ["byte", N]
The implications of the above are that "list" requires a length mixin, where as a "tuple" does not.
Proposed Changes
byte
s are serialized (i.e. when recursively serializing a Cava Bytes
type), they should be serialized as a uint8
.Bytes
type of length 1 should be serialized as a uint8
. [still undecided on this one]Bytes
type of length >1 should be treated and serialized as a "list".BytesN
types (i.e. Bytes32
) should be treated and serialized as a "tuple".Bytes...
(varags list) as either a "list" or a "tuple"
length
parameter.List<? extends Bytes>
as either a "list" or a "tuple".
length
parameter.Additional Information
See https://github.com/ethereum/eth2.0-specs/blob/v0.5.1/specs/simple-serialize.md.
List support is currently missing for convenience methods of the form write(U?)(Long?)(\d*)IntList. Functions of this form should also support varargs and a java.util.List.
This was just changed in the SSZ spec - it is now possible to have lists with variable element length.
See this PR for context:
ethereum/consensus-specs#484
Hi - thanks for working on these Java libraries for using the scuttlebutt protocol!
I have encountered a bug while attempting to use the scuttlebutt-handshake
library to
whoami
which simply reports back the server's public key. I also tried createFeedStream
by copying and pasting the body from the protocol docs and using a key for someone the scuttlebot instance is following.receivedMessage
should be called back with the response bytes from the RPC request.The secret handshake appears to happen.
Any encrypted RPC response is given - however, the SecureScuttlebuttStream
class fails to decrypt the header ( null
is returned at this line, then a NullPointerException is thrown): https://github.com/ConsenSys/cava/blob/14b34e7a89baf95c7873976256392dfdc9e666f8/scuttlebutt-handshake/src/main/java/net/consensys/cava/scuttlebutt/handshake/SecureScuttlebuttStream.java#L86
The Vertx library code swallows the NullPointerException because it is not configured with an exception handler, so it is not visible.
The receivedMessage
ClientHandler
implementation is not called back.
Steps I used to reproduce:
Check out commit c01f2e5745d8f2a1296027f9bda98ad9ac2aed3d
of cava (which was the latest for me at the time - i didn't seem to be able to get the latest from the Maven repos.)
Run gradle build
to create jar
files for the libraries.
Init a gradle project and copy the libs into a 'lib' directory and add the libs directory as a dependency
Define a main class:
*
* This Java source file was generated by the Gradle 'init' task.
*/
package cava.test;
import io.vertx.core.Vertx;
import net.consensys.cava.bytes.Bytes;
import net.consensys.cava.bytes.Bytes32;
import net.consensys.cava.concurrent.AsyncCompletion;
import net.consensys.cava.scuttlebutt.handshake.vertx.SecureScuttlebuttVertxClient;
import net.consensys.cava.crypto.sodium.Signature;
import net.consensys.cava.scuttlebutt.rpc.RPCCodec;
import net.consensys.cava.scuttlebutt.rpc.RPCFlag;
import java.util.Base64;
public class App {
public static void main(String[] args) throws Exception {
int port = 8008;
String host = "localhost";
String publicKeyBase64 = System.getenv("scuttle_public_key");
String privateKeyBase64 = System.getenv("scuttle_private_key");
if (publicKeyBase64 == null) {
throw new Exception("scuttle_public_key environment variable missing");
}
if (privateKeyBase64 == null) {
throw new Exception("scuttle_private_key environment variable missing");
}
byte[] publicKeyBytes = Base64.getDecoder().decode(publicKeyBase64);
byte[] privateKeyBytes = Base64.getDecoder().decode(privateKeyBase64);
Signature.PublicKey publicKey = Signature.PublicKey.fromBytes(publicKeyBytes);
Signature.SecretKey privateKey = Signature.SecretKey.fromBytes(privateKeyBytes);
Signature.KeyPair keyPair = new Signature.KeyPair(publicKey, privateKey);
Vertx vertx = Vertx.vertx();
// The default scuttlebutt network key (the one Patchwork uses.)
String networkKeyBase64 = "1KHLiKZvAvjbY1ziZEHMXawbCEIM6qwjCDm3VYRan/s=";
byte[] networkKeyBytes = Base64.getDecoder().decode(networkKeyBase64);
Bytes32 networkKeyBytes32 = Bytes32.wrap(networkKeyBytes);
SecureScuttlebuttVertxClient secureScuttlebuttVertxClient =
new SecureScuttlebuttVertxClient(vertx, keyPair, networkKeyBytes32);
AsyncCompletion onConnect = secureScuttlebuttVertxClient.connectTo(port, host, publicKey, (senderFn, stopFn) -> {
// Tell the client handler how to send bytes to the server
MyClientHandler clientHandler = new MyClientHandler(senderFn, stopFn);
// An RPC command that just tells us our public key (like ssb-server whoami on the command line.)
String rpcRequestBody = "{\n" +
" \"name\": [\"whoami\"],\n" +
" \"type\": \"async\" " +
"}";
Bytes rpcRequest = RPCCodec.encodeRequest(rpcRequestBody, RPCFlag.BodyType.JSON);
System.out.println("Attempting RPC request...");
clientHandler.sendMessage(rpcRequest);
// We hand over a ClientHandler so we get called back when new bytes arrive
return clientHandler;
});
}
}
Define a ClientHandler implementation
package cava.test;
import net.consensys.cava.bytes.Bytes;
import net.consensys.cava.scuttlebutt.handshake.vertx.ClientHandler;
import java.util.function.Consumer;
public class MyClientHandler implements ClientHandler {
private final Consumer<Bytes> sender;
private final Runnable terminationFn;
public MyClientHandler(Consumer<Bytes> sender, Runnable terminationFn) {
this.sender = sender;
this.terminationFn = terminationFn;
}
@Override
public void receivedMessage(Bytes message) {
System.out.println("We received a message?");
}
@Override
public void streamClosed() {
System.out.println("Stream closed?");
}
void sendMessage(Bytes bytes) {
System.out.println("Sending message?");
sender.accept(bytes);
}
void closeStream() {
terminationFn.run();
}
}
Set the public and private keys as the scuttle_public_key
and scuttle_private_key
environment variables by copying the base64 representations from the secret
file in the .ssb
directory (minus the .ed25519
suffix.)
Notice it doesn't get called back with bytes for the response, and step through debug to see it throwing a NullPointerException while trying to decrypt the header from the RPC response
I'm trying to debug the issue at the moment (i wonder if there's something strange going on with the keys), but I wondered if you had any ideas?
We should update the build logic to gradle kotlin dsl: https://guides.gradle.org/migrating-build-logic-from-groovy-to-kotlin/
It looks like only cava-toml is making it to JCenter.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.