magic-wormhole / magic-wormhole.rs Goto Github PK

https://github.com/warner/magic-wormhole/blob/0.10.5/src/wormhole/_send.py

Send is a simple machine that manages the transmission of encrypted application messages. It is told about the shared encryption key by Key (after it has been verified by receipt of the peer's Version message), and is given outbound plaintexts by Boss. It encrypts these and delivers the ciphertext to Mailbox for delivery. Any messages which arrive before the key has been computed are queued until afterwards: this allows applications to call w.send_message() without waiting for something like w.get_verifier() or w.get_next_message().

I've been updating our dependencies, and moving to the current sodiumoxide-0.1.0 crate is a hassle because it requires that the OS have a pre-installed libsodium (of a newer version that what came in debian/stretch). Getting that installed is frustrating, and it means that cargo install magic-wormhole won't work out-of-the-box.

We don't need much from libsodium: we only use the SecretBox construction, to encrypt the post-PAKE records in the Mailbox/rendezvous protocol (and again for the Transit records). We don't need the more general Box construction (the public-key thing), or any of the lower-level tools.

It looks like SecretBox might have been standardized in RFC7539, at least there's an AEAD construction that uses ChaCha20 and Poly1305. If the layout is in fact the same, then we've got some pure-Rust options for our symmetric crypto. The chacha20-poly1305-aead crate looks promising, although it hasn't been updated in a couple years and says that it needs Nightly rust. The well-respected ring crate has an RFC7539 AEAD construction; I can't immediately tell if it's implemented in C or in Rust, but I believe it all builds with a simple cargo build so it doesn't matter too much. Also I see a RustCrypto experiment that seems to have the right pieces in it, which would be ideal (we're using several RustCrypto libraries already).

So the task here is to identify the smallest easy-to-build ideally-pure-Rust-but-at-least-self-contained most-respected crate we can find, port our use of SecretBox to its API, and then drop our dependency on sodiumoxide.

crate has been renamed to cipher

This crate has been renamed from stream-cipher to cipher.

The new repository location is at:

<https://github.com/RustCrypto/traits/tree/master/cipher>

See advisory page for additional details.

Hi, I'm really interested in this project and would like to know what's its current development state is. I'm considering to contribute to it, but I'd need some guidance for a start: which issues need tackling and how to get familiar with the code.

Some "help wanted" and "good first issue" labels might help for orientation.

As mentioned in https://github.com/warner/magic-wormhole.rs/issues/9#issuecomment-379272954 , we still need to implement the top-level w.derive_key(purpose), and make sure it can only be called when we've got a verified key.

API errors are things like calling w.set_code() twice, or using a badly-formatted code, or calling w.derive_key before the key has been established. In the Python code, these are exceptions that are raised synchronously while processing the API call.

In the Rust version, we could do the same (which means defining some error classes, and changing some function signatures to return Result, and probably changing the top-level WormholeCore entry point to return a Result too).

Or we might do something more asynchronous, and emit "error" events along with the other API actions that go out. The challenge with this approach is how to correlate the errors with the pending API call that's responsible for them.

The API/IO glue layer has to decide how to present errors. If it's providing Futures to the application, then it probably wants to return a Future that errbacks (rather than returning Result<Future<..>>. That doesn't obligate the WormholeCore to deliver errors asynchronously: the glue layer could react to a synchronous error by returning a pre-failed Future (the equivalent of twisted.internet.defer.fail(exception)).

One other thing to consider is how much state might have been changed at the time we decide to signal an error. Maybe Boss (which handles all API calls) should do its error checking before firing any other events, and no other machine should be allowed to signal an error.

We should also think about the close() API, which in Python is defined to return a Deferred that fires when all network and timer operations have stopped (so all sockets are closed, all cleanup is done). close() signals an error if the wormhole was unable to connect, which includes cases like:

• the network connection to the Rendezvous/Mailbox server couldn't complete the first time (with information about the error, so the user can figure out if the server is offline, if their DNS resolution is broken, or if they're simply not on the internet right now)
• we connected, but we were interrupted before hearing from the other side (aka "Lonely")
• we connected, but at some point we received a badly-encrypted message, which indicates a man-in-the-middle attack (aka "Scared")

Currently magic-wormhole.rs does not compile on Windows due to use of libsodium which is not natively available on Windows. Is there any alterantive to libsodium which we can rely on?. Or is there a way to provide crate for Windows?. Probably we should look at how cargo does it?.

I know this is not priority right now but I just thought of keeping this on our radar.

https://github.com/warner/magic-wormhole/blob/0.10.5/src/wormhole/_lister.py

Lister is responsible for fetching a recent list of active Nameplates from the rendezvous/mailbox server. It is used by the Input machine each time the user hits TAB while typing in the nameplate portion of the wormhole code (or when a GUI application wants to populate a drop-down menu for code entry). It remembers that we currently want (or don't want) a listing, and it knows whether we're connected to the server or not. It sends a NAMEPLATES request at the right time (through the Rendezvous machine), and delivers any response to the Input machine.

https://github.com/warner/magic-wormhole/blob/0.10.5/src/wormhole/_boss.py

The "Boss" machine is the top-level driver of the wormhole object. It is the only machine that interacts with the API layer (all API commands are delivered to the Boss, and only the Boss emits events that go out to the API layer: the joke is that only the boss is allowed to talk to the customers).

The Boss does some small processing on the inbound messages (e.g. it routes the Versions message for internal use, but delivers the app_versions key to the application, and in the future it will peel off other messages like "dilate" for the new Dilation protocol's use). It avoids delivering messages when the wormhole is supposed to be closed. It manages the close/shutdown process, waiting for the Terminator machine to report resource release and socket shutdown, then delivers a Closed message to the API (with a "mood" indicating whether things went well or if there was an error).

In the Python version, key.py has two nested state machines: the outer one accepts two events (got_code and got_pake) and ensures that the inner one always receives the got_code event first. This was a quick hack to fix a bug, in which I didn't make the inner state machine tolerant of getting the PAKE message first (this can happen if you're using input_code, and you wait long enough between typing the nameplate and typing the rest of the words: the peer's PAKE message can arrive any time after you commit to the nameplate, but we don't know the code locally until you finish typing the words).

On the other hand, if you use set_code or generate_code, then you'll know the code long before you learn the peer's PAKE message. And you need to actually generate+send the PAKE message as soon as you can: if both sides wait until they've heard from their peer before sending anything, then no progress can be made.

The key.rs machine currently defers sending both messages until both have arrived, so it only works if run against a different implementation which starts sending the PAKE earlier.

https://github.com/warner/magic-wormhole/blob/0.10.5/src/wormhole/_code.py

The Code machine is responsible for acquiring the wormhole code, in one of three modes corresponding to the API calls: w.allocate_code(), w.set_code(), and w.input_code(). The set_code method is trivial, and the other two delegate to the Allocator and Input respectively.

The input_code process is split into phases: first the nameplate is set, then the words are set, so there are separate input events for each phase.

When the code is complete, it notifies the Key machine (which uses it as input to the PAKE function) and the Boss (which notifies the application, which is most important when using allocate_code so it knows what to display to the user for transcription.

https://github.com/warner/magic-wormhole/blob/0.10.5/src/wormhole/_order.py

Order is a tiny machine, it's only purpose is to make sure that the mailbox's PAKE message arrives before anything else. Clients are supposed to send their PAKE message before anything else (in fact they can't sent anything else until they receive their peer's PAKE, so they can compute the encryption key for those other messages), but the server is not currently obligated to retain this ordering. This machine sits behind the Receive machine and queues all non-PAKE messages (which go to Receive) until after it sees the PAKE message (which is delivered to Key).

The server stores these messages in a SQLite database, without timestamps. If the other peer is connected when the message is posted, it will receive it immediately (and thus in order), but if the peer loses their connection and reconnects, the server will re-deliver everything in the DB, and this is the part that might get out-of-order.

I think I'm going to change thet server to retain ordering of messages (see magic-wormhole/magic-wormhole#122), but we should still tolerate out-of-order servers just in case.

At the moment it merely is a proof of concept. First of all, sends should be done in parallel. This means that once a connection to the other side is established, the actual file transfer should be moved into a background task in order for the main loop to accept new receivers.

The loop also needs an exit condition, based on time or number of receivers. I think 1h could be a sensible default.

We can use newtypes to wrap Strings etc to make a zero-cost type for say, Side, Phase, Body and so on.. This would make the code a lot more typesafe.

https://github.com/warner/magic-wormhole/blob/0.10.5/src/wormhole/_receive.py

The Receive machine is responsible for decrypting inbound messages, and keeping track of when our key becomes "verified" (meaning we've seen at least one validly-encrypted message using our PAKE-generated shared encryption key).

It receives the encryption key from Key, and messages from Order, and sends decrypted plaintexts to Boss. It also notifies Send when the key has been verified (which waits until that point to encrypt and send any queued outbound application messages), and tells Boss about mood changes ("happy" when the key is verified, and "scared" when we see a badly-encrypted message).

I was just wondering how to put in w.input_code in io/blocking. Does io/blocking should provide tab completion using rustyline or w.input_code should simply initialize InputMachine and leave it to app using io/blocking to complete tab completion logic?.

I think letting app do the hard work seems correct way as w.input_code can have different meaning in different kinds of client. @vu3rdd and me were discussing cross compiling wormhole_core to Android and developing an Android app where he brought up that tab completion is probably meaning less for Android apps and scanning qr code is more meaningful. In such cases probably w.input_code will be simply scan qr code and use w.set_code.

Thought of documenting this here before I forget the discussion.

PS: feel free to make title more readable :).

I've seen curve25519-dalek use some tools/libraries to zero out the memory of sensitive data structures (keys, intermediate values used to derive keys) just before they get dropped. We should do something similar. A lot of our temporaries are on the stack, so we should zero those too.

Basically, if the application above us is careful with their own memory, then our library should not introduce any additional exposure of secrets.

I think @david415 had some ideas on this one.

https://github.com/warner/magic-wormhole/blob/0.10.5/src/wormhole/_nameplate.py

The Nameplate machine manages our use of the server-side "nameplate" object, which corresponds to the short integer at the start of a wormhole code (the 4 in 4-purple-sausages). See #11 for the relationship between nameplates and mailboxes.

It starts with the name of the nameplate to open, and sends a CLAIM message once the server connection is established (which creates the nameplate object if it didn't exist already: this is used by offline codes, in which the nameplate was selected by the users instead of being allocated by the server).The state machine keeps track of whether the nameplate might exist, whether we want it to still exist (this changes when the Mailbox machine tells us we can release it), and whether we know it's been released or not. It notifies Mailbox when the mailbox-id is received (this arrives in the CLAIMED response).

It also notifies the Input machine with the wordlist to use for code completion. In the current version this is always the same, but eventually we'll have some "nameplate attributes" stored on the server to indicate which wordlist is being used (to support i18n and language-specific wordlists, or simply user preferences), and once we receive the CLAIMED response we'll look up the wordlist and deliver it to Input.

https://github.com/warner/magic-wormhole/blob/0.10.5/src/wormhole/_mailbox.py

Mailbox manages the messages our client exchanges with our peer client. It's one of the more compex machines (along with Nameplate).

On the server, Nameplates are pointers to a Mailbox object. Nameplates are short strings, Mailboxes are long random strings. The Nameplate is released as quickly as possible (once both sides have opened their mailbox), to free up the namespace and let the nameplate be used by some other client.. The Mailbox stays open as long as both clients are running, and since they're long and random, there's no pressure to conserve them or free them up quickly.

We might acquire the Mailbox by going through a Nameplate, or we might open one directly (e.g. if we use a Wormhole Seed, or we're re-opening a previous connection somehow). But we want to release the Mailbox when we're done using it. And our connection to the server might go up and down while we're running.

Our Mailbox machine keeps track of mailbox allocation: it remembers whether we want the mailbox to be open or not, it remembers if we've sent an Open request that might not have been received yet, and the same for a Close. If we're told to open the Mailbox but we don't currently have a connection, then the next time the connection opens we'll send the Open message. Freeing/closing the mailbox is similar.

The Maibox machine also manages the messages we send out, and the ones we receive. It queues outbound messages until we have a connection to send them on, and doesn't remove them from the queue until the server echoes it back (indicating that it's been received by the server, which then takes responsibility for delivery to the peer).

Any inbound message tells us that our peer has opened the mailbox, which means they don't need the Nameplate anymore, so we tell our Nameplate machine that it can release the nameplate.

Right now, we are generously using .as_bytes(), .clone(), to_vec() and so on without caring much about the cost involved in the conversion (like Copying). We need to look at the code that gets generated and see if we can eliminate as many of them as we can by using the right types where ever possible.

https://github.com/warner/magic-wormhole/blob/0.10.5/src/wormhole/_input.py

The Input machine handles the w.input_code() case, and helps with interactive (tab-completion) entry of the wormhole code. In the Python API, w.input_code() returns an "Input Helper" with its own API: for our sans-io approach, the API layer will own this helper object, and we need to add API events/actions to let the API layer's object communicate with the core's Input machine.

The Input machine communicates with the Lister object to obtain the list of active Nameplates (from the mailbox/rendezvous server), and delivers its results to the Code object (both the intermediate got_nameplate phase and the final finished_input event).

The Python version has a --dump-timing=FILE option which writes out a JSON-formatted file, containing a list of timing events. Each message to and from the server is recorded, along with timing data that the server provides, to give us a sense of round-trip time and whether delays are resulting from computation time in the local client, network travel to the server, turnaround time within the server, network travel from the server to the other client, or computation within the remote client. These JSON files are processed by a tool in https://github.com/warner/magic-wormhole/blob/0.10.5/misc/dump-timing.py and a neighboring single-page web app (using https://d3js.org/) to render a scrollable zoomable timeline browser.

The Rust version should have support for emitting these files too. The event-based workflow should make it pretty easy: just add a Timing event, emit them in various places, route them to a new tiny machine named "Timing" which gathers them into a vector. Then, add some new API call that fetches the vector and writes it into a JSON file.

Eventually the rendering tool should probably be pulled out into a separate repo, and updated (my javascript/web skills are horrible). It might be interesting to record more internal events too, although except for the PAKE computation they'll probably be instantaneous compared to the network delays.

https://github.com/warner/magic-wormhole/blob/0.10.5/src/wormhole/_rendezvous.py

Magic-wormhole uses a server to queue and deliver messages to the other client. Originally I called this the "Rendezvous Server", because it's a place where clients meet up. But that word was too long, so recently I've started calling it the "Mailbox Server", which makes its role a bit more obvious. The code for this server now lives in its own repo (https://github.com/warner/magic-wormhole-mailbox-server), but the client-side code still sometimes refers to it as "Rendezvous". This server owns the "Nameplate" and "Mailbox" data structures, and our client-side Nameplate/Mailbox machines deal specifically with these two structures, so we call the overall connection-management machine the "Rendezvous Machine".

(until we come up with a better name for it)

The Rendezvous machine manages the WebSocket connection to the server. It knows whether we want one or not (we always want one until w.close() is called), whether we have a connection or not, and reconnects after a short random delay after it drops (to avoid the "thundering herd" problem). It also sends the BIND message upon connect, and formats all other server messages on behalf of the other machines (e.g. it knows how to format the Add message which Send transmits, and the Claim message that Nameplate generates).

It also handles inbound messages, by parsing them into a type (Claimed, Nameplates, Message, etc), and delivering them to their appropriate machines.

It also records debug/timing information. Each message to the server includes a short random id value, and the server responds with an ack message that copies the ID. The messages also include server_tx with a timestamp of when the server sent it. This allows port-mortem timing analysis (see https://github.com/warner/magic-wormhole/blob/0.10.5/misc/dump-timing.py) to find ways to speed things up.

The Python code delegates a lot of the connect-lose-wait-reconnect logic to a Twisted class named ClientService, so the python Rendezvous object doesn't really have a state machine. The picture above shows what it would basically look like, if we had one. So we'll need to build one and replicate a lot of the ClientService functionality.

Rendezvous is (currently) the only machine that performs any IO. In our sans-io model, it does this by emitting "IO Actions", which are events that get delivered to the API/IO glue layer, with names like "start timer" and "open websocket connection". That layer then figures out some way to perform the requested action, and delivers "IO Events" when they finish (like "timer expired", "connection made", and "data received"). There are handles to associate the Events with the Action that provoked them.

Flaw in hyper allows request smuggling by sending a body in GET requests

Vulnerable versions of hyper allow GET requests to have bodies, even if there is
no Transfer-Encoding or Content-Length header. As per the HTTP 1.1
specification, such requests do not have bodies, so the body will be interpreted
as a separate HTTP request.

This allows an attacker who can control the body and method of an HTTP request
made by hyper to inject a request with headers that would not otherwise be
allowed, as demonstrated by sending a malformed HTTP request from a Substrate
runtime. This allows bypassing CORS restrictions. In combination with other
vulnerabilities, such as an exploitable web server listening on loopback, it may
allow remote code execution.

The flaw was corrected in hyper version 0.12.34.

See advisory page for additional details.

Insufficient size checks in outgoing buffer in ws allows remote attacker to run the process out of memory

Affected versions of this crate did not properly check and cap the growth of the outgoing buffer.

This allows a remote attacker to take down the process by growing the buffer of their (single) connection until the process runs out of memory it can allocate and is killed.

The flaw was corrected in the parity-ws fork (>0.10.0) by disconnecting a client when the buffer runs full.

See advisory page for additional details.

In order for the CLI to display a fancy progress bar, the transfer API needs to expose such information. Together with progress, we probably want to track transfer speed.

One possibly way to implement this is to pass some callback to the file transfer that calls the method once something happens. The alternative way would be to spawn the file transfer (or the UI updating) in a new task and let them send events over a channel. Note that the latter can be implemented using the former outside of the API (by having the callback writing into a channel).

The difficulty I see here is to know when the caller should be informed. After every packet? Every percent? Every second? Probably some combination of multiple. Can the transfer expect the UI updating to be reasonably quick? Otherwise we a forced to make this multiple tasks.

Prior art/related work:

• progress-streams crate: Wraps a Reader/Writer using the callback approach. Not sure if this fits us as we are on a higher abstraction level.

The actual CLI implementation is secondary to this issue. Of course, it make sense to develop both together in order to test the ergonomics of the API, but we can always make the output pretty later on. There are plenty of fancy progress bar crates out there :)

What is the reason for using u32 rather than u64 here (and everywhere)?

https://github.com/warner/magic-wormhole.rs/blob/a1bc3a7f4d8c7b289bd253b9094080afcf471759/src/core/transfer.rs#L19

In #48 @copyninja improved the demo code that performs the equivalent of wormhole send --text=, using some structs to encapsulate the client-to-client messages like Offer and Answer and Error.

I'm thinking we should put this code in a separate crate, or at least a separate module. WormholeCore doesn't know anything about the protocols run on top of a wormhole, it just provides the w.send() and w.get_message calls to convey the messages used to implement such protocols. The entire core/examples/ directory should probably go away, moved into maybe io/blocking/examples or io/tokio/examples depending upon what sort of IO it uses.

I want to use the future magic-wormhole crate to contain the file-transfer executable, so cargo install magic-wormhole gets you a wormhole binary that behaves mostly like the Python version. We could put all the transfer-related stuff into that, so to get ~/.cargo/bin/wormhole you would cargo install magic-wormhole and you'd get three crates installed: magic-wormhole, magic-wormhole-io-blocking (or -tokio if we go that route), and magic-wormhole-core. Applications which want to use wormholes for provisioning would just depend on magic-wormhole-io-blocking (or -tokio), which would pull in magic-wormhole-core but not magic-wormhole.

One open question is whether other applications might want access to the text- or file- transfer portions of the wormhole executable, but in library form. I'm not 100% convinced that this is likely, but if it is, we might want to split magic-wormhole into a small cargo init --binary crate (named magic-wormhole) that installs the executable, and a larger --library crate that contains everything else. so those apps can get the library without the executable. If we go that way, then the Offer and Answer messages (and the rest of client-to-client stuff) would go in that library.

Another question is where the Transit code (which I think @vu3rdd was hoping to tackle) should go. In the current protocol, it's mostly independent of the wormhole library: it produces messages that need to be delivered safely to the other Transit instance (so we drop them into w.send()), but that's it. So maybe we should make a new crate named magic-wormhole-transit for it. There's not much sans-io protocol to be written, I think it'll need to depend upon a particular IO style right off the bat, but maybe there's an argument for having both magic-wormhole-transit-blocking and magic-wormhole-transit-tokio, I don't know.

The upcoming Dilation protocol work is much more tightly integrated into the Wormhole object, so that will need to go somewhere else. I haven't figured very much about this out yet. The Python version is all very Twisted-centric, and w.dilate() returns a Deferred that fires with a trio of Endpoints, and then you can run normal protocol connections or Listeners over those endpoints. The Rust equivalent is going to look like the normal TCP socket API (which could be like std::net::TcpStream, or some Tokio/Mio equivalent). So maybe we say that the Dilation API's IO framework will match the rest of the Wormhole's IO framework, which would mean that magic-wormhole-io-blocking gets you a Dilation-capable Wormhole object with an API that resembles std::net::TcpStream. And magic-wormhole-io-tokio gets you one with a Tokio-flavored Dilation API. The shared parts of the Dilation protocol might go in magic-wormhole-core, or maybe in a separate shared crate (magic-wormhole-dilation-core?) since I think there might be enough sans-io -style code to make that worthwhile. I dunno, I need to finish the Python work on Dilation first, and that's going slowly.

It would be nice if core is a library (or core + some io facilities), so that the library can be built standalone and used in other projects. It would also be nice for cross compiling it into wasm.

@copyninja mentioned (in warner@46d31b8#commitcomment-29139497) that to_string should be part of the standard Display trait, but I used it as a standalone method on the Mood struct. I'm up for making this Display, but the specific strings used there are part of the wire protocol, so they're fixed and should not be changed in the future (e.g. as part of i18n language-specific translations). So we need some comments to that effect, and some unit tests that ensure the strings remain fixed.

Error handling is a big mess right how, but this is maybe the easiest one to fix. All structs that are send expose a serialize/deserialize method that delegates to serde_json together with unwrap(). This should be cleaned up. Errors need to be exposed. Maybe we should inline all of them and get rid of them entirely?

https://github.com/warner/magic-wormhole/blob/0.10.5/src/wormhole/_terminator.py

The Terminator machine handles wormhole shutdown. We want to be polite and clean up after ourselves, so we try to release the Nameplate and Mailbox structures on the server. Terminator tracks which of these tasks have been finished and which have not. If w.close() has been called but we don't currently have a server connection, we wait until we've got one, send the Release and Close messages, wait for them to be acknowleged, then tell Rendezvous that we don't need the connection anymore, wait for it to be dropped, and finally notify Boss that we can shut down. Boss then delivers a Closed message to the API layer.

If Nameplate is told to release the nameplate for other reasons (i.e. we have the mailbox, and we see evidence that our peer has the mailbox, so we don't need it any more), and it successfully releases it, it tells Terminator so we don't need to wait for anything else during shutdown. The same happens with the mailbox. And if both have been released, we don't need a connection (and if the connection just happened to have been lost at the moment the application calls w.close(), then when Terminator tells Rendezvous to stop, it will send back Stopped immediately because it's already disconnected).

The state names refer to the set of resources that are still active: n for nameplate, m for mailbox, and o for an open (non-closed) Wormhole. The initial state is Snmo, meaning all are still active, and as we receive word that resources have been freed, we remove letters from the state name.

The Terminator machine, like all machines, is created at the same time as the rest of the Wormhole, and receives events as the rest of the machines do their business. So the most likely progression of states is from the initial Snmo, to Smo as the nameplate is released, then Sm as w.close() is called (upon which Boss tells Mailbox that we don't need one anymore, so it releases it), then S_stopping after the mailbox CLOSED is received (at which point Terminator tells Rendezvous to tear down the websocket connection, and not reestablish a new one), then S_stopped when the connection is finally lost. At this point it tells Boss that we're done.

This doesn't really have to be managed by a state machine: a simple vector of booleans might be sufficient.

Using Magic Wormhole as a replacement for tools like ngrok, to tunnel raw sockets and/or HTTP traffic, would be really useful.

From a cursory glance at the underlying implementation, this seem doable on top of the existing model.

https://github.com/warner/magic-wormhole/blob/0.10.5/src/wormhole/_key.py

The Key machine owns the PAKE process. It generates the PAKE message (giving it to Mailbox for delivery to the peer), it receives the local wormhole code and the remote peer's PAKE message, and combines them to form the shared encryption key. The Python version actually has two machines, "Key" merely gathers both events (local code and inbound PAKE message) and delivers them in sorted order (code first, inbound PAKE second) to the "SortedKey" machine, which does the PAKE computation and computes the final key. (it was easiest to do this in Automat by using a separate state machine.. maybe we don't need to do that here).

When the key is computed, Key is responsible for encryption the Versions message and adding to the Mailbox (this is used as a key-confirmation message). It also notifies Receive with the key, so it can decrypt subsequent messages.

Transferring folders is not that much more complex than transferring files: simply zip all files and send that one with some special metadata. The receiving end might even be able to unzip the thing on the fly, so no temporary file here.

The problem for a streaming send is that we need to know the number of compressed bytes in advance, so that might be a bit tricky but probably nothing insurmountable.

In #58 we noted that our symmetric encryption primitive (XSalsa20/Poly1305) is pretty much only used in libsodium, and the rest of the world (i.e. RFC7539 and ring's AEAD construction thereof) has moved to ChaCha20. Libsodium is a nuisance to use from Rust (since it's an FFI binding to a C library that takes some manual effort to build), so a short-term effort is to run a pure-rust XSalsa20 crate that's compatible with SecretBox.

But the long-term hope would be to move to ChaCha20 and RFC7539 (maybe Noise uses this?). That's a compatibility break, so I want to batch together all the other breaking crypto changes we might want to make and do them all at the same time, since as we all know version negotiation should happen just once at the highest level of a protocol, and it should be on a number instead of a matrix of individual features.

So this ticket is to collect the crypto changes we might want to make. Some others to consider:

• get a PAKE pattern into Noise and use that instead of our initial setup protocol
• replace Transit with Noise (although the new Dilation protocol already does this)
• if we ever manage to establish a better encoding format or well-known-element selection process for SPAKE2, switch to that flavor for the PAKE phase

https://github.com/warner/magic-wormhole/blob/0.10.5/src/wormhole/_allocator.py

The Allocator is responsible for creating a code, which consists of a Nameplate (the short integer at the start of a wormhole code, like the 4 in 4-purple-sausages) and some random words. This is driven by Code, when the application uses the w.allocate_code() API. It must wait until we're connected, then send an allocate message to the mailbox/rendezvous server. If we lose the connection, a new message must be sent upon reconnection. When a response is received, the server-provided nameplate is combined with locally-generated random words, and the combined code is delivered to the Code machine, then we're done.

This will live in core/src/allocator.rs.

As mentioned in https://github.com/warner/magic-wormhole.rs/issues/13#issuecomment-378428542 , there's probably a particular sequence of input messages that would violate the expectations of the Receive machine, which we need to deal with.

I've been thinking we should publish a couple of different crates:

• magic_wormhole_core (no IO)
• magic_wormhole_tokio (yes IO, contains tokio-friendly APIs)
• magic_wormhole (provides the wormhole CLI tool, so cargo install magic-wormhole; wormhole send works, just like python's pip(si) install magic-wormhole; wormhole send)

But.. now I'm reading about the [features] property in Cargo.toml, and maybe we should use those instead?

• core feature gets you the sans-io library
• io-tokio gets you the IO layer that presents a Tokio-friendly API
• cli gets you the command-line tool

and then we could make all three be the default (so cargo install magic-wormhole gets you a functional CLI). Applications that don't want the CLI (but use Tokio) could use feature=[core, io-tokio] (or however they spell it). And if they want a different IO library, then [core, io-threads] or whatever could work.

We'd have just the one repository, and just the one published crate.

I haven't used these "features" much.. does that seem reasonable?

The choice of concurrency/scheduling mechanism is probably the most-significant bit of the design. I'm coming from 15 years of programming under Twisted (the primary "event-loop" / non-threaded approach in the Python world, slowly being augmented/enhanced/coopted by asyncio), so my instinct is to look for the most Twisted-like system in the Rust world, and apparently that's Tokio. I know there's a learning curve (I remember going through that with Twisted), but my hunch is that it's the right direction to go, because it's my experience that threads are always the wrong choice.

Except.. things are a lot better in the Rust world, and threads are not necessarily immediately fatal. And golly, Tokio is pretty daunting. So I could be convinced that, instead, each part of magic-wormhole that waits on a different source should run in its own thread. I can't readily imagine how that'd all fit together (does most of the logic live in a single thread that gets spawned off when you create the Wormhole object? and then each TCP connection attempt lives in a separate one? and does the top-most API run through a channel of some sort?), but I'm sure lots of other projects do it that way, so it must be possible. We'd start with the ws crate instead of websocket.

One constraint to keep in mind is that we want this library to be usable from other languages: this should provide a core of functionality, to which other folks can write bindings (from Node, or Ruby, etc). Other environments are going to have other approaches to networking (and concurrency). So we should try to write the main part of this in "sans-io" style, with somewhat-swappable portions written to the actual IO functions (the Python implementation is not very sans-io, but all the state machines interface to the websocket connection in just one place, so it might not be too hard to fix that). Again, I'm not sure quite what the result will look like: probably a core that accepts calls like "hey I just got a websocket message for you", and a separate piece that takes a URL and makes the websocket connection and sends the results to the first piece, and then the do-it-all-within-Rust library glues the two of them together (presenting a single wormhole::create() API), while the e.g. Lua bindings would expose just the core, but also provide a do-it-all-within-Lua API that uses a Lua-specific websocket thing glued to the core bindings.

We do not want to error out and exit if the other sends us messages with null fields or optional fields. eg: the message of type message has an id field that may be set to null. We would like the deserialization of such messages from binary to json be handled leniently.

I just saw the machine crate.. looks like a nice way to build state machines. Maybe we should use it instead of the manually-written ones?

https://github.com/rust-bakery/machine

One hassle with our current implementation is ownership issues when a new state enum is generated using pieces from the previous state's enum. I had to make a lot of those values Copy so they could be copied freely, but it feels like that shouldn't be necessary (we drop the old state in the process of building the new one, so maybe we could use move instead or something?). I don't know if machine would affect this or not.

In the Python code, Wormhole is the object that does the PAKE-based key agreement and then gets you queued delivery of small messages. There's a separate object named Transit which gets you a direct TCP connection (possibly going through a "transit relay"), encrypted with a "transit key", using "connection hints" that are delivered through the Wormhole channel. The CLI tool builds the Wormhole, then sends an "offer" through the wormhole describing what it wants to send to the recipient (text message, file, or directory). If it's a file or directory, it includes the Connection Hints from its Transit object, and gets hints from the recipient (which it then submits to its Transit). Once the Transit connection is established, the file/directory is streamed through it.

I'm working on replacing Transit with a feature named "Dilated Wormholes", in which the direct-connection is managed by the Wormhole object itself, using a new API named w.dilate(). This will make it much easier to send multiple files over the same connection, or to send them in either direction, on demand. We'll need this to build a proper GUI in which either side can drag+drop files into the application window and see them sent to the other, regardless of which side initiated the original connection.

But, to build a tool that can interoperate with the current (0.10.5) python version, we'll need to implement Transit in Rust. https://github.com/warner/magic-wormhole/blob/0.10.5/src/wormhole/transit.py is the relevant code, and https://github.com/warner/magic-wormhole/blob/0.10.5/docs/transit.md has the docs.

We'll probably need to extend the IO/API layer to support Transit (and Dilation in the future), with actions like:

• create an outbound TCP connection, notify on connection, write data to it, receive data from it
• open a listening socket, learn its TCP port number
• enumerate the host's IP addresses
• get notifications when the outbound write queue is full, so we can pause production of transit data
• tell the IO layer to pause receipt of data, since our inbound queue is full
• allow the application to ask to open a new transit/dilation channel, write data to it, receive data from it, pause it, be paused by it, resume/be-resumed, close, be notified about close

It might make sense for the API layer to provide connect/listen functions that look like that platform's native network-socket connect/listen facilities. For Rust in particular, the thing that you write data to should probably have the same Trait as a native TCP socket, so other protocols can be layered on top easily.

• Allocator
• Boss
• Code
• Input
• Key
• Lister
• Mailbox
• Nameplate
• Order
• Receive
• Rendezvous
• Send
• Terminator

As described in #8, the tab-completion type-in-the-code API uses an object named the InputHelper. In the sans-io approach, this object will be defined (and owned) by the IO/API glue layer, but the Core will define some API input and output events to communicate with it.

The Python API docs specify that you obtain a helper from the wormhole object with a synchronous function likehelper = w.input_code(), and then defines the Helper's API as follows:

• refresh_nameplates(): requests an updated list of nameplates from the
  Rendezvous Server. These form the first portion of the wormhole code (e.g.
  "4" in "4-purple-sausages"). Note that they are unicode strings (so "4",
  not 4). The Helper will get the response in the background, and calls to
  get_nameplate_completions() after the response will use the new list.
  Calling this after h.choose_nameplate will raise
  AlreadyChoseNameplateError.
• matches = h.get_nameplate_completions(prefix): returns (synchronously) a
  set of completions for the given nameplate prefix, along with the hyphen
  that always follows the nameplate (and separates the nameplate from the
  rest of the code). For example, if the server reports nameplates 1, 12, 13,
  24, and 170 are in use, get_nameplate_completions("1") will return
  {"1-", "12-", "13-", "170-"}. You may want to sort these before
  displaying them to the user. Raises AlreadyChoseNameplateError if called
  after h.choose_nameplate.
• h.choose_nameplate(nameplate): accepts a string with the chosen
  nameplate. May only be called once, after which
  AlreadyChoseNameplateError is raised. (in this future, this might
  return a Deferred that fires (with None) when the nameplate's wordlist is
  known (which happens after the nameplate is claimed, requiring a roundtrip
  to the server)).
• d = h.when_wordlist_is_available(): return a Deferred that fires (with
  None) when the wordlist is known. This can be used to block a readline
  frontend which has just called h.choose_nameplate() until the resulting
  wordlist is known, which can improve the tab-completion behavior.
• matches = h.get_word_completions(prefix): return (synchronously) a set of
  completions for the given words prefix. This will include a trailing hyphen
  if more words are expected. The possible completions depend upon the
  wordlist in use for the previously-claimed nameplate, so calling this
  before choose_nameplate will raise MustChooseNameplateFirstError.
  Calling this after h.choose_words() will raise AlreadyChoseWordsError.
  Given a prefix like "su", this returns a set of strings which are potential
  matches (e.g. {"supportive-", "surrender-", "suspicious-"}. The prefix
  should not include the nameplate, but should include whatever words and
  hyphens have been typed so far (the default wordlist uses alternate lists,
  where even numbered words have three syllables, and odd numbered words have
  two, so the completions depend upon how many words are present, not just
  the partial last word). E.g. get_word_completions("pr") will return
  {"processor-", "provincial-", "proximate-"}, while
  get_word_completions("opulent-pr") will return {"opulent-preclude", "opulent-prefer", "opulent-preshrunk", "opulent-printer", "opulent-prowler"} (note the lack of a trailing hyphen, because the
  wordlist is expecting a code of length two). If the wordlist is not yet
  known, this returns an empty set. All return values will
  .startwith(prefix). The frontend is responsible for sorting the results
  before display.
• h.choose_words(words): call this when the user is finished typing in the
  code. It does not return anything, but will cause the Wormhole's
  w.get_code() (or corresponding delegate) to fire, and triggers the
  wormhole connection process. This accepts a string like "purple-sausages",
  without the nameplate. It must be called after h.choose_nameplate() or
  MustChooseNameplateFirstError will be raised. May only be called once,
  after which AlreadyChoseWordsError is raised.

All Helper functions are synchronous (which makes them easier to use from a libreadline completion helper), but some can trigger refresh requests in the background.

For our approach, we'll add three inbound API events, sent from the Helper into the WormholeCore:

• InputHelperRefreshNameplates
• InputHelperChooseNameplate(nameplate)
• InputHelperChooseWords(words)

and two outbound API actions, emitted by the WormholeCore. The IO/API glue layer is responsible for delivering these to the InputHelper:

• InputHelperGotNameplates(nameplates)
• InputHelperGotWordlist(wordlist)

All of the API events get routed to the Input machine. InputHelperRefreshNameplates causes a Refresh message to be sent to the Lister machine, which sends a TxList to Rendezvous the next time it's connected. The server responds with a NAMEPLATES message, which is translated into a RxNameplates being sent to Lister, which sends GotNameplates to Input, which should then send InputHelperGotNameplates to the API layer.

The choose-nameplate event causes the Input state machine to move, which claims the nameplate and retrieves the wordlist (which is static for now, but eventually the server will store+deliver some "nameplate attributes" including a wordlist identifier). This triggers the got-wordlist action, which the API-side Helper object can store and use to support the completion functions. When the user finally finishes typing the code and calls h.choose_words(words), the API layer sends in InputHelperChooseWords(words), and the Input machine sends the completed code to the rest of the state machines.

I've got yet-another-new-approach for the event model. I originally went with a set of hierarchical enums (so MachineEvent::Allocator contained an AllocatorEvent, and then AllocatorEvent::RxAllocated was one variant of the latter). This allowed the top-level Core to dispatch based on the machine name, and then the individual machine processing functions do type-checking for us: Allocator.process(&mut self, event: AllocatorEvent).

But the return value could be any sort of MachineEvent (or API events, or IO events), and so I needed yet another layer of enums on top of MachineEvent (which I called ProcessEvent), and the return signature of the dispatch function was process(..) -> Vec<ProcessEvent>, and every clause in the dispatch function had to build a gigantic tower of return values to make it match that signature (ProcessEvent::Machine(MachineEvent::Allocator(AllocatorEvent::RxAllocated))).

So I switched to a big flat table of events (all events, even IO and API), using the machine name as a one-letter prefix, with names like A_RxAllocated. I added an enum of machine names, and a function that mapped event variant to machine. All the processing functions accepted and received the same Event type. But then the compiler thinks that the processing functions might be given an event that's really always dispatched to someone else (e.g. because A_RxAllocated is in Event, the Send machine has to prepare for receiving it too, even though it's always going to be sent to Allocator). So those process() functions got a lot of uncomfortable _ => panic!() clauses.

But.. now I've learned slightly more about generics and the From/Into traits, and I think I have a way to go back to the nested enums. Basically it's creating a new type (named Events) that behaves a lot like Vec (or maybe VecDeque), except that the .push() method takes anything that can be converted into a MachineEvent:

impl Events {
    fn push<T>(&mut self, item: T) where MachineEvent: std::convert::From<T> {
      self.events.push(MachineEvent::from(item));
    }
}

The processing functions accept machine-specific types (Allocator takes an AllocatorEvent), but they all return an Events. And when they add things to the Events, they can give it specific event objects instead of wrapping them in something first:

let mut events = Events::new();
events.push(Send::GotVerifiedKey(key));

I experimented for a while with having Events be a vector of "objects with trait X", where X meant that you could convert it into a MachineEvent if necessary, but I got stuck trying to deal with the Size issues. Eventually I decided that giving all processing functions the same return signature was important, and that meant the conversion into the common type has to happen as quickly as possible. Having a special vector-like thing which does the conversion on input seemed to do the trick.

I was able to copy the vec! macro from the Rust Book into the file, which makes returning multiple events easier:

// Rendezvous::ConnectionMade
match self.state {
    Connecting => events![Nameplate::Connected, Allocator::Connected, Lister::Connected]
}

Having that macro is sufficient. Which is good, because I don't think I'd be able to implement .extend() or any of the Vec methods that take multiple items at the same time, since the monomorphization is only going to work with one type at a time. And requiring the processing functions to build up the return vector one item at a time would be a drag.

I'm working on rewriting all the stubs to work this way now, so maybe don't put too much time into porting any of the other machines until that's done.

net2 crate has been deprecated; use socket2 instead

The net2 crate has been deprecated
and users are encouraged to considered socket2 instead.

See advisory page for additional details.

Hi,

While reviewing send example code under io/blocking and also core's example ws.rs I noticed that recent changes to input.rs causes it to panic. This happens because InputEvent::GotWordlist is generated before Input machine is started. In case of sending it is normal that input machine is not started, since user is not manually inputting the code using tab completion. 2 possibility of setting code is either via SetCode or AllocateCode both of which does not need Input machine.

Previously code was working fine because I was not generating panic in case if a event comes before it should. I think we should relax input machine a bit by not causing panic. In case of send any event to input machine should be just ignored keeping the current state as it is.

Since input machines whole purpose is interactive user input I think it should be bit relaxed.
@vu3rdd @warner what do you guys think?

The new parser for Welcome messages requires the motd field to be present, but this is optional (and in fact the server only includes one if the server operator wants to broadcast a message to all users, like a service interruption announcement or a plea for donations to cover hosting costs). So the serde_json parser needs to make this optional. (in fact pretty much everything in the Welcome message is optional).

This currently causes a panic in core/examples/ws, which connects to a (local) relay and prints the messages that it receives. The motd-less Welcome triggers an error.

This is a big pony (🐴 ), but it'd be amazing if magic-wormhole.rs worked well under Web Assembly.

My reason for wanting this is that I've long wanted to package magic-wormhole up as a browser extension, but my desire to write JS is nil. I've often joked about wanting to do a magic-wormhole Rust implementation and use WASM, so the fact that you're working on this is amzing!