The asset backends should eventually be implemented and loaded as plugins.

https://golang.org/pkg/plugin/
https://medium.com/learning-the-go-programming-language/writing-modular-go-programs-with-plugins-ec46381ee1a9

Among other things, this will require a module decred.org/dcrdex/server/asset.

Right now, the server is not storing the following match-related data.

1. 4 signatures = (1 'init' ack signature + 1 'redemption' ack signature) x 2 parties
2. The secret hash
3. The swap script
4. 4 coin IDs = (1 'init' + 1 'redeem') * 2 parties

All 10 are needed to reconstruct the signature messages for the match. Additionally, the coin IDs are needed for a DB-based TxMonitored solution, which means the data can't be stored as, say, an encoded blob as is done in #99. I could see a solution which stored all of this data in a separate table though, with its own archive policy.

We need to handle persistent data. A solution will likely utilize a relational database, though some data might be better handled with a key-value database. The primary data to be stored is order data and account information. Orders for storage will come in the form of the order type used in the order book manager / matching engine, so there is some overlap with #5.

We need an order matching engine. Given an order book and an epoch queue, the engine should generate matches and handle cancellations. This is a low-level DEX component that will be used by the order book manager, so no considerations for client notification or other higher-level DEX functions should be made.

This work will involve the creation of an order type and an epoch queue type. The order type will be widely shared among higher-level DEX modules. A market's epoch queue will be shared with an order book manager the market manager, which will be responsible for building epoch queues and sending them to the matching engine along with the order book (standing orders).

Keep in mind that there may be interest in independent verification of order matching, so there will eventually need to be some clean exported types/interfaces.

Relevant specification sections:

• https://github.com/decred/dcrdex#Epochbased_Order_Matching
• https://github.com/decred/dcrdex#Client_Order_Management

The DEX wallet will need a client to interact with a bitcoin wallet. This will likely be a wrapper around the necessary methods of rpcclient pointing to a btcwallet.

• Sync UTXO set (ListUnspent)
• Signing UTXO data (SignMessage)
• Signing transaction inputs (SignRawTransaction)
• Broadcasting transactions (SendRawTransaction)
• Building the swap and redemption transactions

Ideally, the wallet will translate between some common type used by the higher-level DEX wallet infrastructure, and the btcd/rpcclient types. Those common types, and an interface for common wallet client functionality can be specified during this work (and with #36).

This component should be viewed as a plugin component of what could be a simple CLI program or a larger multi-wallet. It is not a stand-alone swap negotiator, so shouldn't be concerned with the negotiation details.

If possible, the wallet should be designed such that it can be reused for bitcoin clones. See ReadCloneParams for more info.

txWaitExpiration is hard coded in swap.go:

// txWaitExpiration is the longest the Swapper will wait for a coin waiter.
// This could be thought of as the maximum allowable backend latency.

This needs to be an operator setting. Discussion:
https://github.com/decred/dcrdex/pull/140/files#r370346846

From conversation on chat.decred.org

...it'd probably be good if there was a "read-only" order book feed too that didn't even include order IDs, at least not 64-character order IDs. No commitments, no match_proof notification. Bare minimum.

The vast majority of bandwidth for order book feeds will be eaten by order IDs, commitments, match_proof notification, and other information not necessary to simply watch the order book. As @chappjc pointed out in regards to client-side validation

a client without an order in a given epoch is unlikely to care [about validating]

Lite subscribers could be differentiated from validating subscribers, and the order book router can send appropriate messaging for each.

The DEX wallet will need a client to interact with a Decred wallet. This will likely be a wrapper around an rpcclient pointing to a dcrwallet.

• Sync UTXO set (ListUnspent)
• Signing UTXO data (SignMessage)
• Signing transaction inputs (SignRawTransaction)
• Broadcasting transactions (SendRawTransaction)
• Building the swap and redemption transactions

Unlike other wallet clients, the Decred wallet client will also need to create/broadcast DEX registration fee payments.

Ideally, the wallet will translate between some common type used by the higher-level DEX wallet infrastructure, and the dcrd/rpcclient types. Those common types, and an interface for common wallet client functionality can be specified during this work (and with #37).

This component should be viewed as a plugin component of what could be a simple CLI program or a larger multi-wallet. It is not a stand-alone swap negotiator, so shouldn't be concerned with the negotiation details.

A swap executor is needed to coordinate the atomic swaps as described in Atomic Settlement between one or more clients whose orders have been matched by the Order Matcher.

The swap executor will interface with:

1. The asset backends, for txn monitoring (receive broadcast ntfns and track confirmations) and validation (sufficient UTXO value, correct contract amount, etc.).
2. The communications hub, to: (a) inform clients of the match, (b) receive swap transaction details from the clients, (c) send swap transactions to the clients. Alternatively, the market manager can handle comms for these events, which would be notified to the market, who owns the swapper.
3. The order matcher, from which it will receive the matched order, including amounts and account public keys. The market manager will relay the matches to the swap executor.
4. The market manager, which controls the swap executor by instantiating executors and receiving swap outcomes from the executor.

https://github.com/decred/dcrdex/pull/133/checks?check_run_id=406320272#step:6:94

--- FAIL: TestRouter (0.01s)
##[error] routers_test.go:1118: (first link, market 1): no response sent for subscription
FAIL

The subscription router will handle requests to the orderbook route. The order book updates will probably be sent from the market manager and sent through the subscription router to any number of subscribed comms.Links.

Since the subscription router needs to have some concept of what a market is, it could potentially be part of the market package.

This relates to issues #122 and #123, which require changes to the config response.

The config response, which is not yet implemented, needs some changes to communicate per-market variables clearly. The proposed response:

Market object

Asset object

As defined by the current spec (Pseudorandom Order Matching), order matching begins by shuffling the epoch queue in a deterministic manner by seeding a Fisher-Yates shuffle with the hash of the concatenated order hashes.

There are some concerns about this:

• Part what affects the order ID (e.g. order1, etc) are the client and server time stamps.
• (independent, and more importantly) The server(s) and all the clients know what the epoch order would be if it were to close with its current orders.

The change proposed (by @behindtext) to the epoch shuffling algorithm will eliminate the above concerns, but not without a complexity trade-off:

1. Each submitted order will include a commitment. This can be the hash of a random number known only to the client submitting the order.
2. At the close of the epoch, each client will be required to submit the value to which they committed. EDIT: The close of the epoch should be marked by a special server broadcast message containing the epoch's order IDs and commitments. The clients must not submit their preimages prior to receiving and validating this server-signed message.
3. The seed is based on the hash of the concatenation of the commitment preimages.

With the above shuffling algorithm, it is impossible for the others to collude against you (as a client) as they would need to know your commitment value.

Issues:

1. If a client does not respond with their value? End of queue or fail their order?
2. If no clients respond with their value? Invalidate all orders?
3. (the obvious one) Order matching is delayed by X seconds until clients have provided their commitment value. Is the delay acceptable? Is the complexity acceptable?

1. A good description of the DEX
2. Contribution guidelines

From the discussion in #81 (review)

The setting will have to be moved from a top level payload field in the config route response to a field in the asset objects list.

https://github.com/decred/dcrdex/blob/master/spec/fundamentals.mediawiki#exchange-variables

_Originally posted by @buck54321 in #81 (review)

The specification does not yet describe any API endpoints or a messaging protocol for retrieving account-related trading data such as orders and matches. Re-connection handling will likely require the ability to query the status of at least 1) active orders and 2) active matches, so a minimum implementation would retrieve at least those two things. These routes are authenticated, so will use the AuthManager (#50) to register the routes.

While eventually, we will want to serve this info through HTTP, initial work should just use the websocket framework in place.

The spec will need to be updated too.

The specification could benefit from some pagination. Could we keep a table of contents and a DEX overview where the spec is, but break off some of the more technical sections to their own mediawiki files? The "chapters", each with its own file, could roughly follow the current table of contents.

1. Communication Protocols
2. Exchange Fundamentals
3. Account Management & Community Conduct
4. Data API (including order book subscriptions)
5. Order Management
6. Exchange Administration

The specification is bound to grow in size. I think this would provide some sanity to the information.

We could use a shell script to iterate our packages and run go test and golangci-lint.

See https://github.com/decred/dcrdata/blob/master/run_tests.sh for an example.

The BTC exchange wallet was designed to be re-used in much the same way as the BTC server backend was. That means the LTC exchange wallet can (hopefully) be implemented with very little effort. See the server/asset/ltc package for an example.

Previous discussion

Reconnect handling for the Decred server backend is shoddy. At the very least, we need to resend the height on the rpcclient.Client's OnClientConnected notification, which I believe will still be sent even if auto-reconnect is used.

The server needs to handle various aspects of user authentication. The auth manager will minimally

• verify request and acknowledgment signatures
• handle client communications, connecting the account ID to the websocket connection and buffering some messages across reconnects.
• handling conduct violations

For convenience, I think that the auth manager could also handle server signatures. See this auth manager interface to see how it will be used.

The auth manager will depend on the upcoming storage module #7, but the storage methods can and should be implemented as an interface, so there's no reason to wait.

One of the only respectable technical critiques the DEX spec has gotten came in an article by the Komodo AtomicDEX team.

It's a rudimentary atomic swap implementation because it currently only supports assets of the Bitcoin protocol.

... the protocol only supports UTXO-based assets. Whereas most hybrid DEX protocols only support Ether and ERC tokens, limiting their capabilities, the Decred DEX falls on the opposite side of the fence but runs into the same problem— only supporting UTXO coins excludes support for the Ethereum ecosystem, as well as a number of other prominent blockchain protocols and platforms.

The valid part is that as specified, Decred DEX seems like it will only work with UTXO-based assets. But here's the thing. There is no fundamental barrier to supporting trades with ETH-based assets and probably others. With a general enough API, we can accomodate virtually any asset that supports some equivalent of hash time locked contracts. And while I'm not suggesting that we should put our efforts towards this work right now, getting the API correct now greatly eases our pains in the future.

What I'm proposing here is that we replace every txid-vout pair in the specification with a single parameter called coin ID, coin being a general term here meaning some discrete amount of unspent asset identifiable on the blockchain. It is represented as a byte array. The backends and wallets for each asset can encode and decode whatever identifying information is needed in the coin ID. For the assets we are implementing now, it will likely be just the concatenated transaction hash and big-endian 32-bit vout.

Coincidentally, this resolves some questions we had earlier on how best to serialize the transaction ID, since the backends are probably byte-reversing the transaction hash. The answer, I'm hoping, is that the DEX doesn't need to concern itself with such details if you just give it a byte array.

This change also simplifies a lot of things internally for the DEX. Two columns become one in some tables, two arguments become one in some functions, two fields become one in some types. Outside of the asset backends, the DEX itself never has to decode the coin ID.

As it stands, each DEX account has its own private key. The key is stored unencrypted, which is obviously nonsense.

1. Does each DEX need its own key, or should there be a single app-wide key?
2. Do we need to generate a mnemonic seed so that accounts can be recovered?
3. dcrwallet's snacl was my first instinct, but @jrick has expressed some reservations about it, providing a possible path forward, though.

secretbox is still pretty good, but i would like to replace scrypt with argon2id

I will note that I don't fully understand the objection to scrypt or the snacl implementation in general, but if there are better options, it's certainly worth consideration.

Luckily, this is an implementation detail that doesn't require any consideration for future alternative clients.

The client app will need a reasonably powerful UI to be useful. This means:

1. one or more menus for choosing actions
2. a subdivided terminal window
3. tables (buy and sell book, user's own orders, market trade history)
4. charts (rudimentary text-based depth chart, and perhaps a market history).

Some resources for accomplishing this:

• https://appliedgo.net/tui/
• https://github.com/gizak/termui
• https://github.com/mum4k/termdash
• https://golangexample.com/a-cross-platform-customizable-terminal-based-dashboard/ (termdash)
• https://golangexample.com/compositional-widgets-for-terminal-user-interfaces-written-in-go/ (gowid) / https://github.com/gcla/gowid
• https://github.com/marcusolsson/tui-go
• https://github.com/jroimartin/gocui
• https://github.com/rivo/tview
• https://github.com/gcla/gowid
• basic prompts: https://github.com/AlecAivazis/survey

Note that zkclient uses some kind of terminal UI, but I have not looked to see what it is.

This is maybe more of a micro-component, and could maybe be integrated into the auth module.

We need to handle the register and notifyfee routes, used to register new client accounts. These routes are not authenticated. Basic workflow is

1. Accept client pubkey, and supply DEX credentials: registration fee payment address, DEX pubkey, etc.
2. Validate on-chain registration fee payment, activating the client account.

There is an open question about how the registration fee addresses will be generated, so we'll have to figure that out too.

TxMonitored replacement, for identifying outputs that were from DEX txns. For a DB powered solution, this requires #112, the match DB updates, because the coinIDs are added to the match DB there.

To penalize accounts that exceed the cancellation threshold, it is necessary to query the number of matched cancel orders created by a given account over N of their most recent orders.

The AuthManager handles account penalization, and there are two logical places where the AuthManager can be notified of executed cancellations:

1. Directly by (*Market).processEpoch, immediately after the Matcher unbooks the matched order from the book. Each Market has an auth AuthManager field that will facilitate this.
2. By (*Swapper).Negotiate, which is called by (*Market).processEpoch after all epoch matches are processed and coins are locked/unlocked.

I believe cancellation statistics should be updated as soon as possible after an order is removed from the books by a cancel order, hence (1) is my choice. Further, processEpoch is a blocking call in the Market's epoch processing loop, so the penalization is more likely to into effect in a timely manner, although there are multiple markets.

Since cancellation statistics are DEX-wide (not per-market), the AuthManager must:

1. Keep a user's recent (last N=25) order history in memory (just the order type and ID), using a slice as a deque. e.g. map[account.AccountID]orderDeque where orderDeque implements push/pop methods on an order slice. Question: specifically what orders should be counted in this list? It should NOT include orders that "failed" in epoch processing because users could just create limit orders with TiF immediate and a rate that is certain not to match, thus easily padding their recent order history with non-cancel orders without having to preform any trades. Hence only the doneOK output of (*Matcher).Match should be the orders to count in this list.
2. Querying the DB for recent order history when no in-memory history is available.

To directly communicate executed cancel orders to AuthManager, I suggest adding a RecordCancel(user account.AccountID, oid order.OrderID) method to the AuthManager (and the interface in the market package). (*AuthManager).RecordCancel should update it's recent order deque for the user, and if the deque does not exist in the account map, query storage for the N latest relevant orders.

... I was thinking of specifying a delay duration instead of absolute time, that way a production server could set it and forget it, e.g. "markets re-open 10 minutes after the server starts".

_Originally posted by @buck54321 somewhere

In order for the client to be able to independently verify matching, they need to be able to produce the same series of random numbers based on a given seed. This means that we must use the same exact random number generator. Right now, the generator used is the default from Go's math/rand, which is only described as algorithm by DP Mitchell and JA Reeds. Googling gives some indication of the source, but it's not clear how common this implementation is. If Mitchell/Reeds is uncommon, but there is some more common algorithm we could use, we should.

This should be operator configurable. I see in the spec that btimeout is returned as part of the config response, but not listed in the Exchange Administration section, which I think is a mistake.

_Originally posted by @buck54321 in #81 (comment)

The market buy buffer is configurable per-market. This needs to be changed in the spec too.

Pertaining to the change in #81 (comment)

We need a communication hub. The hub will accept incoming websocket connections and route bi-directional communications. The communications hub is a low-level component and should have zero knowledge of what a DEX is. Incoming communications will likely be routed based on the JSON-RPC method, without parsing of the params. Outgoing messages will be directed at a particular client, all clients, or a subset of clients (as in the case of an order book subscription).

Relevant specification sections:

1. https://github.com/decred/dcrdex#Communication_Protocols
2. https://github.com/decred/dcrdex#Client_Order_Management

The read-only HTTP (REST) API should be part of a standalone microservice. The computational load for API requests need not burden the DEX server and potentially interfer with the critical operations of order and swap processing. The data API service could connect to the same postgresql database server, or better to a replicated database to further isolate dcrdex from the API service.

This is maybe more of a micro-component, and could maybe be part of the market package. That said, it is a largely independent component, so it can be worked on separately.

Basically, we need a component to manage the limit, market, and cancel routes. The flow is essentially

1. unmarshal
2. authenticate
3. translate to order type
4. send to market manager

With assets that require best block polling (vs. notifications), the cached best block may not be the previous block when the best block is retrieved.

e.g.

[INF] ASSET[btc]: Reorg from 00000000000016b77e1c1de8276574f3069f82e3d6b94f9e1e2fe0e19178367b (1657353) to 0000000000000e9e107dee915650dd2edaddfc11fc3685810ec3eca979debed4 (1657355) detected.

1657354 and 1657355 were so close together that when the BTC backend with a current cached best block of 1657353 polled for a new block, it saw 1657355 and decided it was a reorg. Fortunately it would find that 1657354 is mainchain and not delete anything from cache, but the logic may need to be adjusted so as not to mislead future code depending on accurate reorg detection, or at least accurate logging.

The txsizes package in dcrwallet was previously internal, so we just copied a lot of the code for dcrdex. We can use directly now. decred/dcrwallet#1573

diff --git a/.github/workflows/go.yml b/.github/workflows/go.yml
index df46458..fec64c9 100644
--- a/.github/workflows/go.yml
+++ b/.github/workflows/go.yml
@@ -29,4 +29,5 @@ jobs:
       - name: Test
         env:
           GO111MODULE: "on"
-        run: go test -v -short ./...
+          GORACE: "halt_on_error=1"
+        run: go test -race -v -short ./...

finds a bunch of races

Several types' constructors launch goroutines automatically. These types either have no lifetime management yet, or implement the Stop/WaitForShutdown pattern.

After some discussion on Matrix, it seems worth while to refactor these types so their goroutines are launched manually by the consumer of the type via Run(ctx context.Context). Alone this is more work for the consumer since they have to call Run in their own goroutine with a WaitGroup, but as @davecgh pointed out, it is trivial to wrap this in a "StartStopWaiter" type that will do the Start/Stop/Wait... stuff for any such type that implements a Run(ctx context.Context) method.

Each market (asset pair) on the DEX server will be run by a Market Manager. The market manager integrates a number of core components, employing a book manager, order matcher, swap executor, etc. to run each market. The DEX manager will control a number of such market managers.

A market manager has the following functions:

1. Coordinate epochs (timing), and the creation of their queues.
2. Validate incoming orders from the communications hub, relying on the asset backend (e.g. DCR backend) and account manager.
3. Add valid orders to the active epoch queue.
4. Initiate order matching at the end of an epoch via the matching engine.
5. Update the order book with new standing orders, and remove matched orders, via the book manager.
6. Initiate swaps for matched orders via the swap executor.
7. Recording all of the above operations via the persistent storage backend(s).

This will arguably be ❤️ of the DEX.

Here's an initial suggestion for how the dcrdex repo might be organized:

dcrdex
├── client             # client libraries
│   ├── ...            # user, ws/comms, orders, etc.
│   ├── cmd
│   │   ├── dexclient      # CLI app
│   │   └── dexwebclient   # e.g. http://127.0.0.1:7345
│   └── docs
├── README.md
├── server
│   ├── accounts    # account/user manager
│   │   └── pki     # possibly separate for sig. and verif. functions
│   ├── archivist   # persistent storage
│   │   ├── kv      # e.g. bbolt, badger
│   │   └── sql     # e.g. postgresql
│   ├── assets      # i.e. interface.go
│   │   ├── btc     # implement the interface
│   │   └── dcr
│   ├── book        # order book manager
│   ├── cmd
│   │   └── dcrdex  # main()
│   ├── comms       # communications hub, ideally abstract transport
│   │   ├── jsonrpc # primarily the types and things the client uses too
│   │   └── ws      # websocket (perhaps other transports)
│   ├── dex
│   │   ├── admin      # administrative tools and portal (may need RPC server too)
│   │   └── controller # controller for multiple markets, users, api, comms, etc.
│   ├── docs
│   ├── marketman   # market manager
│   ├── matcher     # order matching engine
│   └── swap        # the swap executor/coordinator
└── spec <- README.mediawiki moves here

We need a client to interact with the DEX server. A quick review of the spec indicates that the client would need at least the following functionality.

• Registering a new account
• Maintaining a websocket connection
• Querying for asset info
• Sending orders, progress notifications, etc.
• Syncing orderbook data

The client should be mostly free of dcrdex dependencies, but sharing some types via /server/comms/rpc (#33) is expected.

Likely realted to switching to go 1.13.

go: github.com/ltcsuite/[email protected] requires
	github.com/ltcsuite/[email protected]: invalid pseudo-version: does not match version-control timestamp (2019-05-07T13:33:22Z)

Each market requires a book manager, which is a relatively small component responsible for:

1. Keeping the order book (inserting and removing orders)
2. Providing book stats like depth tables for DEX clients and generic HTTP API consumers.

The book manager will be employed exclusively by a market's market manger, a higher level server component that employs most of the core server components.

The core server modules according to the DEX proposal are:

• Messaging and communication hub
  • JSON-RPC websocket client messaging
  • Interfacing with other DEX components that require sending and receiving messages or notifications.
  • Issue #6
• Order book managers. For a market, keep the book:
  • Insert and remove orders
  • Provide book stats like depth tables for charting
• Order matcher
  • Operates on (1) epoch queue and (2) order book to make matches for swap
  • Issue #5
• Swap executor
  • Coordinate clients from match to completed swap.
  • Issue #9
• Asset backends
  • Define an interface to be satisfied by all backends, and used by other components to interact with the assets.
  • For each asset, implement txn monitoring for swaps, fee payment, etc.
  • DCR: #3
  • BTC: #4

Compared to the server's persistent storage needs (#7, #47), the client is not as constrained in terms of performance requirements, so pretty much anything is on the table here. It may be worth looking at dcrwallet's bbolt-based database, which I think has no external dependencies.

In addition to the Core Client Components milesone issues, a command line interface (CLI) application needs to be created for the DEX client.

Work on this issue should follow development and testing of the core client components.

A suggestion I have for making the CLI app as interactive and intuitive as possible is to make use of menus and a clear workflow. To this goal, one tool that I have had good luck with in dcrdata's democlient CLI for the psclient package is https://github.com/AlecAivazis/survey.

We need a backend for BTC. For this work, it is assumed that the DEX operator has access to a full node with txindex, and that queries will be performed using the blockchain node software's standard RPC.

Part of this work will involve creating some common DEX interfaces. I anticipate there will be an interface for the asset backend itself, one for UTXOs, and probably some for transactions and blocks. These interfaces will necessarily need to be defined in a separate module. This work should also be coordinated with the analogous DCR backend development (#3), so that the interface can be accommodating for both. The DCR and BTC backends will be baked-in, but the interfaces defined during this work will also be used for plugins.

The asset backend will need to have at least the following functionality.

• Retrieve UTXOs
• Broadcast notifications for new blocks and mempool transactions
• Validate signatures
• Calculate script sizes (some script sizes might be constants)

https://github.com/decred/dcrdex/pull/133/checks?check_run_id=406422759#step:6:32

--- FAIL: TestClientMap (0.01s)
##[error] webserver_test.go:395: connection not closed on server shutdown
FAIL
FAIL decred.org/dcrdex/client/webserver 0.044s

It'd be handy to be able to quickly initialize the state of a set of simnets with > coinbase maturity blocks, the block reward being directed to a set of wallets that the CLI client can access. We'll also need the ability to trigger reorgs.

@chappjc has previously done much of this using shell scripts and tmux, and maybe that's the right approach here, but we'll need to add the wallet functionality. I could also see doing this in go.

Seems like shell script is the way to go. Really just need to bring together the script from @davecgh

https://gist.github.com/davecgh/3da3c48bd9ab1914ff134c0b26b8661c

and the reorg scripts from dcrdata

https://github.com/decred/dcrdata/tree/master/dev

We need a backend for DCR. For this work, it is assumed that the DEX operator has access to a full node with txindex, and that queries will be performed using the blockchain node software's standard RPC.

Part of this work will involve creating some common DEX interfaces. I anticipate there will be an interface for the asset backend itself, one for UTXOs, and probably some for transactions and blocks. These interfaces will necessarily need to be defined in a separate module. This work should also be coordinated with the analogous BTC backend development (#4), so that the interface can be accommodating for both. The DCR and BTC backends will be baked-in, but the interfaces defined during this work will also be used for plugins.

The asset backend will need to have at least the following functionality.

• Retrieve UTXOs
• Broadcast notifications for new blocks and mempool transactions
• Validate signatures
• Calculate script sizes (some script sizes might be constants)