@Gromak

mpcVerifyBlock assumes that block continues main chain, but it shouldn't do it, it should check MPC-related predicates about block in isolation.

@neongreen

Currently it's been decided to store Utxo for each block. This can potentially waste a lot of space. There are several better options available:

• Continue storing Utxo for each block, and hope that Map sharing works – any update or delete requires O(log n) extra space, so applying a transaction would require O((txInputs + txOutputs) * log n) extra space. When serialising, store only Utxo for the first block in the epoch. When deserialising, recompute Utxo for each block.

• Use whatever Bitcoin uses (undo database).

• Use deltas. A UtxoDelta is a (Map, Set) tuple – when something is added to the Utxo it's actually added to the first Map, and when something is deleted it's added to the second Set (original Utxo itself never changes). This would require, in total, as much extra space as there were transactions in the epoch. Lookups in Utxo would cost O(log n * number of deltas) (deltas can be chained).

Also, local data should be kept in memory.

@Gromak

It should be added to verifyHeader. In other words, if there are two blocks from different epochs, there must be genesis block between them.

@neongreen

@Gromak

As you probably know, we have a time-warp library which provides some features, one of the most important is emulation mode. In emulation mode some things are emulated, for example, time (functions like threadDelay are called wait and don't take real time), network communication (these functions don't really used network and allow you to emulate network delays/failures/etc.). The purpose of it is to be used for testing.

Eventually we want to have tests where whole system will be launched in emulation mode, some arbitrary operations will be made, so that system reaches arbitrary (but valid) state, and then properties (QuickCheck) will be checked. Also we can add byzantine actors, emulate errors, etc.

Note that all the code is written in WorkMode, which contains several monads, For all monads we know how to implement emulation mode. Unfortunately, after recent changes some monads don't have emulation mode implementation, but it's temporary. However, it doesn't stop you, because you can temporary use real mode (which will be slow and not stable, but will work).

Example of doing such stuff can be found in [rscoin|https://gitlab.serokell.io/rscoin/rscoin/tree/master/test/Test/RSCoin/Full]. Please how it's done there and ask me questions if you have any. This code should give some inspiration maybe. Anyway, I hope now it's more or less clear what to do, but if not, please ask questions.

@Gromak

As far as I understand, currently such check is not performed. Well, everything is checked, except the main part: list of leaders.

This should be done in verifyBlocks I guess.

@volhovm

@Gromak

There are at least two problems:

1. If node becomes offline during one of the first k slots, it wants to participate in MPC and it can, but it won't do it in current version;
2. Currently secret stored in MpcStorage is unset when genesis block is processed by this storage. And it's set only at the beginning of epoch. There is a problem, because genesis block for k-th epoch may be processed after the beginning of k-th epoch. For example, if there was a rollback which affected k - 1-th epoch. In this case stored secret will be unset and will never be set again for k-th epoch.

@georgeee

From Charles:
2.
https://en.bitcoin.it/wiki/Protocol_documentation
check out the block sections
https://bitcoin.org/en/developer-reference#block-headers
in the block format

there are two components
block header and block body
for the header, I'd like you guys to explore using both Cherpernoy trees and merkle trees

George:
2. We also thought of integrating both Chepurnoy & Merkle trees. Former has benefit of allowing cool verification, later for allowing SPV nodes check tx achieving block w/o downloading blocks (which as far as we understand doesn't hold for Chepurnoy)

@Gromak

1. Pass SlotId so that message is completely ignored if it's not relevant for current slot. It's already stored in main Storage. Be careful to consider case when current slot is not the same as last seen by Storage (btw, I am not sure if it's possible at all).
2. For opening and share check that corresponding commitment exists.

@Gromak

@georgeee

Sometimes node is stuck in time-slave mode, doesn't switch to regular node mode.

Need to debug this.

@georgeee

@georgeee

Go through BIPs, get understanding, mark ones that might be useful for Cardano and comment out in [Research] tasks.
https://github.com/bitcoin/bips

From Charles:

bips 70-75 are payment related
it might help a bit when deciding on address structure
bip 120 ans 121 as well
bip 130 is likely not in haskoin
https://github.com/bitcoin/bips/blob/master/bip-0130.mediawiki

@Gromak

In this mode node (i. e. pos-node executable) accepts --dht-leader flag which means that this node should set its start-time to Just currentTime.
All other nodes set their start-time to Nothing by default and request it from peers until they receive this information. We don't care about consensus, we assume that all nodes are honest with this regard and dht-leader selection is offloaded to deployment code.

@Gromak

@georgeee

Remove this workarounf after TW-47 would be processed

See TODOs in code (with mention of TW-47)

@Gromak

1. New listener should be added to src/Pos/Ssc/DynamicState/Instance/Listeners.hs
   1. (vssPk, spendingPk, sig) should be provided, where sig is signature of vssPk by spendingPk
   2. spendingKey should be checked against UTXO to actually hold some stake
   3. Then vssPk should be added to MainBlock, into VssCertificatesMap
      1. For this we need vssPks be broadcasted in same way as transactions to be further put to blocks
      2. Probably need to abstract code, which propagates transactions now to work with vssPks as well (and maybe w/ smth other in future)
2. Code, which processes new block, should be updated in order to handle these VssCertificatesMap updates . But only last certificate for concrete spending key is to be assumed as valid.
3. Add code, which sends certificate on node start

@Gromak

1. It sounds more reasonable.
2. Looks like secret sharing functions don't work with empty lists (e. g. 0 participants). Even if they somehow work, it doesn't make sense to pass empty list there. So Pos.Crypto.SecretSharing should be modified to use NonEmpty.

@georgeee

Now we have a lot of evidence collected (both locally & remotely) that reply still works not always as expected (and it's not really hard to reproduce).

Need to finally get this shit work.

@Gromak

I am not 100% sure that such situation is possible (i. e. that we received block, processed it and after sometime use PBRmore with this block), but if it's possible, we will be in infinite loop, because we ignore already known blocks.
Also it is optimization, but we don't really care.

@Gromak

When node generates a secret (currently it happens at the beginning of 0-th slot of any epoch), it should add corresponding commitment to mpcLocalCommitments and put it into generated block later (if it's leader). Currently it's not done.
I don't know about openning for own locally stored secret, but it should also be added to local openings and to block.

@Gromak

--I propose to use structure similar to the one used by TxStorage: i. e. store local data only for the current version and global data for all versions. When rollback happens, we filter local data to make sure that it's valid.--

I doesn't make much sense to store local MPC data on disk. Size of it will be limited, because size of block will be limited (for example, I think block must fit into RAM). It means that we should move local MPC data from GtStorage. It also means that we need to move getLocalSscData from SscStorageClass. As well as sscProcessMessage. Good question is where to move it :)

I propose to create SscLocalDataClass type class, as well as SscLocalData type family in Ssc. SscLocalDataClass will contain:
• sscNewLocalData :: MonadIO m => m (LocalData ssc)
• sscDestroyLocalData :: MonadIO m => LocalData ssc -> m ()
• sscProcessMessage :: MonadIO m => SscPayload ssc -> SscMessage ssc -> LocalData ssc -> m () -- first argument is global payload which is taken from main persistent storage
• sscGetLocalPayload :: MonadIO m => SlotId -> LocalData ssc -> m (SscPayload ssc) -- maybe should take global payload too
Note: it's pseudocode.
It can be implemented as IORef, for example.
I want @neongreen to comment this solution, maybe it won't work for some reason.
Regarding atomicity: I don't see any fatal problems. We must be careful though, because, for example, when we pass global payload to sscProcessMessage it may happen that global payload changes. But actually we just need to check local payload before adding it to block, that's all.

@georgeee

CSL even on it's early stage is a system with non-trivial workflows and not always opaque behavior. Sometimes one needs to peer in logs for hours to understand why system in whole works not as expected.

This issue is for such kind of activity.

Please if you find a bug (which is more than 10 mins to fix), create a separate issue for it.

@Gromak

Review all checks and verifications we have in all parts, such as receiving block/tx/whatever.

Need to provide extensive document describing all of them.

Also it would be the best if this document would be actually mapped to code.
E.g. in a similar way to github comments (some pieces of text between lines of code), but with possibility to extract all of comments as single document.

@georgeee

• don't launch supporter
• support no_dht mode (i.e. when node explicitely communicates to all peers)
• support launchin N > 3 nodes

@Gromak

It makes sense to make all checks optional.

@Gromak

Let's use usual base64 encoding where necessary.

@georgeee

From Charles:

Charles:
we also have to decide on block size
my advice is start with something like 2 mb and then include a metric to gradually increase block size over time
something connected to estimated rate of growth in disk space and also network capacity
it depends upon block generation rate
and also average transaction size
but we have to have an opinion on it

George:
Do we need to explicitly restrict block size? Can't we just include all transactions ready-to-go and don't care much about block size? (edited)

Charles:
yes because you can spam the network
I can create a block of massive size and no one can download it

George:
Sure, obviously.

Charles:
there is a lot of content on this. It's an interesting topic to consider

George:
But can't we just hardcode this value and increase it eventually in new releases. Having well-written update system nodes won't operate with blocks of new size if they didn't update yet

Ivan:
Increasing block size limit manually means changing protocol rules, it's not easy (AFAIK)

Charles:
I suggest you guys add this to your list of topics to review
google bitcoin blocksize debate
ivan is correct
the size is a bit premature to set
usually you have three factors driving the conversaion
the expected TPS
the rate of blockchain growth
and network propagation of blocks
refer to this paper

http://www.gsd.inesc-id.pt/~ler/docencia/rcs1314/papers/P2P2013_041.pdf

we need to understand more about the PoS algorithm
and it's impact on the system
as well as expect Transaction size
so it will likely be the last thing we fix prior to the launch

https://bitcoincore.org/en/2016/10/28/segwit-costs/

@Gromak

@Gromak

When node receives valid MPC message, she should send it to neighbors, like it's done with transactions. The simplest approach (used in Tx and Block parts) is to use worker.

@Gromak

I think we need to provide instructions for local deployment and for remote deployment.

@Gromak

See CSL-15. It's already done for commitments/openings.

@Gromak

1. Provide per-module and per-exported-function documentation with correspondence between the implementation and the paper. (Customer: Galois, Chepurnoy's Team)
2. For infrastructural code, provide reasoning behind picking given approach and plans regarding further work. (Customer: WellTyped, Chepurnoy's Team)
3. Emphasize threading model. (Customer: WellTyped)
4. Emphasize how time-warp is used and how we want to improve it. (Customer: WellTyped)
5. Write a guide to pos-prototype for newcomers (Customer: IOHK, Serokell)

@georgeee

Get understanding what exactly is segwit and why it's so needed for us to have it.

From Charles:

I'm really interested in how MAST and segregated witness will allow us to throw away as much signature data as possible
if it's possible, then we will be free to use a lattice signature scheme
As for 3, the seg witness and MAST, really try to get a deep understanding here
if we can launch with post-quantum crypto
that makes my marketing life a hell of a lot easier

@shersh

It will be good to generalize logWarningLongAction so that if given duration passed, it waits again (for the same interval or in geometric progression, dunno) and prints if action hasn't finished yet.
This way we'll know whether these actions take too much time or infinite time

@Gromak

When we calculate seed for new epoch, we should check that we have commitments representing majority of stake. If it doesn't happen, we shouldn't calculate seed using GodTossing data from current epoch. Similar thing should happen when we can't recover enough secrets.

• If commitments from participants controlling more than 50% of stake are used for randomness generation, we consider MPC round successful and use generated randomness for FTS.
• Otherwise we consider MPC round failed and use fallback solution discussed earlier.

@georgeee

Not sure it's still reproducible

@Gromak

@Gromak

Sometimes block is created much later than it should be. Dunno why it happens, need to investigate.

@Gromak

For example, blocks processing, forks resolution, utxo maintenance, rollbacks, etc.
Currently it's done in a way which came into mind first.

@Gromak

ensureOwnMpc uses last seen slot id. Sometimes it happens that we need to create block for slot older than last seen (it's very suspicious, probably it's bug as well, but let's handle it gracefully). For example, if last seen slot is 2 * k-th, but we create block for older slot, we will put opening there.

@Gromak

If we receive block which can continue main chain, we adopt it and stop. However, we should check whether alternative chains can be adopted after this change and do it if possible.

@Gromak

wait $ for (slotDuration - networkDiameter) is not the best way to find time to create block, because onNewSlotWhenLeader may trigger not at the beginning of the slot (e. g. if we launch application in the middle of slot). Better solution is to add slotBeginning :: MonadSlots m => SlotId -> Timestamp function and replace wait for with wait till.

@Gromak

@Gromak

What do you think?

Btw, AFAIK, verify is synonym of check. I try to use the former for functions which return VerificationRes and the later for functions which return Bool.

@Gromak

@georgeee

From Charles:

For address structure, let's do a double hash with sha3 and then blake2 with checksum in a human readable base58 format
we also have to consider the idea of address compression. For example, we could potentially have a large amount of addresses aggregate to a single address via a merkle tree
this would be relevant for redemption from RSCoin to Cardano SL
https://en.bitcoin.it/wiki/Technical_background_of_version_1_Bitcoin_addresses
Have your guys review this document to gain a fuller understanding of addresses
I'd like to repeat using two hashes in the process of address generation
it's a client side cost and doesn't impact the core protocol so it's worth spending
let me know about checksum preference
as for address format
human readable addresses is a vogue topic in bitcoin land

I'd like to see some ideas about what is possible

@Gromak

• Refactor logging after TW-74 done
  ** Suppress error messages from kademlia about expected errors
• Extract constants
• Prettify code that is easy to prettify

Please don't spend more time on this issue, than is scheduled. We don't need extra nice code here.
But eventually we may think of extending Kademlia algo ans thus rewriting part of this library.
But this is separate issue.

@Gromak

Currently we are using acid-state as the only persistent storage. There are many reasons why we do it and it's fine for experiments, but we are not sure about its applicability in production. I have written some thoughts about it in [slack|https://serokell.slack.com/archives/pos-prototype/p1477579716003107], let me copy them here (but better click link and see there):
Ok, about DB. First, about ⁠⁠⁠⁠acid-state⁠⁠⁠⁠.
Pros:
• convenient interface in general, you have some data type which stores all the data you need and you operate in ⁠⁠⁠⁠MonadState⁠⁠⁠⁠ and ⁠⁠⁠⁠MonadReader⁠⁠⁠⁠. Amazing. That's probably the best way to write code which uses db. Especially when we have lens. With other databases you'll need to invent something cool to work with them this way. Maybe ⁠⁠⁠⁠something cool⁠⁠⁠⁠ already exists (e. g. ⁠⁠⁠⁠persistent⁠⁠⁠⁠ for ⁠⁠⁠⁠SQL⁠⁠⁠⁠), but I have never used it.
• We know how to use it, moreover @neongreen (and partly I) knows how it works under the hood and made at least one patch. We have experience with it.
• Migrations are easy because of ⁠⁠⁠⁠SafeCopy⁠⁠⁠⁠. Note, however, that it's advantage of ⁠⁠⁠⁠SafeCopy⁠⁠⁠⁠, not ⁠⁠⁠⁠acid-state⁠⁠⁠⁠. We can use ⁠⁠⁠⁠SafeCopy⁠⁠⁠⁠ with other approaches.
• Very good guarantees about ACID properties, SQL databases usually provide slightly less strict guarantees, AFAIK.
• Probably easy rollback because of checkpoints. I am not sure.
• Code which uses it is already written 😄
Cons:
• last commit was in August, though some issues and PRs are pendings. Site is dead for a long time, nobody cares. That's one of the main reasons why I want to forget about ⁠⁠⁠⁠acid-state⁠⁠⁠⁠ for production. Alternatively we can concentrate on our fork and maintain it by ourselves.
• All state is in memory, nothing smart is done when you have a big map and want to get a single element. It's just loaded in memory and used as map. Well, we can store limited amount of data there and store blocks and old data in other database. Apart from blocks we probably have to store utxo on disk, don't we? In a long run I mean.
Actually, we need to understand which data should be persisted and which part of it can definitely fit in memory even in long run. If we see that we need to use another DB for big data, then does it make sense to use two different DBs? I don't know now.
• No support for reporting errors in ⁠⁠⁠⁠Update⁠⁠⁠⁠s. What should I do if whether update will succeed or not depends on data stored in state? Currently we are forced to use ⁠⁠⁠⁠throw⁠⁠⁠⁠ which sounds dirty. Well, it can also be fixed in our fork if we decide to maintain it.About SQL vs NoSQL.
I have never used SQL except university course, my personal feeling is that SQL is not for our purpose, but I am not sure. For example, bitcoin reference implementation doesn't use it, quick googling about ethereum told me that at least one implementation uses leveldb too.
I would rather use leveldb or rocksdb (the latter is more modern, but haskell bindings are less mature and probably we'll need to maintain them by ourselves).
Also I want to know about iodb, whether it's applicable in our case, what are cons and pros, etc. I only know that it exists and was developed bu Jan Kotek, but don't know anything about it.
Can you say how you're planning to do migrations with LevelDB? I don't have any experience with LevelDB so I don't know whether it'll be easy or hard, but it's better to have it figured out now than “oops, we used wrong LevelDB settings / integrated acid-state with LevelDB in an improper way / something else, and now migrations are goddamn hard and nobody knows how to do them”
I see at least two approaches:
• use ⁠⁠⁠⁠SafeCopy⁠⁠⁠⁠ serialization
• steal ideas from bitcoin (I don't know how they deal with it, but we can see) 😄
Why LevelDB and not e.g. postgresSQL?
I guess you ask it because you used postrgreSQL more often than leveldb. For me it's vice versa. So for me it's more reasonable to use leveldb. But it's also very subjective.Actually, this whole task is to do research and understand what to use. It's not only about implementation, it's also research.