Currently all methods on the Client and Store are implemented in a single module in a single impl block. This will quickly become messy as the number of methods increases. To improve modularity and code hygiene we should split the impl up into logically isolated components and have a module with an impl with the methods relevant to the module. We can follow a similar module structure to the database design - account, input notes, output notes, state sync, transactions etc.

Currently there are some unwrap's in the parsing logic of the API response. It is not safe to assume that Option types are populated and as such we should handle this with robust error handling:

After a transaction gets executed, correctly proven and submitted, the following needs to happen:

• Account gets applied the AccountDelta and its data changes (vault, storage and nonce). This needs to be stored in the DB.
• Created notes need to be stored in the input_notes table
• Transaction data needs to be stored in the transaction table for tracking its lifecycle

This is currently done (for now, on branch igamigo-complete-sync), but everything should happen within a single SQL transaction which is not the case right now.

The resulting query will yield a i64 value that needs to be casted as u64, we are currently trying to map the i64 directly to u64, causing a parsing error.

As part of the account creation process we insert data to various different tables sequentially. If one of the inserts fails the state will be incorrect, so the inserts/updates should be done in an atomic manner, through an atomic transaction.

• Clone miden-node and checkout igamigo-export-seed
• On miden-node's path, create genesis file: cargo run --bin miden-node --features testing -- make-genesis and run the node: cargo run --bin miden-node --features testing -- start
• In the client's dir, checkout igamigo-data-store and load the genesis file data: cargo run -- load-genesis --genesis-path {path where you created the genesis files}
• Perform sync state: cargo run -- sync-state -s
• In the client's dir, run a test mint transaction cargo run -- transaction new mint 0x168187d729b32a84 10 0x871e602b3953f7cb 100

You should get the following error: transaction executor error: ExecuteTransactionProgramFailed(AdviceMapKeyNotFound([6280925791586927585, 15735186907095191003, 4209276716593169044, 12827351902656088934]))

As our Wallet PoC will only support P2ID transaction, so we need to have and endpoint where we can pass the faucetId, amount and destination address and the return would be the note object that will be created which can be passed to the transaction input.

Some initial thoughts about Miden Client CLI interface were listed in 0xPolygonMiden/miden-base#122 (comment). I think the interface we'll end up with won't be exactly the same, but some general themes from that comment still apply. Specifically, I'm still thinking we'd do it as a non-interactive CLI (this is how I started implementing this in #10).

Instead of specifying the commands exactly, I'll describe general functionality and we'll hone in the exact command semantics later. I'm sure I'll probably miss some aspects on the initial pass - so, any feedback is appreciated.

Accounts

The CLI should provide the ability to explore existing accounts managed by the client as well as create new accounts. For now, I'm limiting this only to off-chain accounts. Here is what this could look like:

1. A command to list all managed accounts with some basic info about each account. This info can include account ID, latest nonce, latest state hash, status of the account (i.e., committed or pending).
   a. In the future, this can be extended to include other info - e.g., last update timestamp or block number, total value of assets etc.
2. A command to view details of a particular account - e.g., list of all assets in the account, contents of the storage, account code etc.
   a. We can also have a command to view details of a particular account "snapshot" (i.e., historical state).
3. A command to view history of an account - i.e., a list of account's historical states.
4. A command to create a new account. This command would create an account locally. A transaction would need to be executed to record this account on chain.
   a. There could be multiple versions of this command. For example, we could have a fully generic version where the command would take references to .masm code file and initial storage file and then create an account from these. Another variant could be to use pre-defined templates (e.g., create an account for a basic wallet). I'm thinking that initially we should add support for the latter, and implement the fully generic version later on.
5. A command to delete an account - this would remove the account (and all related objects) from the client. I think we can skip this for the initial implementation.
6. Commands to import/export accounts - these can also be delayed for after the initial implementation.

Notes

The CLI should provide the ability to explore notes associated with the accounts managed by the client. I'm thinking that notes should be separated into two groups: "inputs" and "outputs" (open to other terminology suggestions). Input notes would be the ones which can be (or have been) consumed by any of the managed accounts, while output notes are the ones which resulted from executing transactions against any of the managed accounts.

Input notes

Input notes could be in two states: "consumed" or "pending". For input notes the client would know all the details (i.e., note vault, note script, inputs, serial number etc.). The CLI would expose the following commands related to input notes:

1. A command to list all input notes with some basic info about each note. This info could include note hash, note status (e.g., consumed or pending), for consumed notes, account ID for the account which consumed this note, and for pending notes a list of account IDs for accounts which can consume this note.
   a. We should also include some info about note assets and note script - though, not sure how to summarize this info a list. So, maybe something we'd do after the initial implementation.
2. A command to view details of a specific note - this would include full note script, note vault, inputs etc.

There is another group of input notes which we could call "expected" - i.e., notes which haven't appeared on-chain yet, but we are expecting them to appear in the future. I think we can skip these for now and add them after the initial implementation.

Output notes

Output notes could be in two states as well: "pending" (created but not yet included in any block) and "committed" (included in some block). For these notes, the client may or may not know all the underlying details. For the initial implementation, we won't track full note details even if we know them - this can be changed in the future. The CLI would expose the following commands related to output notes:

1. A command to list all output notes with some basic info about each note. This info could include note hash, note status (e.g., pending or committed), account ID for the account which created this note.
   a. We should also include some info about note assets and note script - though, not sure how to summarize this info a list. So, maybe something we'd do after the initial implementation.
2. A command to view details of a specific note - this would include full note script, note vault, inputs etc.

Transactions

The CLI should provide the ability to create new transactions and explore already executed transactions. Transactions can be in two states: "pending" (executed and accepted by a Miden node) and "committed" (included in a block). When a transaction is marked "committed" a related account state and notes are marked committed as well.

The CLI would expose the following commands related to transactions:

1. A command to list all transactions created by the client with some basic info about each transaction. This info would include the account against this transaction was executed, list of input and output notes, current status etc.
2. A command to view details of a transaction. This could also include the effect of the transaction on the account (e.g., which assets were added/removed, how storage changed).
3. A command to create a new transactions. In the initial version of the CLI this would create, execute, prove, and submit the transaction to a Miden node. In the future, we'll add the ability to skip local proving.
   a. There are many options of how we can implement this command (and this may require a separate discussion). For the initial implementation, I think we'd support something very simple where the user can execute a transaction which consumes an unconsumed input note, or a transaction which executes a tx script to create a P2ID note.

Syncing

The CLI will need to provide a command to retrieve the latest data from a Miden node. This command would do the following:

• Retrieve all relevant block headers up to the current chain tip.
• Retrieve info for all potentially relevant input notes which have been created since the last sync time (this would be done using note tag).
• Retrieve nullifier info for all committed output notes which haven't yet been consumed.
• Retrieve the latest state commitments for all managed accounts.
  • These can also be used to check pending transactions have been committed - i.e., if account state is updated from x to y we know that a transaction which updates account from x to y has been executed.

Regarding last point above: one open question is how to get transaction confirmations. I think using account state transitions as a proxy for transaction execution may be brittle - so, we may want to do something more explicit here (e.g., get info on specific transactions).

Other commands

The probably could be a bunch of other commands (e.g., commands to show summary view of assets across all accounts), but I think these can be delayed beyond the initial implementation.

Accounts in Miden

Accounts are the core concepts in Polygon Miden. Accounts represent users and autonomous smart contracts. In Pioneer's testnet, we will only have private accounts. The Miden Node stores only a commitment to an account state, see https://0xpolygonmiden.github.io/miden-base/architecture/accounts.html.

User Story / Goals

As a developer, I can create a Miden account. There is an interface I can use for that. The account already has some pre-configured functions to send and receive assets.

After account creation, I can inspect this account and see the ID, storage, nonce, vault, and code.

Details

There should already be a simple endpoint and Miden Client interface from #1. Now, we need another feature for the Miden Client. The user needs to be able to grind valid account IDs, the format is well-defined.

https://github.com/0xPolygonMiden/miden-base/blob/dc9c013b528f17371cacc507e4e812d75545a6fb/objects/src/accounts/account_id.rs#L16-L28

The interface must allow the user to inspect the account contents. The code should have the basic functionality in MASM.

We also need comprehensive error messages in error cases.

Note: This tasklist and the issues are preliminary. The developer who picks this tracking issue up, should plan accordingly and create the necessary issues

Right now we are printing outputs to stdout, but we should look into having more complete and comprehensive logger (in line with the other projects)

As a temporary solution we are storing the account key pair in the database unencrypted, but we need to decide on a more flexible and future-proof approach as @bobbinth suggested. The main use-case we need to support now is to be able to provide authentication easily (but we will also probably want a way to import/export accounts in the future).

endpoint is required to get token metadata like Name, symbol, Decimal which is necessary to show in wallet extention.

with this ENDPOINT, we will be able to show all these details for all the assets user has in his account.

Just an example for the endpoint structure

requestParams: {faucetId}
response: {faucetId, name, symbol, decimals}
it can be 3 different endpoints also giving name, symbol, decimal separately. and decimals is optional as there are no decimals for non fungible faucets

Wallet will only have a miden client instance which will be lost whenever user turns of his device or even close browser. The question here is how miden client is handling the account state? is it creating a temp file in user local which has the account state and refer to that file once user will come back or do we have to handle that for miden to provide the latest state everytime?

Add a readme to the repo with some basic information about the client:

• Creating accounts
• Importing/exporting notes
• Submitting transactions to a local node

When executing transactions, we need to retrieve multiple block headers from the store. Currently this is done iteratively (multiple queries to retrieve 1 block each time) but should be done in a single query to which we can pass a list of block numbers.

we think that this is the object for transaction https://github.com/0xPolygonMiden/miden-base/blob/0dc0f0345ed086bbdd007ba54c9edd7f71acf9bc/objects/src/transaction/executed_tx.rs#L7

we need an ENDPOINT to get this object created. this object can be created in local too but for that we need to find out to get all the different params that is required to create this object.

this object is needed so that we can send this transaction object in the transaction execution endpoint where user will be able to sign this transaction and then send it to the network

There are a few cli commands that leverage the .store() accessor on the Client. I think we should remove the store accessor and instead all cli command should go via public methods of the Client.

Here are two instances of where this is done.

The command miden-client sync-state add-tag doesn't check for duplicate tags:

We are currently not storing the init_seed account_seed that we get after generating accounts in the Accounts table. We need to store it so that we can use it for transaction execution when required.

We need to persist the init_seed that we use to create an account on the client. It will be needed in the first transaction which involves the account (this transaction will actually create the account in the blockchain).

Right now we are importing and exporting notes with client note import {note_hash_file} and client note export {note_hash} respectively. An alternative could be to have top-level commands for this (client import {note_hash_file}). This might make sense if we implement importing/exporting accounts or other entities.

We are using the rand crate to generate an initial seed randomly, but it's not no_std compatible out of the box, so we need to look for a way to change this.

This is an open end discussion on how we are going to create falcon key.

1. miden client can expose functions which can create random falcon key which we can then save in users's local but along with that, other endpoints will also need to be exposed like signing a message, verifying it.

2. Or we can use already created rust library https://crates.io/crates/pqcrypto-falcon/0.2.10 which already has functions exposed. we need to make sure that miden is using the same inside node for all the verifications and other things

Current P2ID implementation uses mock objects, so we need to change this after #71 is merged and we have the proper DataStore available

As part of #95, we added an account_indices parameter to the CLI which would allow a subset of account file indices to be imported to the client. The original intent was to be able to simulate different clients importing only a subset of accounts in order to test different interactions with the node. As it is this might not be very useful/beneficial to maintain, so we should decide whether we want to remove or modify this feature.

An option would be (as suggested by Bobbin in one comment in the PR) to import via filenames if we want to keep this ability. Another alternative is to remove it altogether and simulate it by making separate directories by hand when we want to split accounts.

Currently, all notes created by a transaction are inserted into the input_notes table. The table has an inclusion_proof that gets updated after a state sync, effectively transforming the Note into an InputNote. This can be a little confusing and ambiguous, and we should discuss changes and other approaches.

We should likely add an output_notes table. Right now the TransactionExecutor returns the OutputNotes object, but this does not have all the details that we want to store (such as the serial number) - these are known to the client as creator of the notes. As such, essentially the output_notes table would just contain Note structs, and input_tables would be InputNote objects.

In relation to this comment, we need to what the best way to store felts is.

We need to update the client to work with the updates made in 0xPolygonMiden/miden-base#414 and 0xPolygonMiden/miden-node#170.

Currently we use different patterns for querying data from the database.

Multi row query

Single row query:

Insert row:

We should decide on a standardised pattern to use for each and refactor the codebase to use the standard.

We need to consider how to test the miden client. This includes both integration testing with the miden node and unit testing. I would propose that initially we use mock data from the mock crate in miden-objects to generate test data. We can use this as a basis for our unit tests. Regarding integration testing with the miden-nodes I would once again suggest that we first mock the rpc requests using data from the mock crate. We can then look to build real integration tests with a framework like docker.

Right now, the importing and exporting of notes is implemented exclusively for RecordedNote (#67), but we will want and implementation for Note as well.

As discussed, we want to be able to send an asset from Alice to Bob using our remote Node on Wednesday.

This is just a placeholder issue because I don't know what is missing to do that.

Currently, every transaction stores its own script-related information. Instead, the transaction table should reference another table that has this information in order to avoid storing it every time.

Breaking changes in the miden-base prevents compilation of the miden-client.

A few thoughts about how the Client struct could look like. These are just proposals at this point and we don't need to implement everything here right away. If we agree on this general approach, we can move the Client struct close to this over time.

Overall, the main goal is to move more logic into the Client struct (or related structs) and make the CLI part relatively "dumb". The benefit of this is that this Client struct could be used by other "front-ends" without requiring this front end to implement lots of redundant logic.

Another goal is to make the client "modular" - that is, have several internal components in the client. For now, these components can be just structs, but over time, we can make them into generic types.

For example, the client struct could look something like this (just a sketch, probably missing something):

pub struct Client {
    /// Local database containing information about the accounts managed by this client.
    store: Store,
    /// Connection to Miden Node.
    node: Node,
    /// Component responsible for executing transactions.
    tx_executor: TransactionExecutor,
    /// Component responsible for proving transactions.
    tx_prover: TransactionProver,
    /// Component responsible for authenticating transactions (i.e., generating signatures).
    authenticator: Authenticator,
    /// Component responsible for generating new keys, serial numbers, etc.
    rng: Rng,
}

And in the future, this could become something like:

pub struct Client<S: Store, N: Node, A: Authenticator, R: Rng> {
    store: S,
    node: N,
    tx_executor: TransactionExecutor,
    tx_prover: TransactionProver,
    authenticator: A,
    rng: R,
}

In terms of API, the client could look something like this:

impl Client {
    // ACCOUNT CREATION
    // --------------------------------------------------------------------------------------------

    /// Replaces current `insert_account` but contains the logic for processing account templates.
    /// 
    /// [AccountTemplate] could include a variant which passes an already constructed account to
    /// allow users to build accounts externally. Alternatively, we could have a separate method
    /// for this - something like `new_account_raw()`.
    pub fn new_account(&mut self, template: AccountTemplate) -> Result<AccountId, ClientError> {
        todo!()
    }

    // DATA RETRIEVAL
    // --------------------------------------------------------------------------------------------

    // client metadata retrieval methods
    // account retrieval methods
    // note retrieval methods
    // transaction retrieval methods

    // STATE SYNCHRONIZATION
    // --------------------------------------------------------------------------------------------

    /// Retrieves the latest state from the chain and returns the most recent block number (i.e.,
    /// the block number to which we are currently synchronized).
    pub fn sync_state(&mut self) -> Result<u32, ClientError> {
        todo!()
    }

    /// Registers a note communicated via a side channel with this client.
    /// 
    /// This is needed so that when the client gets a note hash from the chain, it can figure out
    /// the details of the actual note.
    pub fn register_note(&mut self, note: Note) -> Result<(), Client> {
        todo!()
    }

    // TRANSACTION
    // --------------------------------------------------------------------------------------------

    /// Creates and executes a transactions specified by the template, but does not change the 
    /// local database. The template variants could be something like:
    /// - PayToId - to create a P2ID note directed to a specific account.
    /// - PayToIdWithRecall - same as above but with P2IDR.
    /// - ConsumeNotes - to consume all outstanding notes for some account.
    /// - Custom/Raw - to specify transaction details manually.
    /// 
    /// The returned [Transaction] object would probably be similar (if not identical) to our current
    /// [TransactionResult].
    pub fn new_transaction(&self, template: TransactionTemplate) -> Result<Transaction, ClientError> {
        todo!()
    }

    /// Proves the specified transaction, submits it to the node, and saves the transaction into
    /// the local database.
    pub fn send_transaction(&mut self, transaction: Transaction) -> Result<(), ClientError> {
        todo!()
    }
}

The main idea is that the methods listed above are the only way to modify the client state (i.e., we don't update account states or insert notes directly).

And again, this is just a general sketch - so, any feedback is welcome.

Should we consider the introduction of an ORM with native support for migrations to manage our data model. Some options include:

https://github.com/diesel-rs/diesel
https://github.com/SeaQL/sea-orm

Currently the client will attempt to connect to the node as it is created, but there is no need if something is attempted locally (reach for objects in the store)

After 0xPolygonMiden/miden-node#138 is merged, we want to update the load-genesis command to utilize the newer format. Furthermore, we might want to expose a client method (or multiple) for this as well.

The client is currently using the mock_inputs on some tests in the following manner:

    // generate test data
    let (account, _, _, recorded_notes) = mock_inputs(
        MockAccountType::StandardExisting,
        AssetPreservationStatus::Preserved,
    );

Latest miden-base commit changed the function and now returns 5 parameters:
(Account, BlockHeader, ChainMmr, Vec<RecordedNote>, AdviceInputs)

Likewise, the client is using an old version of the mock_new_account, that takes a single argument, Assembler, since the latest miden-base commit, it now also requires AdviceInputs. This type is from miden-processor, so it needs to be added as a dev-dependency.