moonstream-to / moonworm Goto Github PK

Currently, when setting paths during interface generation, there is no:

• checks for absolute and relative paths relative to current folder
• default values

Current implementation has the CLI generating its own handlers.

For me it is not comfortable for now to use generated cli, since you have to type:

python3 -m generated.erc20_cli ...

I think it will be good if we will generate executable bash scripts that looks like

erc20.bash:

python3 -m erc20.erc20_cli "$@"

• Event state classes do not all offer the same API.
• It is hard to distinguish between function crawl state and event crawl state. Maybe names should reflect this?
• The only state implementation for function crawling is MockState, and we are implementing our own DB writes outside of the state. We should make a proper FunctionCrawlState class that properly handles writing function crawl data to the database in a single transaction.
• Get rid of state providers and use lower level functions like _fetch_events_chunk.

Since there will be multiple crawlers, and some of them might use same transactions, it is better to store tx_reciept in journal as cache and call web3 only if it is missing.

See: https://github.com/bugout-dev/moonworm/blob/9bd62ee7e4d58ad9124edc1a3aae5e1b3a6834e6/moonworm/generators/brownie_contract.py.template

PROJECT_DIRECTORY is hard-coded. This makes it difficult for users who want to publish their libraries to PyPI to do so while retaining ease of autogeneration. They have to write a script which modifies this variable after autogeneration.

We can make this easier by replacing the PROJECT_DIRECTORY value with the path of the project directory relative to the output directory (as passed to moonworm generate-brownie).

Just found out that eval parses yuple correctly

Moonworm can also generate a REST API interface to any smart contract.

This would be incredibly useful functionality for many projects.

Following the model of the nft ethereum crawler on Moonstream, it should be possible to build a general-purpose interface + crawler for any smart contract given the smart contract source code/ABI.

Requirements

All functionality assumes connection to a web3 provider. The entire interface should be autogenerated and should not require any manual editing by a human programmer.

Interface

1. Get values for each external method on the smart contract.
2. Create transactions for each method on the smart contract which requires a transaction.
3. Allow users to sign transactions (using --interactive flag on command line and supplying their private key) and submit these transactions to the web3 provider.

Crawler

1. Set up a watch for any or all events emitted by a deployment of the contract.
2. Set up a watch for any or all contract method calls mined into the blockchain.
3. Scan blockchain for all deployments of a smart contract (given it's source code or deployment byte code).

Integration with Moonstream

We plan to use this tool in Moonstream to add support for additional smart contracts (beyond the nft support we currently have).

Need to make ability to verify smart contract code while deploying the contract

EthereumStateProvider

In function call crawler I am using abstract class for EthereumStateProvider to get blocks, transaction.

I've already implemented EthereumStateProviderW3Impl which uses web3. But for crawling in moonstream we can use mix of our db and web3 for better performance. Since we are crawling all transactions, our db is enough, however if we face missing block from our db we need to get data from web3

FunctionCallCrawlerState

This is abstract class that implements saving/getting data from db. Need to write MoonstreamState for this crawler. Which will save data to labels

Now, crawlers are tested against mainnet. Need to use ganache, deploy some mock contract with events. And test crawlers against it

Now it is required to give --bytecode to deploy, even though bytecode could be found in the same directory with <name>_bytecode.bin prefix

You can currently specify a URL as the web3 provider. However, you can't use IPC if you happen to be running your own node.

I think this could be fixed by changing line 119 of cli.py from:

    web3 = Web3(Web3.HTTPProvider(args.web3))

to

    # Assume "args.web3" is an URL; if no connection,
    # see if it's an IPC file.
    web3 = Web3(Web3.HTTPProvider(args.web3))
    if not(web3.isConnected()):
        web3 = Web3(Web3.IPCProvider(args.web3))

I don't have permission to submit a PR or I'd do that myself :)

If ABI contains an item with no "name" field, moonworm throws an error.

Found in contract https://etherscan.io/address/0x13d15d8b7b2bf48cbaf144c5c50e67b6b635b5cd#code

moonworm generate-brownie codegen interfaces have many more users (and have seen many more improvements) than the moonworm generate codegen interfaces.

We should modify the code generated by moonworm generate to produce interfaces that are identical to the moonworm generete-brownie-generated code:

1. Transaction parameters
2. --sender and --password handling from CLI
3. CLI and class should be in the same file

The generated get_transaction_config function has no rule for args.value: https://github.com/bugout-dev/moonworm/blob/main/moonworm/generators/brownie.py#L235

Although the generated argument parser does have a place for it: https://github.com/bugout-dev/moonworm/blob/main/moonworm/generators/brownie.py#L664

https://github.com/bugout-dev/moonworm/blob/main/README.md

Brownie contracts expose a publish_source method that we can use for this purpose: https://eth-brownie.readthedocs.io/en/stable/deploy.html?highlight=verify#verifying-deployment-source-code

Users will have to set environment variables for the blockchain explorer of their choice: https://github.com/eth-brownie/brownie/blob/0fa4477a178bd55b6683f60d077b7060df02b2c5/brownie/network/contract.py#L60

Need to provide ability to get version of moonworm from cli

We like three features in brownie:

1. brownie compile
2. brownie networks
3. brownie accounts

We should provide users with an option to use these interfaces in their generated code (especially in CLI).

Even though --verboseis not set, generated code will try to print detailed tx.info()

Now there are tests only for functionality of methods.
It is becoming hard to test if generated cli is working how it is supposed to

--dry-run would simulate the transaction at the current block and return any errors in its simulation, but not submit the transaction if it looked like it would be successful.

• Find code generating library for js, candidates:
  1. babel
  2. hygen

• Explore Truffle/hardhad/Web3. Check out how projects are structured, how to deploy/interact with smart contract

• Make template for static functions, that don't needed to be programmatically generated

• Code generation

https://discord.com/channels/867156607644860418/867156608118947864/1021107016502091856

This will make it easier to work with contracts which fire events with side-effect specifiers (e.g. Terminus Pool IDs).

Currently, moonworm watch functionality only allows us to do this by block. This issue is about being able to inspect events by transaction.

An example error we saw in Moonstream Engine API:

Before:

            message_hash = dropper_contract.claimMessageHash(
                claimant_drop.claim_id,
                claimant_drop.address,
                claimant_drop.claim_block_deadline,
                transformed_amount,
            ).call()

Error:

web3.exceptions.ValidationError:                                                                                                                                                    
Could not identify the intended function with name `claimMessageHash`, positional argument(s) of type `(<class 'int'>, <class 'str'>, <class 'int'>, <class 'str'>)` and keyword argument(s) of type `{}`.                                                                                                                                                              
Found 1 function(s) with the name `claimMessageHash`: ['claimMessageHash(uint256,address,uint256,uint256)']                                                                         
Function invocation failed due to no matching argument types.

After:

            message_hash = dropper_contract.claimMessageHash(
                claimant_drop.claim_id,
                claimant_drop.address,
                claimant_drop.claim_block_deadline,
                int(transformed_amount),
            ).call()

This is not an issue with the moonworm generate-brownie interfaces because I believe brownie does the type coercion before making the web3 call.

It would be useful if we did this ourselves in the generated classes for moonworm generate.

When I run moonworm -h (globally installed) , I keep getting this warning message :

INFO:blib2to3.pgen2.driver:Generating grammar tables from /home/yhtiyar/.local/lib/python3.9/site-packages/blib2to3/Grammar.txt
INFO:blib2to3.pgen2.driver:Writing grammar tables to /home/yhtiyar/.cache/black/21.12b0/Grammar3.9.7.final.0.pickle
INFO:blib2to3.pgen2.driver:Writing failed: [Errno 2] No such file or directory: '/home/yhtiyar/.cache/black/21.12b0/tmpr2fildlt'
INFO:blib2to3.pgen2.driver:Generating grammar tables from /home/yhtiyar/.local/lib/python3.9/site-packages/blib2to3/PatternGrammar.txt
INFO:blib2to3.pgen2.driver:Writing grammar tables to /home/yhtiyar/.cache/black/21.12b0/PatternGrammar3.9.7.final.0.pickle
INFO:blib2to3.pgen2.driver:Writing failed: [Errno 2] No such file or directory: '/home/yhtiyar/.cache/black/21.12b0/tmpa7s5bs4u'
usage: moonworm [-h] {watch,watch-cu,generate-brownie,generate} ...

Or to Bugout journal.

• Python linting (black, isort, etc.)
• Push to PyPI when release is created
• Locust

Like in moonworm generate-brownie it would be better to convert to snake case.

For example, it fails to crawl the first even from this transaction: https://polygonscan.com/tx/0x361aba278b01e23614ef2b1a87e725aeab7e131f55a4d5002646f4ba0023dfa7#eventlog

Using this ABI:

[
  {
    "anonymous": false,
    "inputs": [
      {
        "components": [
          {
            "internalType": "address",
            "name": "facetAddress",
            "type": "address"
          },
          {
            "internalType": "enum IDiamondCut.FacetCutAction",
            "name": "action",
            "type": "uint8"
          },
          {
            "internalType": "bytes4[]",
            "name": "functionSelectors",
            "type": "bytes4[]"
          }
        ],
        "indexed": false,
        "internalType": "struct IDiamondCut.FacetCut[]",
        "name": "_diamondCut",
        "type": "tuple[]"
      },
      {
        "indexed": false,
        "internalType": "address",
        "name": "_init",
        "type": "address"
      },
      {
        "indexed": false,
        "internalType": "bytes",
        "name": "_calldata",
        "type": "bytes"
      }
    ],
    "name": "DiamondCut",
    "type": "event"
  },
  {
    "inputs": [
      {
        "components": [
          {
            "internalType": "address",
            "name": "facetAddress",
            "type": "address"
          },
          {
            "internalType": "enum IDiamondCut.FacetCutAction",
            "name": "action",
            "type": "uint8"
          },
          {
            "internalType": "bytes4[]",
            "name": "functionSelectors",
            "type": "bytes4[]"
          }
        ],
        "internalType": "struct IDiamondCut.FacetCut[]",
        "name": "_diamondCut",
        "type": "tuple[]"
      },
      {
        "internalType": "address",
        "name": "_init",
        "type": "address"
      },
      {
        "internalType": "bytes",
        "name": "_calldata",
        "type": "bytes"
      }
    ],
    "name": "diamondCut",
    "outputs": [],
    "stateMutability": "nonpayable",
    "type": "function"
  }
]

It does correctly call the method invocation.

Working example: https://github.com/bugout-dev/inspector-facet/blob/4e6682fe7c2111944fe8ebdb7ff1dc0456d48c9f/inspector_facet/DiamondLoupeFacet.py#L15

Needed to clean up crawlers code

Example with block_number as a view method argument:

    def get_block_hash(
        self, block_number: int, block_number: Optional[Union[str, int]] = "latest"
    ) -> Any:
        self.assert_contract_is_instantiated()
        return self.contract.getBlockHash.call(
            block_number, block_identifier=block_number
        )

• #56
• Only commands corresponding to payable methods

For any contract method which is not deploy.

This has led to a lot of pain and a lot of wasted time.

Reported by GarrettWorlds while using game7ctl:

I was trying to get the tool to work earlier and I was able to stumble across the solution. Turns out I wasnt using the --address argument when using the inventory subcommand. My mistake. Normally it says the following arguments are required: ... but it said contract has not been instantiated when I didnt give it an address. Not sure if this is something you guys would fix but I thought I would let you know.

running
python3 -m setup.py install

Is causing following error:

Processing dependencies for moonworm==0.0.4
error: typing-extensions 4.0.0 is installed but typing-extensions<4,>=3.7.4 is required by {'trie'}

Current output is not so useful, so instead of doing print(result) in generated code, we should instead do result.info().

It will remove build folder dependency and the files will become shippable

https://github.com/bugout-dev/moonworm/blob/b1693ecdeda7d598de7e4f2e0fa141e27971a306/moonworm/watch.py#L66-L96