moonstream-to / api Goto Github PK

For the launch, in the interests of getting things up and running as soon as possible, we will store our blockchain crawl data on a Postgres instance we set up on Google Cloud Platform.

Pricing for AWS RDS and GCP Cloud SQL instances will come it at more than $400 per month with 4 TB of storage. Since this is quite expensive, we will set up and manage the database on our own Compute Engine instance. We do not need high availability here (for now).

The expected cost of the GCE instance is ~$200 per month. We will use an API instance powered by this database to benchmark alternative storage solutions.

The very first alternative we will test (after Moonstream launch) is to use S3 as the storage layer instead of a live Postgres database. This is something Quickwit has been doing to implement an Elasticsearch-like search engine with S3 as the storage mechanism.

We will see if we can get reasonable API performance using Activeloop's Hub to store and retrieve data from S3.

I will create a separate issue for implementing and benchmarking a Hub-powered API against a Postgres-powered API.

Add possibility to set HTTPProvider for web3_client connection.

This token will be used, for example, to read Brood resources that are exposed over unauthenticated Moonstream API endpoints (e.g. /subscriptions/types).

We would like to have ability to define frontend view (layout) where background is static (does not moves when user scrolls). That requires changes so that background is defined in element that scrolls also the footer.

Possible solutions

• Make this a default view and test all views that use footer
• Make this as variant of layout and use it when needed

For long-running queries into a stream, we should implement an asynchronous workflow (e.g. POST /streams/async) which generates the results of a query and makes them available to users using S3 signed URLs.

This is analogous to how we handle statistics on the Bugout analytics screens.

Transaction hashes

Would be nice to see transaction hashes on the cards in the stream. These are quite commonly used to verify that you're looking at the right transaction (across sites, from Javascript console, etc.). It should be prominent.

(Currently, only shows up in transaction details.)

Colors

Cards feel a little too colorful. Would prefer that colors are used to display information about subscriptions only. Rest of the card can be white or dark depending on theme.

Examples to consider:

1. @kompotkot : Suggested this site - https://dribbble.com/search/card%20component%20list
2. @Andrei-Dolgolev : liked this for minimalism - https://dribbble.com/shots/15221832/attachments/6968098?mode=media
3. @zomglings : Likes the GitHub repository cards that show up in Slack -
   

Transaction details should not take over full view

Would be nice to be able to switch between transactions when looking at details. This could be similar to entries list vs. entries on Bugout frontend.

@kompotkot can fill in more details if this ever becomes confusing.

Currently, it's hard to navigate the site using the back button because every page has the title My Subscriptions (even the stream page).

Harriatte needs to send icons for these pages.

@SophiaAr and Daria to provide content.

@peersky to add them to frontend code.

https://monstream.to/register should take users to the landing page with an open registration modal.

Similarly, https://monstream.to/login, etc.

We don't have these for Alpha, but need to add them in Beta.

We need to fill up following metatags:

Metatags

Landing page

title: ""
description: ""
keywords:  ""
url: ""
image: ""

https://bugout.dev/app/personal/31bca7c2-9126-4481-8d56-eefbf48dd8e0/entries/8fcaf7bf-3ac0-4e44-9e76-99b84d0b9c16/

For streams, it makes sense to paginate based on time rather than based on a cursor, IDs, etc.

Pagination requests can have the following form:

{
    "start_time": "1628093198",
    "end_time": "1628093524",
    "include_start": true,
    "include_end": null,
    "limit": 10
}

Pulls at most the first 10 entries in the stream from the backend starting at the "start_time" and ending at the "end_time".

Response format:

{
    "data": "[<at most 10 entries>]",
    "next_start_time": "null or <if there is an 11th entry, timestamp for 11th entry>"
}

We want to see following dashboards on the mixpanel:

• Page views by page
• How long user spends time on each page
• Which buttons user clicks on the landing page and navbar/footer
• What is session duration
• If user gets any errors
• What is user speciality
• Did they complete onboarding successfully
• How far did they get in the landing page before they bounce
• Which page in the app users trends to close session
• Which page not in the app user trends to close sssion
• How long user spends on each onboarding screen

A user can subscribe to data from multiple sources. For example:

1. Uniswap data from the Ethereum blockchain
2. Uniswap data from the Ethereum transaction pool
3. Metaplex NFT data from the Solana blockchain

Multiple users may subscribe to the same source - e.g. Uniswap is a very popular contract at the moment and many different users may want to have Blockchain or Transaction Pool subscriptions to Uniswap events.

Our crawlers do not care how many users have subscribed to a particular source. All they need to know is which sources they need to get information about from various blockchains or mempools/transaction pools.

Once a crawler extracts information about a source, it adds it to a central data store, and our API chooses the data from that central repository to display to the user, based on their subscriptions (and when those subscriptions were activated).

Proposed CLI structure:

mooncrawl
    - ethereum
        - sync
            - realtime (confirmed transactions, txpool)
            - add (blocks, transactions)
            - missing (blocks, transactions)
        - esd
        - identity
    - solana
        - ...

When a user has many tabs open, it is difficult for them to find the Moonstream tabs because the favicon is too low key.

@Andrei-Dolgolev : Coordinage with @peersky to understand what endpoints he needs on the backend.

Coordinate with @kompotkot to understand how to read data from blockchain database.

Coordinate with @Yhtiyar to understand how to expose smart contract decompilation metadata to @peersky on the frontend.

User should be able to Update subscription on

POST /subscription/<subscription_id> :

• Edit color
• Edit name

This query (runs over a ~ 21 hour period):

time curl -H "Authorization: Bearer cd967dc4-db0a-4199-8e67-7b3ecc68942d" "localhost:7481/streams/start_time=1629283337&end_time=1629359066"

takes between 1.5 and 4 minutes to return data.

There are three ways for us to handle this from a UX perspective:

1. More prominent and informative loading bar which even says that it can take up to 5 minutes to fetch the results for this query.
2. Just prompt the user for confirmation if they are submitting a query over a long time period.
3. Split up a query with a long stream boundary into multiple queries with short stream boundaries. (best solution, because then we can also display a progress bar)

It could be because the transaction cards are too colorful, but users have missed the green arrow element on the frontend completely during onboarding. Maybe we should make it more obvious.

When someone deploys a smart contract to the blockchain, the "to" field on the transaction is null.

The way to resolve the smart contract address is to read the "contractAddress" field from the transaction receipt (eth.getTransactionReceipt).

We need to create a separate worker which scans the transactions table for transactions with empty "to_address" columns and processes the receipts for those transactions.

Initial version of Moonstock frontend.

UI will have 7 screens:

1. Landing page (In progress)
2. Register (API ready, make hookup)
3. Login (API ready, make hookup)
4. Subscriptions - where users can add subscriptions to addresses on supported blockchains, pay for additional subscriptions, and remove subscriptions (Ready, PR)
5. Billing - user can specify what billing action they want to take (adding subscriptions, removing subscriptions, etc.) (Not in alpha)
6. Stream - live stream of events that user is subscribed to
7. Statistics - Give user an aggregate view of activity for their subscriptions (e.g. most active subscriptions, total trading volume per subscription, etc.) (Under construction in alpha)

These should be used to configure the Bugout client in settings.py.

Features:

• Moonstream discord (@SophiaAr)
• #1
• Landing page content for https://moonstream.to (@SophiaAr )
• #7
• #12
• Authentication, user management (@kompotkot )
• Demo streams - Ethereum blockchain stream of transactions from popular defi smart contracts: Uniswap, Compound, Cryptokitties, etc. (@kompotkot )
• Ability to set watches on any Ethereum blockchain data (ethereum_blockchain streams) for free. This matches Etherscan watchlist functionality. (@kompotkot for API, @peersky for frontend)
• Button to upgrade from blockchain demo stream to transaction pool events for the each blockchain subscription - this should initially just take them to a waitlist form. (@peersky )
• Streams should show raw data from susbscribed Ethereum blockchain (ethereum_blockchain streams). (@Andrei-Dolgolev )
• Contextual information about Ethereum smart contract transactions. (For such transactions, our crawlers can add a program tag to the entries, which will be a signal to the frontend that it should present context about smart contract calls to the user.) #10 (@Yhtiyar )

Post-alpha features:

• Subscriptions - Users should be able to create ethereum_blockchain and ethereum_txpool subscriptions.
• Subscriptions - Users should be able to subscribe to streams using stripe. (we will give a 30% discount for alpha users)
• Shortcuts to add subscriptions to transaction pool data for particular Ethereum contracts/ecosystems.
• Raw data from Ethereum transaction pool (ethereum_txpool).
• Ability to give users free subscriptions (to ethereum_txpool streams) from a CLI. This will be useful for onboarding alpha customers.
• Group subscriptions - users should be able to create teams and the whole team will see the same event stream.

Currently, if the node has to roll back accepted blocks because it got out of sync with the majority of its neighbors, the synchronization process crashes and requires manual reconciliation.

We will put two measures in place to solve this problem:

"--confirmations" argument on "ethcrawler synchronize"

Default value should be 0 indicating that the synchronization should be with the latest block.

--confirmations n for n > 0 indicates that synchronization should always be n blocks behind the latest block on the node.

This is the quick and dirty fix.

Check parent hash before storing block

If parent hash doesn't match block hash for last block stored in database, step backwards on node and in database till hashes match up. Then delete all subsequent blocks from database and start a resynchronization from that point onwards.

This is the principled fix.

Getting function call information of smart contracts is hard task, however for now we can only show basic information about contract such as:

• Name of contract (If it is known for us)
• Link to source code (If it is known for us)
• ABI of smart contract
• Estimated gas

Probably I will use python to easily integrate to API.
Ready solutions to use:

• https://github.com/beched/abi-decompiler - Does the abi decompilation, but it is written in python2, probably I need to rewrite it in python3.
• https://github.com/MrLuit/evm - this is easy to use, also decodes events, but it is in JS
• https://github.com/palkeo/panoramix - very powerful with great decoder, ,but it requires to run in eth node (I think it gets data from storage). example

UPD:
I crawled function and event signatures from https://www.4byte.directory/. Now we can decode ABI of smart contract bytecode

Now if it fails, it will fail silently

Work already started: #93

Crawler should be written in Go (so that we can deploy it to different devices).

Define data structures and interface on this issue @kompotkot .

One of the biggest problems with the subscription model is that, if the user does not know addresses to subscribe to, they have an empty stream.

A really good idea that one of our early users had was to show addresses that were experiencing very high transaction volume within the last x hours.

This could be its own subscription - ethereum_trending. Our subscription model might have to change slightly, as this subscription would not take an address, but it would have a color and a label.

This is a fantastic idea and is a good way to implement an always on default stream when a user signs up. High priority to add this.

EIP 1559

The Ethereum blockchain caller should be callable from the command line.

Crawler interface

Transaction blocks

Add blocks (optionally adding transactions at the same time)

ethcrawler blocks add \
    --blocks <list of block numbers or block range in the format {start_block}-{end_block}> \
    --db <database connection string> # This is probably better as environment variable - MOONSTREAM_DB_URI \
    [--transactions]

Latest block to have been added

ethcrawler blocks latest \
    --db <database connection string> # This is probably better as environment variable - MOONSTREAM_DB_URI

Check if there are any missing blocks

ethcrawler blocks missing \
    --db <database connection string> # This is probably better as environment variable - MOONSTREAM_DB_URI

Transactions

Add transactions

ethcrawler transactions add \
    --block <block number> \
    --db <database connection string> # This is probably better as environment variable - MOONSTREAM_DB_URI

Get transaction by hash

ethcrawler transactions get \
    <txhash> \
    --db <database connection string> # This is probably better as environment variable - MOONSTREAM_DB_URI

List transactions in given blocks

ethcrawler transactions list \
    --blocks <list of block numbers or block range in the format {start_block}-{end_block}> \
    --db <database connection string> # This is probably better as environment variable - MOONSTREAM_DB_URI

Jobs

See which jobs failed (with hopefully some log messages or link to Humbug journal entry)

ethcrawler jobs errors \
    --jobtype {blocks|transactions|pending_transactions} \
    --db <database connection string> # This is probably better as environment variable - MOONSTREAM_DB_URI

ethcrawler jobs pending \
    --jobtype {blocks|transactions|pending_transactions} \
    --db <database connection string> # This is probably better as environment variable - MOONSTREAM_DB_URI

ethcrawler jobs replay \
    --jobtype {blocks|transactions|pending_transactions} \
    --db <database connection string> # This is probably better as environment variable - MOONSTREAM_DB_URI

Proposal(Andrey): On stream view right window with detail transaction window can be resizable or be able to toggle.

Currently, there is some confusion because the backend removed the "subscription_plan_id" field, which the frontend was using to determine which icon to put next to each subscription on the /subscriptions view.

We should just use a "subscription_type" field, with possible values: ethereum_blockchain, ethereum_txpool, etc.

Add Bugout humbug reporter to production Moonstream servers:

• backend
• crawlers

We can present this information to users in a more intuitive way.

Ethereum developer docs, for example, have nice content explaining gas: https://ethereum.org/en/developers/docs/gas/

Few issues:

• Going to nonexisting page will keep app reloading instead of showing 404
• We fixed going to existing page by adding custom error redirect on our cloudfront distribution

Currently, when a user registers for an account, they see a blank streams screen.

The easiest thing we can do for now is implement a modal which explains the product to them and how they can use it which only shows up when they first log in and is later accessible through a ? icon on the navbar.

When a new user visits the subscriptions screen, there is no explanation of what a subscription is or how it works. We need more guidance here for the user, perhaps also through a modal. Maybe we can make the ? contextual?

Currently, it is directly using the Brood API and is not Brood application aware.

"Which token is this?"

This was asked of a sample token I used when onboarding a user.

We should crawl information about available NFTs on Ethereum blockchain from sites like OpenSea and Nifty Gateway.

We will use this data to build an index of the NFT market.

The statistics we would like to incorporate into our index are:

1. Number of NFTs listed for sale on primary markets in a given time period (e.g. the past week)
2. Number of NFTs sold on primary markets in a given time period (e.g. the past week)
3. Amount of ETH spent on NFTs on primary markets in a given time period (e.g. the past week)
4. Number of NFTs listed for sale on secondary markets (e.g. auctions, OpenSea, etc.) in a given time period (e.g. the past week)
5. Number of NFTs sold on secondary markets (e.g. auctions, OpenSea, etc.) in a given time period (e.g. the past week)
6. Amount of ETH spent on NFTs on secondary markets in a given time period (e.g. the past week)
7. Amount of ETH earned by NFT creators in primary markets in a given time period (e.g. the past week)
8. Amount of ETH earned by NFT creators in secondary markets in a given time period (e.g. the past week)

Think about these statistics as you decide which labels to associate with crawled NFTs.

Hook into the labels + Ethereum addresses workflow that @kompotkot and @Yhtiyar built and are using for the Etherscan and CoinMarketCap crawls.

• moonstream.to, www.moonstream.to S3 website + CloudFront distribution + A records
• api.moonstream.to - VM image, launch template, autoscaling group, A records (use Bugout load balancer)
• Frontend CI/CD - deploy to S3, invalidate CloudFront cache
• Backend CI/CD - deploy to API servers via SSH

Need to add crawling of the:
https://web3py.readthedocs.io/en/stable/web3.eth.html#web3.eth.Eth.get_transaction_receipt

With it we will have actual gas used for the transaction, which can help calculate transaction fee

Wee need following assets to make look nice moonstream alpha frontend:

Icons

• Moonstream brand icon (white, with text)
• Moonstream brand icon (white, no text)
• Moonstream brand icon (black, no text)
• Moonstream brand icon (black, with text)
• Account
• Dashboard
• Subscriptions
• Stream
• Analytics
• Analysis
• Search
• Delete
• Confirm
• Reject
• Question mark
• Draggable icon (2x3 dots)
• 3 vertical dots (mobile menu)
• Left
• Right
• Settings (gear)
• View/Preview (eye?)
• Contact
• Add new (plus sign)
• Email
• Tokens (key)
• Password (lock)

Illustrations

• Favicon (32x32 png)
• Moonstream brand (SVG, looks good at 44 pixels height)
• Moonstream brand + text (SVG, looks good at 44 pixels height)
• Under construction (1024x1024 square png + SVG version (if acceptable) )
• Contact us (1024x1024 square png + SVG version (if acceptable) )

Landing page illustrations

• Hero illustration (top illustration)
• Paragraph illustrations (TBD)

We want to control storage costs for data we crawl from the Ethereum blockchain. To this end, we want to experiment with using S3 as the storage layer instead of a Postgres database.

For our first evaluation, we will use Activeloop's Hub to store and retrieve blockchain crawl data from S3.

Related to #14

We will only do this post-launch.