Moonstream users + subscriptions + sources data model about api HOT 3 CLOSED

One possible architecture:

1. We create a private journal called moonstream-data. Only our crawlers, the Moonstream API, and the bugout team have access to this journal.
2. Crawlers put all crawled data into the moonstream-data journal using tags that denote the source type (e.g. ethereum_blockchain, ethereum_txpool, etc.).
3. When a user calls the /stream endpoint on Moonstream API, Moonstream queries the moonstream-data journal for events from that user's subscriptions and returns only those events back to the user.

from api.

For some sources, we will show the user all crawled data (e.g. ethereum_blockchain).

For other sources, we will only show the user data from when their subscription started (e.g. ethereum_txpool). This prevents us from leaking information to one user about when other users became interested in an address in the transaction pool.

from api.

Two invariants:

1. Single journal (moonstream-data) into which all crawled data goes.
2. Users only have access to moonstream-data through the Moonstream API, and their access is determined by their subscriptions.

This way, we don't have to duplicate data from multiple sources across multiple users, which will save us a lot in terms of storage costs.

from api.