At the moment blockprint's DB is not handling sidechains very elegantly.

The database is designed so that a block can be inserted without knowing its parent. This is nice because it provides the ability to stay online processing new blocks, even if some old blocks have been missed due to temporary downtime. There's a background process in background_tasks.py which is meant to query the API for gaps to fill in, and patch them up.

The problem is that the logic for determining gaps returns some gaps that are impossible for the background task to heal. A gap is currently defined as a slot interval between a block with a parent missing from the DB (end_slot) and the last known block prior to that missing parent (start_slot):

E.g. the current output from https://api.blockprint.sigp.io/sync/gaps is:

[
  {
    "start": 8253079,
    "end": 8253086
  },
  {
    "start": 8253031,
    "end": 8253045
  },
  {
    "start": 8253127,
    "end": 8253151
  },
  {
    "start": 8254502,
    "end": 8254511
  },
  {
    "start": 8277096,
    "end": 8277097
  },
  {
    "start": 8299646,
    "end": 8299647
  },
  {
    "start": 8299650,
    "end": 8299651
  }
]

Looking at the first gap, we see that the block with missing parent that triggered this must be one at slot 8253087, which is reorged out (beaconcha.in doesn't even know about it): https://beaconcha.in/slot/8253087. Our Lighthouse nodes saw it though:

Jan 21 18:17:49.412 DEBG Cloned snapshot for late block/skipped slot, block_delay: Some(2.040814714s), parent_root: 0xd429dc371766b1d71fdad731879aafe7c2df990b402fbc5704b29144009cce8f, parent_slot: 8253079, slot: 8253087, service: beacon

Now the interesting thing here is the parent slot, https://beaconcha.in/slot/8253079. It's also empty! In order to heal the gap, we would need to load this parent block at 8253079, which we can't do because it has also been pruned.

In summary, blockprint's gap healing is broken for sidechains of length > 1. I can think of two ways to fix it:

Give the background task the ability to either delete or mark orphaned blocks in the database when: the slot of the missing parent has been finalized as a skipped slot. If we just mark them as orphaned, then we get to keep them in the DB (moar data) but won't block the gap healing process on them. On the other hand, marking them orphaned would require a new database column and a little DB migration (not too bad, given the small number of live blockprint instances).

It would be cool to have features named and capable of being toggled on and off from the CLI.

It would also be nice to include feature names on the matplotlib graphs, with a 2D cross-section for each pair of features.

We should support Grandine in the DB. This will be a breaking schema change unfortunately, as it introduces a new column (pr_grandine). We could provide a simple migration script to upgrade existing DBs so they don't need to be rebuilt from scratch.

Some graffiti regex should only be enabled on e.g. Prater. The ./prepare_training_data.py script should take a --prater or --mainnet flag to switch regex on/off, or a path to a file containing the regex.

Description

I just downloaded the "raw" block rewards for ~6M main net slots (~1TB of data), which after the classification, stays in ~130GB of "model" (see table below).

The model (size):
2.5G    ./model/Prysm
4.3G    ./model/Nimbus
31G     ./model/Teku
1.2G    ./model/Lodestar
92G     ./model/Lighthouse
130G    ./model

There are a few questions I would like to open here:

1. Is this an overkilling number of block rewards to train the model?
   I'd like to have an accurate enough model, but this seems a bit overkill to me. I'm unsure if you could guide me here based on your experience.

2. Loading the model currently takes around 50mins, is this an expected duration for the range of time I'm managing?
   I checked if there is any concurrency parameter that I could be missing when loading the model (meaning creating an object Classifier from the given model/folder), but it seems to be sequential by itself.

3. I also have some doubts about the possible features of the Classifier. I'm currently using the default ones, and I'll like to know if you guys have better performance using a different combination of features.

Many thanks in advance, and great work with this repo!

Currently blockprint's API will return 0 values when querying a section of the database for which history is missing. This can occur if the beacon node backing blockprint has been falling in and out of sync or is re-syncing and lacks a portion of history.

At the moment these gaps can be detected via the /sync/gaps API, but this is error-prone and not obvious. It would be better if blockprint detected queries affected by the gaps and failed them explicitly so that the user doesn't have to check /sync/gaps manually. E.g. if there is a gap from slot 3623682 to 3687456 (as there is currently) then blockprint should 404 on queries like /blocks_per_client/113625/113850 rather than returning a response with 0 blocks per client for all clients.

The reason the API is the way it is currently is that the SQL query to detect the gaps takes several seconds to evaluate, because it scans the entire table. If this query can be made to run quickly, then it will be feasible to run it for every query.