This subnet is going to generate a lot of data, and we are in need of a way of storing data. We need to know:

1. Possible providers if not Runpod volumes
2. When to use wandb (what data / plots)
3. How often should we log results?
4. How much data does each simulation create (record not only datasize, but parameters about the simulation, so we can determine if file size is related to things like protein shape, complexity, ect...)

• #27
• #28
• #29

gromacs takes forever to install so we need a dedicated way to spin up pods with this environment. I recon Docker?

How can we know if a pdb is going to be a good candidate for simulation? Are the initial conditions that stop us from being able to simulate? Estimate the total number of eligible candidates.

There are lots of variables that can be tuned to determine what pdb configurations can be sent to the miner.

There are some issues tracking aspects of this:
#17, #18

This issue will keep an open record of how we are increasing the problem space.

This issue is to address the questions of removing HETATM and CONECT records from the .pdb file before creating a topology. This is an approach that was found in one of the tutorials, and it is noteworthy to understand what its doing to see if we need to apply it to the pipeline or not.

Deadline for Folding MVP is April 19th.

Here is the timing that we are going to impose for the following week with key deliverables highlighted:

Tuesday April 9th

@mccrindlebrian

• Identify the 3/4 most important parameters for the validator hyperparameter space. Identify the most popular options via literature for these parameters and use this as the basis for the grid search
• Put sampling routine into code, recording what parameter combination you have already sampled, and record this data in a local wandb proj (pdb_id --> parameters --> success/fail)
• Simulate at least 100 validator forward passes. If certain flags are present in pdbs, then it is possible that the pdb will not run at all.

@schampoux

• How can we make all pdb simulations look like the ones you did? Specifically, identify what the important parameters that need to be set in the .mdp files (or any other file used for simulation) for us to achieve this goal. Parameters like nstenergy. Look at your old simulations, there should be a clear signal here. We are currently altering some important parameters the Protein class in the edit_files method. Issue

• Tied to this is, of course, the amount of (tabular) data that the miner is saving to the corresponding energy/rmsd files. Once you are confident regarding point (1), please integrate your data extracting logic into the codebase. folding/utils would be a good place for this. We will need this code for sending it downstream to validators.

Wednesday April 10th

@mccrindlebrian

• Create the code infrastructure for the reward landscape. Deliverable here is V0: miner with the lowest loss wins 100%. To do this, I will need you data parser from above @schampoux

@schampoux

• The data that you parse needs to be recorded, and plots need to be made. Setup 2 Wandb projects, one for the miners and one for the validators. Incorporate code that uses the data parsing methods you made to parse the output files, save them into dataframes, and save the data to wandb. Data should include ALL information we need (uid, hyperparameters, ect..). Make sure that the data we care about is being plotted on wandb correctly! I can also support this wandb effort. We should prioritize data being recorded from the validator side of things. The miner can come later, if time limited.

Thursday April 11th

@mccrindlebrian

• Now we should be well set up to run a set of simulations entirely in-loop. We want to run simulations where we have a 1-to-2 validator-to-miner setup. This means that we can properly test out the miner scaling with the reward mechanism.
• Furthermore, we can generate a lot more data. Scale this up to 5 validators. We will need multiple machines. Spin up runpod instances that have gromacs installed so this is not a blocker.
• Run these experiments where you have all validators on one side, and all miners on the other. This will give us better clarity on how we save and send data over the wire.
• Work that @schampoux did regarding save frequency will be useful here. Ideally, we do NOT have to touch miner code at this stage. Should be relatively set in stone.

@schampoux

• Keeping this as a buffer day, as the above could take some time. We will evaluate your bandwidth when we get here. I might need you for the above.

@RodrigoLPA

• Looking at default gromacs force-field information. What fields are available, does gromacs do anything smart when it comes to default parameters/choices based on the pdb?
• Looking into MISSING flags, and general cleaning of PDBs to increase success rate of running a simulation on the validator side (for reference, look into folding/protein.generate_input_files()

Friday April 12th

@mccrindlebrian & @schampoux

• At this point, we should have many simulations, and a ton of logged data to wandb. Create scripts to pull the data (copy from analysis repo) so we can do an analysis of the success rate on the validator side. KPI here is > 50% of pdbs sampled were able to find a set of hyperparameters that finished and set the completed data to the miner
• Get wandb working with miner architecture, if not done.

RUN EXPERIMENTS OVER THE WEEKEND

Monday/Tuesday April 15/16th

This week is for refining the codebase. We should have identified some clear oversights in either design, logging, ect.. at this point.

@mccrindlebrian

• Reward mechanism can be tightened up here. Do some data science on the reward curves to see variabilities between miners, test different scaling methods for rewards. Condition miner variability (std, kurtosis ect..) on hyperparameter combinations. There could be some interesting relationships that pop out here, which might ultimately indicate things like protein complexity (size, number of heads, ect...). This is where @steffencruz could be helpful

@schampoux

• Improve wandb logging. Probably needs more fine-grained information
• If we haven't done so already, we need to determine the best way to send data over the wire to the validator from the miner. csvs, files directly, binary??? Please look into this

Wednesday-Friday April 17-19th

• Run many many simulations. Get data. Fix bugs. Release!

are there uncontrolled exceptions, or other runtime issues (not related to folding necessarily)

There are many ways to design the reward mechanism.

We need to benchmark the chosen reward mechanism(s) in many ways before we can deploy this on mainnet. We need to understand the expected results in terms of miner rewards, competitiveness, dependency on hardware, etc. Main points are:

• #25
• #20
• #21
• #22
• #23
• #24
• #26

How similar are miner coordinates after running a simulation on the same initial state (stability) and how similar are their energies? Quantify any relationships.

sometimes no miners respond for a job. Do we kill this job after some amount of time?

We can execute GROMACs commands in two ways in python:

1. Using the GromacsWrapper, or
2. shell commands

We need to know which one is more suitable here (pros/cons). Things like

1. running in background threads/processes,
2. error handling,
3. code clarity and extensibility.

We need to understand how large this problem space is so that we do not exhaust all of the proteins too quickly and effectively kill the PoW component (since lookups would become the norm). Key points are:

• #19
• #16
• #17
• #18

Are there any issues that arise when there are edge cases (multiple validators querying the same miner, no miner responses available, invalid miner responses)?

Validators seem to generate lots of backup files when running rerun commands which is problematic. Find a graceful way to stop checkout generation

Can we prevent lookup attacks by submitting intermediate results?

The gro_hash method in folding/utils/ops.py encounters errors for certain proteins/DNA. This method is responsible for generating the hash for a specific gro file. It does this by parsing the .gro file, connecting the residue name, atom name, and residue number from each line together into a single string. The error arises because some of the atoms contain apostrophe's.

The goal for this issue is to increase the robustness of the gro_hash generator so that it can accommodate for cases like this.
Another goal for this issue is to track other errors.

There are different parameters that you can set to ensure that simulations are deterministic. This is mainly done by applying mdrun -seed. However, there is also some hyperparameters that can be set for grompp commands. We should implement this for V0.

• the miner forward should always use the miner uid as the seed
• is the seed verifiable in any of the files? xvg...

if a job has not been queried in t > neuron.miner_job_timeout, then we should remove the job.

Failed to attach files for pdb 1qcc with error: No files found for md_0_1 happens when we have old data, but we never finished the simulation. We should see this behaviour and start the simulation from the most advanced checkpoint we have if we haven't finished

Instead of implementing ES logic externally we prefer to use GROMACS to do this. This should reduce bugginess, instability and complexity.

What is the characteristic timescale for simulations? In order to answer we need to evaluate:

1. How many steps do simulations require to convergence in general,
2. and how much wall time is this?
3. What should we set timeouts to? Are there any stability-based metrics we can come up with that warrant when we should shut-down a simulation?
4. How long is a forward pass?

https://gromacs-py.readthedocs.io/en/latest/installation.html

To analyze and identify optimal hyperparameters for protein folding simulations using based on a dataset of PDB IDs. The goal is to determine which hyperparameters correlate with successful simulations and to explore why simulations are failing.

This issue is for informing us about .mdp files so that we can structure our experiments with intent.
Please see the official guide for all mdp options.

Here are the details for the final week's sprint on Folding. Folding is currently in a place that can operate such that is has high success rates for validator hp search, but validators in general are not busy, and there are open questions about how miners are asked for work. Here are some key components

1. gro-based energy calculations (exploit resistant)
2. Validator scheduler & DB
3. Hyperparameter tuning (timeout, wait_time, sample_size, max_concurrent_pdbs) for optimal workload (adaptive?)
4. Miner process pool executor for multiple jobs
5. Some restructuring of data directories (low priority?)
6. Logging and persistence of final results
7. Data viz
8. Loads of experiments

We can try and break these down into tasks for each member of the team.

@steffencruz

• Validator scheduler and DB management.
  - Validators to ping miners to see if they are alive
  - Validators can process/wait more than 1 pdb at a time
  - Validators can query miners at some pre-defined interval to acquire .gro files

@mccrindlebrian

• Mining infrastructure
  - Miner process pool executor (aka: being able to pull out pdbs out of the queue and not always having to wait for another Pdb request from a validator), and spinning up an N number of child processes based on the capacity of their CPU. Need to check if this design is a problem with GPU-compiled Gromacs?
  - Being able to be interrupted during simulation, return requested data, and continue simulation (with or without a new ckpt file?)
  - Therefore a part of this is storing which validators get what pdb? I guess this is handled within the child process itself so we don't need to explicitly keep track of this?

• Logging and persistence of final results
  - This infra is already in place, I just need to add more to the miner side. This requires some refactoring of the run_commands function

@schampoux

• Gro-based energy calculations
  - Implement a methodology on the validator side that can apply the necessary steps to compute the next step (or next N steps) of any energy file via the -rerun method.

• Data visualization
  - We can currently plot data if given in the reward stack (we log this to wandb) but it would also be good to see if we can do this on the wandb side (if the data is available)
  - More importantly, we want to see the protein! Find a way to show the protein given the necessary files. This can be separate from wandb

@RodrigoLPA

• We need to you to run your parallelization pdb script and gather all the statistics.
  - Number of successful pdbs
  - What are the properties of these pdbs? Distribution of size/ other metrics of complexity. It looks like simple pdbs might be out of the question since they optimize too quickly. We need to know how large this space is.

Everyone

We need to run a boat load of experiments, specifically:

• Run with multiple validators (3) on different machines
• Run with at least 10 miners

I think this is a good opportunity for @RodrigoLPA to learn how to spin up miners and validators, use pm2, and manage across multiple machine. I am happy to support here with 1 on 1 time getting him up to speed

Fine tune hyperparameters: number of steps and expected reward/loss, as a function of protein 'difficulty'

Do we let miners decide where they start, or should we be giving them an initial condition? What about a random seed? This ties into being able to verify their computation.

This issue will uncover the cause for large file buildup in the validator directories, as well as strategies to mitigate excessive file growth from the simulations.

There is a parameter epsilon that determines the minimum threshold needed to indicate what is an "improvement" in loss. This is set to some arbitrary number. Gather data and measure what this value should be. Static, or dynamic?

In order to prove that this system can work as a subset, It is essential that we have a subset of pdb files to test the system. This is not easy because not all proteins behave the same way under similar experimental conditions. The purpose of this issue is to outline how to effectively find these subsets of "similar" proteins. If the goal of the simulation is energy minimization and we wish to switch out PDB files with minimal adjustments to hyperparameters, structural similarity is the most relevant criterion.