The ligands have several cases where there are multiple conformers & protonation states for the same ligands.

We need:

• An overview of which datasets are affected & how many "duplicates" there are, including the type of duplicates
  • Note: an easy way of getting this is by looking at the original benchmark results for the ligands: https://github.com/schrodinger/public_binding_free_energy_benchmark/blob/main/21_4_results/ligand_predictions/bayer_macrocycles/ftase_extraligs_custcore_stereo_out.csv
• An overview of how many more edges you get when creating a lomap network with those extra edges
  • Maybe just a few systems so we can show a subsample?

Based on feedback from indutry partners

Should we remove the request to copy over the edges.csv file and ligands.sdf file and just go ahead and ask that folks select whichever ligands they think is best at the preparation stage?

We need to create a script that does the following:

• Takes a set of inputs (ligand SDFs, cofactors SDF, protein PDB)
• Create a LOMAP LigandNetwork (note needs fixing to deal with charge changes)
• Create an RBFE AlchemicalNetwork
• Selectively apply a different set of Protocol settings when dealing with net charge changes

This is about trying to define the attributes, properties that we need to get out of the benchmarking in order to be able to use the data for many things (benchmarking analysis, building better scoreres, etc.)

I wrote an initial list and would like to get your feedback and additions

Please:
https://docs.google.com/spreadsheets/d/1z7W-WxeW2OOnh2B5SaJb8sacYmN9cVE5JtV5PQdLJH4/edit?usp=sharing

We should have a TOC that does something like this:

• Industry Benchmark (Overview)
  • Benchmark phases
• Phase 1: structure prep
  • Preparing system
  • Contributing system
  • Review instructions
• Phase 2: running simulations
  • Simulation overview
  • Carrying out the simulations
  • Hardware requirements
• Phase 3: analyzing results
  • Foo
• Phase 4: paper writing

We need to update the readme explaining:

1. What this repo is
2. What the structure is
3. What folks should look at when getting started

We need a set of instructions for reviewers of submitted inputs

Based on partner feedback, we should tell folks to work from a fork rather than a direct branch (as they won't have access unless we add them to a specific OFE team).

We need a file that tells folks who are unfamilliar with github how to contribute inputs / open issues etc...

Possibly take a brief primer from somewhere else?

We need a utility script that will run a very short simulation and check that a given PDB input, and optionally cofator SDF, will be correctly ingested and simulated by OpenFE.

Make a snapshot copy of the Schrodinger v2.0 inputs from https://github.com/schrodinger/public_binding_free_energy_benchmark/tree/v2.0

We need a list of systems that have been claimed for benchmarking (and hopefully by whom).

We need to define how the ligands will be selected for the public benchmark set.

Currently the schrodinger inputs contain a mixture of different conformers & protonation states, should we use the ligands as defined by Schrodinger or should individuals select whichever ligands they deem to be most appropriate.

We need to add a "how to prepare your inputs" instruction file that contains: https://docs.google.com/document/d/1UOIt2bv2MLvxuHZ-ZvZcruRvmVW2hSt_SVni6mCFT0I/edit?usp=sharing

This file should be continually updated based on participant feedback.

We should create a plan for what analyses do we want to do on the benchmark results, i.e. draft a general overview of the analysis section of a paper.

https://industrybenchmarks2024.readthedocs.io/en/latest/#public-dataset-benchmarks has a few broken links

Based on today's call:

• maybe inputs to structure_inputs
• organize prepared structures as directories under challenge_set/target_name/
• try to enforce the same names in submitted files (i.e. protein.pdb, ligands.sdf, cofactors.sdf, edges.csv) [worst case we normalize it ourselves after the fact]

We need:

1. A PR template that outlines the key steps for input file submission
2. An example directory with examples of the relevant files (i.e. a PDB, ligand SDF, cofactor SDF, and short markdown file with relevant details on how the system was prepared).

This doesn't have to be complete, but we need to give an overview of each phase, relevant timelines and key instructions that individuals should follow.

The current theme isn't suitable for easy navigation. We need something that can be easily navigated via a sidebar TOC.

We need an issue template for FAQ queries