Hello, very nice and inspiring work!

However, I noticed that the size of Metal Ion Binding dataset is smaller than the number in the paper.
In paper, the dataset size is (valid:1066, test:1083)，but in the lmba file uploaded, the dataset size is (valid:664, test:667)

How could I get complete data?

Thank you!

I noticed during the finetuning process of the downstream task: overfitting phenomenon occurs when the finetuning epoch is set to 100.
For example, when fine-tuning with ESM-2 8M model on GO/BP annotation task, the valid-f1-max keeps rising, but the valid loss overfitting.

Test on the lowest valid loss ckpt (like 50 epochs), the test-f1-max = 0.2259
Test on the highest valid-f1-max ckpt (like 100 epochs), the test-f1-max = 0.2092

Therefore, I am curious about downstream finetuning. During training, will you do an early stop based on the overfitting phenomenon?

Dear Sir,

Why is the EC GO result in Saprot so much lower than the original paper "Enhancing Protein Language Model with Structure-based Encoder and Pre-training"? I wonder if the dataset is different.

Hello, thank you very much for this wonderful work!

I was just wondering if, given a protein sequence and its corresponding 3Di structure-sequence (obtained via Foldseek), is it possible to extract fixed-length protein embeddings using SaProt? Would there be a sample script for this particular task?

Thank you.

Hi,

I'm interested in integrating SaProt into an application and was wondering under what license are you releasing the code.

Thanks beforehand!

Thanks for this impressive solution!

When I run the tokenizer on an input sequence, there are always two additional elements added to the tokenizer output. Why is this and what do the values represent?

For example:
print(len(sequence)) -> 5
inputs = tokenizer(sequence, return_tensors="pt")
print(inputs_1['input_ids'].size()) -> torch.Size([1, 7])

Additionally, I'm trying to generate a fixed length sequence embedding. I saw you answered how to do this with the ESM model, but is there a way to do so with the huggingface model?

Thanks for your help!

Hello, your work is truly impressive!

However, I don’t understand why the length of the SA-token is 9, but the output's length is 11 in the example code.

"""
['Md', 'Ev', 'Vp', 'Qp', 'Lr', 'Vy', 'Qd', 'Ya', 'Kv']
torch.Size([1, 11, 446])
"""

Could you please explain it to me?

Thank you!

Thank you for your great job and the sharing of the repository!
I wonder the meaning of the output size torch.Size([1, 11, 446]) in your example in README. I suppose '446' is the size of the vocabulary, but why it's different from '441' provided in your article?
Moreover, do you provide codes/scripts to load batches/bulks of sequence into your model like ESM?
Thank you very much!

Awesome work!

Will the entire 40 million-sequence pretaining dataset be made available?

Hi this is a fantastic project.

I want to use your model for a research project I am working on. I am having trouble setting my configuration to where the SaProt_650M_AF2.pt file is stored. How do I get this .pt file from the GitHub?

Dear authors,

Thank you for sharing this great work with us!
I wonder if its possible to extract the per residue representations like with ESM-2?

import torch
import esm

# Load ESM-2 model
model, alphabet = esm.pretrained.esm2_t33_650M_UR50D()
batch_converter = alphabet.get_batch_converter()
model.eval()  # disables dropout for deterministic results

# Prepare data (first 2 sequences from ESMStructuralSplitDataset superfamily / 4)
data = [
    ("protein1", "MKTVRQERLKSIVRILERSKEPVSGAQLAEELSVSRQVIVQDIAYLRSLGYNIVATPRGYVLAGG"),
    ("protein2", "KALTARQQEVFDLIRDHISQTGMPPTRAEIAQRLGFRSPNAAEEHLKALARKGVIEIVSGASRGIRLLQEE"),
    ("protein2 with mask","KALTARQQEVFDLIRD<mask>ISQTGMPPTRAEIAQRLGFRSPNAAEEHLKALARKGVIEIVSGASRGIRLLQEE"),
    ("protein3",  "K A <mask> I S Q"),
]
batch_labels, batch_strs, batch_tokens = batch_converter(data)
batch_lens = (batch_tokens != alphabet.padding_idx).sum(1)

# Extract per-residue representations (on CPU)
with torch.no_grad():
    results = model(batch_tokens, repr_layers=[33], return_contacts=True)
token_representations = results["representations"][33]

Thank you in advance!

Hi there.
I do not have a great understanding of 3d structure formats and my questions might be very basic to others:
Can we convert the 3Di tokens produced by foldseek to the actual 3D PDB files?
I want to develop a model to get a protein sequence and returns their 3Di tokens as its 3D structure. Imagine that it would be a perfect model in prediction. How can I use the model in real life for 3D structure prediction??
Should I convert the outputs to a specific format?? How can I evaluate the 3D structure prediction of the model with respect to the true 3D structure?

Many thanks to whom can answer my questions.

SaProt_650M_AF2 link is wrong and links to SaProt_35M_AF2 on huggingface instead.

Hi, thank you very much for sharing this great work.

After loading the pretrained checkpoint, I encounter some warnings:

Some weights of EsmModel were not initialized from the model checkpoint at SaProt_650M_PDB and are newly initialized: ['esm.embeddings.position_embeddings.weight', 'esm.contact_head.regression.bias', 'esm.pooler.dense.bias', 'esm.contact_head.regression.weight', 'esm.pooler.dense.weight']

My code is

from transformers import EsmTokenizer, EsmModel
tokenizer = EsmTokenizer.from_pretrained("SaProt_650M_PDB")
model = EsmModel.from_pretrained("SaProt_650M_PDB").cuda()
model.eval()

I want to use the pre-trained model to obtain representation of a protein directly without finetuning, but I'm not sure if missing these weights affects the quality of the obtained representation, e.g., the absence of position embedding weights.

Hello,

I see you have made .mdb dataset files available. How would one go about simply extracting and using the fine-tune data for downstream tasks? I would like to fine-tune my own model so the training script will not work.
Best,
Logan

Hello, I am currently exploring your project and have two questions regarding the data types:

1. Raw Data Composition: Does the raw data provided in your repository include 3D coordinates? I check some of the mdb data, only find amino acid sequences with 3D structure embedding sequences. Can you provide raw data with 3D coordinates?
2. Data Utilization in Downstream Tasks: In the context of downstream tasks, are you primarily using the real data, or are you employing data generated by AlphaFold? I find all data with the PLDDT.
   Answering these questions will help me a lot. Thanks in advance.

After following the tutorial on GitHub to download the data for downstream tasks and extracting it, we encountered an issue. We are unable to open the MDB files provided for the downstream tasks using Access locally. Upon attempting to open these files, we receive an error message, which I will attach below for reference. We are using Access version 2019. We seek clarification on the possible reasons causing this issue and inquire if alternative versions of the data in different formats can be provided.

Error Message:

We kindly request your assistance in understanding the root cause of this issue and if possible, providing alternative versions of the data in different formats. This would greatly facilitate our progress in the tasks at hand.

1)for some reason esm_foldseek_model.py badly formatted.
2) I can't find config file which correspond to esm_foldseek_model.py
3) I can't find training data for this model on google drive.
4)I really interested to check strategy provided below.

from your publication:
"In Strategy 1, there is a requirement for simultaneous prediction of both residue and Foldseek tokens.
Intuitively, if the accuracy of Foldseek or AF2 structures is insufficient, predicting them simultane-
ously may result in suboptimal outcomes. On the other hand, Strategy 2 intentionally discloses
structure token information during the input phase, allowing the model to primarily concentrate on
predicting residue types. Overall, the performance disparity between the two masking strategies is
not considerable, with Strategy 2 usually exhibiting some improvements"

I would say results are almost identical.

I can suggest Strategy 3 without masking,
input : EsmFold predictions in SA format, output :AF2 predictions in SA format.
as second choice can be taken ProstT5 predictions

what need to be done is training set : it is requires some preprocessing of large amount of data.

compare EsmAtlas data (600M proteins) with AF2 data(200M proteins)
find proteins with identical sequences and convert them into SA (EsmFold and Alphafold2)
my expectation :number of such identical sequences will be large enough.

I don't think we need run training from scratch it can be implemented as fine-tuning procedure.

Hi all, thanks for the open-source release! Are you also planning to release the fine-tuned model weights, namely the Thermostability model?

Thx for your great job!
I wonder when will release the code about Contact Head prediction and evaluation. Since the evaluation pipeline seems quite different from ESM-2 and TAPE.

Hello, I downloaded the ClinVar .tar.gz file from your directory. I noticed that all of the fitness values are '1.0'. The ProteinGym dataset reports various fitness values. Is there a reason you have only kept the '1.0' fitness score ones ?

I download foldseek from here https://drive.google.com/file/d/1B_9t3n_nlj8Y3Kpc_mMjtMdY0OPYa7Re/view
and put into /home/ubuntu/bin/
but maybe it can't be used

thanks for your answer.

Hi Sir,

How about the GPU memory cost when finetuning 650M SaProt model. I got OOM error when I try the finetuning script according to the command which the README file provides. My GPU hardware has 24G memory.

Hi there, thank you for your impressive work.

I downloaded the pretraining dataset that you published here: https://huggingface.co/datasets/westlake-repl/AF2_UniRef50

However when I load the DB I find that the validation set contains only ~20k uniprots. Your paper says:

"B PRE-TRAINING DATA PROCESSING
We adhere to the procedures outlined in ESM-2 Lin et al. (2022) to generate filtered sequence data,
and then we retrieve all AF2 structures via the AlphaFoldDB website https://alphafold.
ebi.ac.uk/ based on the UniProt ids of protein sequences, collecting approximately 40 million
structures."

And ESM-2 Lin et al. (2022) says:

"A. Materials and Methods
A.1. Data
A.1.1. SEQUENCE DATASET USED TO TRAIN ESM-2
UniRef50, September 2021 version, is used for the training
of ESM models. The training dataset was partitioned by
randomly selecting 0.5% (≈ 250,000) sequences to form
the validation set."

So I am wondering whether there is some data missing (i.e., the remaining ~240k validation uniprots) or if I have done something wrong.

Many thanks in advance.

Hello, I'm working at an academy.
I'm trying to run a SaProt following README.

But there is no SaProt model in the weight folder.
I was wondering if you could publicly provide them.

Looking forward to hearing from you.

Could we kindly provide the config and data preprocessing script to reproduce the pretraining of esm2_t33_650M_UR50D ?

esm_foldseek_dataset.py:

label_ids = pad_sequences(label_ids, -1)

NameError: name 'pad_sequences' is not defined

please , close issue , I have found function and just copied to my file.

for some reason in my python environment class method function don't see global variable.
Any advice how it can be fixed without changing code will be appreciated.

I have rewritten it using more straightforward "hack"

class ModelInterface:
@classmethod
def init_model(cls, model_py_path: str, **kwargs):
sub_dirs = model_py_path.split(os.sep)
module_name = '.'.join(sub_dirs[:])
module = importlib.import_module(module_name)
objs = dir(module)
model_cls = getattr(module,objs[1])
#"EsmRegressionModel")
return model_cls(**kwargs)