Language model inference about hyena-dna HOT 9 CLOSED

Thank you! Much appreciated!

That's an interesting use case. It would be closer to a generative use case it sounds like, which we haven't supported yet, but plan to explore this in future work.

Hacking the current code for generative tasks shouldn't be super difficult though, and we certainly welcome any contributions!

from hyena-dna.

yes the weights for the lm_head are there, but the provided loading ckpt code is implemented such that it replaces the lm_head with a new head (scratch) for a downstream task.

If you literally will use the same pretraining model, ie no change to the architecture, then it's even easier, you just pass in the model ckpts using the flag train.pretrained_model_path=path_to_ckpt. This will resume training from the ckpt - you would need to modify code to do inference on some dataset yourself.

Note: this doesn't just give you a DNA generator out of the box. Remember, you're only predicting a single next token given a sequence. If you want to generate a whole new sequence, then you'd have to do this in a loop and prepend 1 token at a time as you decode, which will be relatively slow. There's a whole field dedicated to fast inference and decoding/generating (but we don't support on this repo).

Going to take some hacking :)

You would need to just modify that code.

from hyena-dna.

Hi Eric, thank you for the update!

I've been playing since yesterday, after I saw announcements about implementation in HF.

I have a simple Colab notebook to evaluate the (averaged) log-likelihood of a sequence. Does it look correct?

BTW, is it possible there's a small bug when computing the loss? I wonder if it should use self.vocab_size.

from hyena-dna.

Just to follow on this topic, does the weights on HF also contain the lm_head weights ? And if not could you easily share those weights ? :)

from hyena-dna.

Alright it's perfect, thanks for clearing that up :)

from hyena-dna.

A way to get logits:
https://github.com/HazyResearch/hyena-dna#getting-logits-from-pretrained-model

To clarify though, it's not generating new sequences necessarily from this, it's generating 1 new token at the very end, so you'd to do a for loop to actually generate.

from hyena-dna.

Hi @gonzalobenegas, you're totally right about the bug - my fault, I wrote that part of the port! It should be fixed now.

from hyena-dna.

@gonzalobenegas looks good to me!

from hyena-dna.

Great, thank you!

from hyena-dna.