swfsql / dfdx-mamba Goto Github PK

To reduce the amount of dependencies (or rather concentrate more on dfdx), the Tokenizer (token encoding/decoding) should be ported to dfdx.

This also applies to the token processor, which makes a token selection based on the Vocab output logits.

Some data structures must have specific initialization so they allow the model to efficiently get trained and also so some properties can emerge from the structure of the data and the model is able to work correctly.

Currently the model is initialized at random, which is incorrect.

This is more important when it's possible to train the model, and when it has some cuda support.

The complete softplus operator, which takes the threshold into account to avoid loss in precision, should be added as an actual operator on dfdx and dfdx-mamba should make use of it.

Currently the model only implement forward methods, and no forward_mut. Both styles (stateful and stataless) should implement the forward_mut methods.

But higher priority should be given to the stataless forward_mut implementation, since this is more efficient at training.

Saw something related to this in reddit.

Many underlying operators don't support cuda, and thus dfdx-mamba also doesn't.

Note: this does not refer to a fused Cuda kernel, just a minimal Cuda support based on underlying operators.