swfsql / dfdx-mamba Goto Github PK
View Code? Open in Web Editor NEWMinimal (non-optimized) port of Mamba.
Minimal (non-optimized) port of Mamba.
To reduce the amount of dependencies (or rather concentrate more on dfdx), the Tokenizer (token encoding/decoding) should be ported to dfdx.
This also applies to the token processor, which makes a token selection based on the Vocab output logits.
Some data structures must have specific initialization so they allow the model to efficiently get trained and also so some properties can emerge from the structure of the data and the model is able to work correctly.
Currently the model is initialized at random, which is incorrect.
This is more important when it's possible to train the model, and when it has some cuda support.
The complete softplus operator, which takes the threshold into account to avoid loss in precision, should be added as an actual operator on dfdx and dfdx-mamba should make use of it.
Currently the model only implement forward
methods, and no forward_mut
. Both styles (stateful and stataless) should implement the forward_mut
methods.
But higher priority should be given to the stataless forward_mut
implementation, since this is more efficient at training.
Saw something related to this in reddit.
Many underlying operators don't support cuda, and thus dfdx-mamba also doesn't.
Note: this does not refer to a fused Cuda kernel, just a minimal Cuda support based on underlying operators.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.