lucidrains / mlp-mixer-pytorch Goto Github PK
View Code? Open in Web Editor NEWAn All-MLP solution for Vision, from Google AI
License: MIT License
An All-MLP solution for Vision, from Google AI
License: MIT License
Thank you for the clear and well-executed implementation.
Following up on this issue: #11
May I kindly ask why you chose to expand the token-mixing MLP while bottlenecking the channel-mixing MLP? Is there a particular reason behind this design, or is it simply because this setup provides the best performance?
Since Google has been released the pretrained model of MLP-Mixed, how to load the pretrained model with your model?
Thanks for your implementation. In comparing your codebase to the author's implementation, I discovered that while you have a single expansion factor in your configuration, the authors have separate values - one for tokens and one for channels.
Specifically, their channels expansion factor is 4, but their tokens expansion factor is 0.5. (The hidden_dim is the base projection size). Note that they actually use a feature count, but I'm translating to the mechanism you use in this codebase.
Thus, when executing the MixerBlock, the tokens "expansion" is actually a bottleneck.
The parameters can be verified as well in Table 1 ("Specifications of Mixer Architectures") at the top of page 4 in version 4 (the current version as of Feb 14, 2022) of their paper.
I'm not suggesting that anything necessarily needs to change in your implementation. However, if you wanted to align your codebase to be able to fully replicate the author's work, you may consider allowing for two separate parameters - token_expansion_factor and channels_expansion_factor.
Thank you again for this work, and for all your contributions generally. You are a an incredible asset to the community.
Amazing work! How difficult is it to implement mlp into Dall-E? As the whole idea around Dall-E evolves around attention layers and transformers, I wonder if this simpler model would enable smaller, equally capable models...
Why are the parameters so different from the one in the paper?
Take the S-16 as an example:
I compared your model with official imlementation model (Jax version), maybe you made a mistake.
I can not find any transpose operation in your model, which is necessary for Mixer layer.
If i do not understand your model, please correct me ,thanks a lot!
Can this work with dynamic size?
From reading the paper, it doesn't look like that's possible.
Is that right?
If so, what a shame...
Do you plan to add any training script?
What GPU did you use to run the experiment? I can't run experiments on V100 32G.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.