Light

sea-snell / grokking Goto Github PK

View Code? Open in Web Editor NEW

50.0 4.0 14.0 1.87 MB

unofficial re-implementation of "Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets"

License: MIT License

Python 100.00%

artificial-intelligence deep-learning grokking neural-network transformer transformer-models python pytorch

grokking's Introduction

GROKKING: GENERALIZATION BEYOND OVERFITTING ON SMALL ALGORITHMIC DATASETS

unofficial re-implementation of this paper by Power et al.

code written by Charlie Snell

pull and install:

git clone https://github.com/Sea-Snell/grokking.git
cd grokking/
pip install -r requirements.txt

To roughly re-create Figure 1 in the paper run:

export PYTHONPATH=$(pwd)/grokk_replica/
cd scripts/
python train_grokk.py

Running the above command should give curves like this.

Try different operations or learning / architectural hparams by modifying configurations in the config/ directory. I use Hydra to handle the configs (see their documentation to learn how to change configs in the commandline etc...).

Training uses Weights And Biases by default to generate plots in realtime. If you would not like to use wandb, just set wandb.use_wandb=False in config/train_grokk.yaml or as an argument when calling train_grokk.py

grokking's People

Stargazers

Watchers

Forkers

langosco zablotchi toontran sambroy yanivbl6 hungvm2 chunsj rsn870 ivnle samacqua sytelus kyr-pol paullzn gizemyuce

grokking's Issues

ModDivisonDataset

Why do you have (a * pow(b, self.p-2, self.p)) % self.p for ModDivisonDataset? Shouldn't it be (a / b) % self.p?

Default p is 96 instead of 97.

The value for p in the train grokk config file is 96, but should probably be 97. As far as I can tell the 96 then overrides the 97 from the dataset config default.

Default weight_decay is 0

According to paper, the weight_decay should be 1. However, your config has it as 0. Is this correct? I am trying to reproduce Figure 1 and I thought that must have been produced with weight_Decay=1 per appendix A.1.2.

Conflicts when installing requirements

When I try to install this repo on my computer, I met some difficulties when installing requirements.txt.
The conflict is caused by:
The user requested numpy
mkl-fft 1.3.1 depends on numpy<1.23.0 and >=1.22.3
torchvision 0.11.1 depends on numpy
mkl-random 1.2.2 depends on numpy<1.25.0 and >=1.24.3

I will appreciate it if you can update your requirements.txt with more simplified package requirements and more detailed version information. Thanks a lot.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.