Comments (3)
haileyschoelkopf
commented on August 17, 2024
1
Yup! I believe all but 1.3B are now corrected, though I need to do another pass through them all and delete bad ones. Should have those soon, the cluster has just been giving me trouble with freezes + not accepting my jobs for some reason.
Also todo:
- Create
.featherfile + instructions to use for plots with Igor's code - Rename
lambadatask tolambada_openaieverywhere?
from pythia.
StellaAthena
commented on August 17, 2024
Crossposting from the Discord for transparency: we found a bug in our code introduced by a new feature we added between the training and evaluation of the models. It has been corrected but many, if not all, of the evaluations need to be rerun.
from pythia.
haileyschoelkopf
commented on August 17, 2024
All evals in this repo should be correct now, as far as I know!
from pythia.
Related Issues (20)
- Has the data been shuffled? HOT 2
- Model Initialization Question HOT 1
- The value of weight decay HOT 1
- Inconsistent init methods of pythia-6.9b model HOT 2
- Replicating the Training Data Order HOT 1
- Deduplicated Pile dataset with Domain Attribution
- Pytia or GPT-neox? HOT 1
- Wrong files in eval?
- . HOT 2
- Missing / undownloadable checkpoints on huggingface HOT 3
- Details about "EleutherAI/pythia-160m-seed*" models HOT 3
- [Pythia on Pile-Dedup] Training for ~1.5 epochs: how to identify the repeated sequences (i.e., the additional .5 epoch)? HOT 3
- Would it be possible to share training loss curves on the original Pythia models? HOT 4
- "gas" configuration doesn't do anything
- Is there existing code to resume training from specific checkpoint? HOT 1
- Weird inconsistency in Tokenizer vocabulary HOT 1
- Optimizer states in HF format HOT 1
- Provide the shuffled index_mapping npy files for ease of reproducing training data
- instruct-tuned pythia
- tokenizer.pad_token HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pythia.