Comments (5)
dasoto
commented on May 13, 2024
2
I will recommend setting this env variables:
DATA_OPTIMIZER_GLOBAL_RANK
DATA_OPTIMIZER_NUM_WORKERS
DATA_OPTIMIZER_NUM_NODES
Otherwise the StreamDataloader will not be aware of the distribution.
from litdata.
carmocca
commented on May 13, 2024
1
litdata is meant to be used with a regular DataLoader, so there's nothing specific to do on a TPU machine. If you use Fabric or PyTorch Lightning, that will take care of enabling the DistributedSampler or do any required XLA steps, but these are common to all TPU runs, not just those using litdata
from litdata.
tchaton
commented on May 13, 2024
1
Yes, as @dasoto mentioned, I didn't add wiring for TPU env detection. Feel free to contribute support for it if you try litdata on TPUs.
from litdata.
github-actions
commented on May 13, 2024
Hi! thanks for your contribution!, great first issue!
from litdata.
tchaton
commented on May 13, 2024
Hey @miguelalba96,
I haven't tried with TPU. Maybe @carmocca would know more.
from litdata.
Related Issues (20)
- Allow a StreamingDataset to wrap around when running in a CombinedStreamingDataset HOT 1
- Prints inside the worker processes mess up the progress bar HOT 1
- Issue with StreamingDataset when not using all GPUs on host. HOT 6
- Assert when deserializing `no_header_numpy` or `no_header_tensor`. HOT 4
- `litdata.optimize` accidentally deletes files from the local filesystem HOT 2
- GCSFuse mount + Vertex AI custom training jobs support HOT 1
- Compression using the optimize function from litdata HOT 5
- Dataset not created when using `map()` on data structure without file paths inside
- Question: is there a plan to support streaming from GCS? HOT 6
- ValueError: buffer size must be a multiple of element size
- Dataloading is not working when used in litgpt's debug pretraining example HOT 4
- Please add s3 path support to optimize (read and write to s3) HOT 5
- optimize function on multiple machine writing to local pathes
- StreamingDataset support for older PyTorch versions HOT 1
- Progress bar missing with `litdata.StreamingDataset` and wrong number of steps in an epoch HOT 4
- Slow Dataset Preprocessing due to CPU affinity (?) issues HOT 3
- Time per sample grows as processed samples grows HOT 4
- Optimizing dictionary data structures fails when using a partially initialized function HOT 2
- Cache directory resolution issues in Google Colab HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from litdata.