Comments (4)
The ".tif" extension is simply not in the list of recognized extensions. TIFF file decoding tends to be pretty slow, so it's generally not recommended in high performance code, however.
The .decode() method is just a convenience method, a kind of special form of .map(). If you want a decoder for TIFF files, you should be able to write:
def tiff_decode(key, data):
if not key.endswith(".tif"):
return None
result = ... decode data ...
return result
...
dataset = dataset.decode(tiff_decode, pil)
...
You can always just decode whatever you like with a .map(), which receives the whole sample:
def completely_custom_decoder(sample):
sample = dict(sample)
sample["tif"] = tiff_decoder(sample["tif"])
return sample
...
dataset = dataset.map(completely_custom_decoder)
...
I'm also making the list of extensions for ImageDecoder configurable, so that you can add other extensions to be handed to PIL. It's in the Github version now. You should be able to say:
dataset = dataset.decode(ImageDecoder(extensions="png tif jpg".split())
from webdataset.
Thanks for the clarification and quick improvement!
from webdataset.
Quick followup if you don't mind:
Whenever I try to define a custom decode method I get the error:
File "/opt/conda/lib/python3.8/site-packages/webdataset/iterators.py", line 228, in map
result = f(sample)
File "/opt/conda/lib/python3.8/site-packages/webdataset/autodecode.py", line 332, in __call__
return self.decode(sample)
File "/opt/conda/lib/python3.8/site-packages/webdataset/autodecode.py", line 327, in decode
result[k] = self.decode1(k, v)
File "/opt/conda/lib/python3.8/site-packages/webdataset/autodecode.py", line 307, in decode1
result = f(key, data)
File "/opt/conda/lib/python3.8/site-packages/webdataset/autodecode.py", line 133, in g
return f(data)
TypeError: __call__() missing 1 required positional argument: 'data'
I now have a "rgb.png" and "msk.png" key in my sample (also txt but this should be ignored)...?
I call it like this:
[...]
wds.WebDataset(train, length=int(1e9))
.decode(
wds.handle_extension("rgb.png", wds.imagehandler("torchrgb")),
wds.handle_extension("msk.png", wds.imagehandler("torchl")),
)
.rename(image="rgb.png", mask="msk.png")
.to_tuple("image", "mask")
from webdataset.
Ok. I think I'm getting the hang of it now! Please disregard the last comments...
Ans thanks again for providing this package to the public! Hopefully this work gains some traction and community support 👍
from webdataset.
Related Issues (20)
- FAQ : What's the meaning of n in `with_epoch(n)` HOT 2
- Distributed Training with videos not working? HOT 1
- [Errno 32] Broken pipe - Download Failed Error with S3 URLs HOT 1
- Webdataset (Liaon115M) + Torchlightning (pl.DataModule) with visualizing progressbar during training HOT 1
- Seed in multiprocessing (DDP) is not fixed in shuffle() HOT 1
- Update pypi with 0.2.88?
- How does shuffling work? HOT 1
- Restricting the number of samples in the dataset HOT 1
- wds.Decoder TypeError: 'functools.partial' object is not iterable HOT 2
- Loop through same tar file 10 times? HOT 1
- Excess memory usage when generating short sequence clips HOT 3
- seed not used in shuffle HOT 1
- Memory leak during training with standard DataLoader coupled with WebDataset dataloader HOT 3
- `pipe:s3cmd` or `pipe:aws s3 cp` ?
- custom batch sampler ?
- PyPI packages not available for 2.90 or 2.88
- ShardListDataset does not work with multiprocessing_context=spawn
- batched augmentations with kornia
- Incorrect Documentation for default_collation_fn
- Validation Set Distributed Sampling when using WebDataset with FSDP
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from webdataset.