Giter Club home page Giter Club logo

Comments (4)

tmbdev avatar tmbdev commented on June 14, 2024

The ".tif" extension is simply not in the list of recognized extensions. TIFF file decoding tends to be pretty slow, so it's generally not recommended in high performance code, however.

The .decode() method is just a convenience method, a kind of special form of .map(). If you want a decoder for TIFF files, you should be able to write:

def tiff_decode(key, data):
    if not key.endswith(".tif"):
        return None
    result = ... decode data ...
    return result

...
dataset = dataset.decode(tiff_decode, pil)
...

You can always just decode whatever you like with a .map(), which receives the whole sample:

def completely_custom_decoder(sample):
    sample = dict(sample)
    sample["tif"] = tiff_decoder(sample["tif"])
    return sample

...
dataset = dataset.map(completely_custom_decoder)
...

I'm also making the list of extensions for ImageDecoder configurable, so that you can add other extensions to be handed to PIL. It's in the Github version now. You should be able to say:

dataset = dataset.decode(ImageDecoder(extensions="png tif jpg".split())

from webdataset.

cwerner avatar cwerner commented on June 14, 2024

Thanks for the clarification and quick improvement!

from webdataset.

cwerner avatar cwerner commented on June 14, 2024

Quick followup if you don't mind:

Whenever I try to define a custom decode method I get the error:

  File "/opt/conda/lib/python3.8/site-packages/webdataset/iterators.py", line 228, in map
    result = f(sample)
  File "/opt/conda/lib/python3.8/site-packages/webdataset/autodecode.py", line 332, in __call__
    return self.decode(sample)
  File "/opt/conda/lib/python3.8/site-packages/webdataset/autodecode.py", line 327, in decode
    result[k] = self.decode1(k, v)
  File "/opt/conda/lib/python3.8/site-packages/webdataset/autodecode.py", line 307, in decode1
    result = f(key, data)
  File "/opt/conda/lib/python3.8/site-packages/webdataset/autodecode.py", line 133, in g
    return f(data)
TypeError: __call__() missing 1 required positional argument: 'data'

I now have a "rgb.png" and "msk.png" key in my sample (also txt but this should be ignored)...?

I call it like this:

[...]

wds.WebDataset(train, length=int(1e9))
    .decode(
        wds.handle_extension("rgb.png", wds.imagehandler("torchrgb")),
        wds.handle_extension("msk.png", wds.imagehandler("torchl")),
    )
    .rename(image="rgb.png", mask="msk.png")
    .to_tuple("image", "mask")

from webdataset.

cwerner avatar cwerner commented on June 14, 2024

Ok. I think I'm getting the hang of it now! Please disregard the last comments...

Ans thanks again for providing this package to the public! Hopefully this work gains some traction and community support 👍

from webdataset.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.