[RFC] Wrapper/Proxy for Stream about data HOT 7 CLOSED

@NivekT

IIUC, HTTPRespond and GDriveRespond should also be wrapped by StreamWrapper

Yea, I believe they should, and they should still work as intended given that StreamWrapper delegates the method to the underlying object.

from data.

Another thing I want to mention is unified API would help us to implement downstream DataPipe:

• Downloader
• Extractor

These DataPipes are going to take file handle (stream) as input and yield either a filename or a new file stream. A unified API would help us to implement function to read data from streams.

from data.

You are right with Saver. I am talking about reading data from file stream. The current workflow is:

urls = IterableWrapper([URL])
fd = urls.open_url() # File handles
data = fd.map(fn=lambda x: x.read(), input_col=1) # <- This one is referred as Downloader in my mind
file = data.save_to_disk()

We can let user to do any map function to download data from file handle. But, if we are going to implement a DataPipe to do the same thing, we need to make sure all file handles (streams) sent to this DataPipe can be read.

The reason that I prefer a read method is the stream type varies:

• String: ".join(fd)"
• Bytes: b"".join(fd)

from data.

In the case of Extractor or Decompressor the yielded file handle should also be wrapped by the Wrapper to make sure fd is closed automatically and have a unified API like read

from data.

@NivekT

IIUC, HTTPRespond and GDriveRespond should also be wrapped by StreamWrapper

from data.

Another thing I want to mention is unified API would help us to implement downstream DataPipe:

• Downloader
• Extractor

These DataPipes are going to take file handle (stream) as input and yield either a filename or a new file stream. A unified API would help us to implement function to read data from streams.

Doesn't Saver already take in a stream (IOBase) as input and yield a filename? How will Downloader be different?
Maybe Downloader can be the combination of HttpReader and Saver (URL -> Filename)?

For Extractor, will it be similar to what Decompressor does in TorchVision?

https://github.com/pytorch/vision/blob/7b1b68d7142aa2070a5592d7c1a3bff3485b5ec1/torchvision/prototype/datasets/utils/_internal.py#L243

from data.

Closing this Issue as the Wrapper has landed.

from data.