Question: is there a plan to support streaming from GCS? about litdata HOT 6 OPEN

Hi! thanks for your contribution!, great first issue!

from litdata.

Hey @dnnspark,

Yes, this is quite simple to add. Simply needs to add the downloader.

from litdata.

Thanks @tchaton, do you have an idea when it's going to land (even very rough estimate)?

from litdata.

Hey @dnnspark,

If you are willing to give it a try, I can look into it this week.

from litdata.

Sorry for the late @tchaton

I'm willing to try! But it's not blocking at the moment, so I will stay tuned about the GCS support (it will be very helpful if you ping on this thread once it's ready).

One thing I notice is that optimize() function assumes the data is stored on local disk (at least in the example). In my case, the raw data is at GCS (because it's too large). Is there a way to transform the data that is stored in the cloud, and save the transformed data to the cloud without having to download the entire data?

from litdata.

Sorry for the late @tchaton

I'm willing to try! But it's not blocking at the moment, so I will stay tuned about the GCS support (it will be very helpful if you ping on this thread once it's ready).

One thing I notice is that optimize() function assumes the data is stored on local disk (at least in the example). In my case, the raw data is at GCS (because it's too large). Is there a way to transform the data that is stored in the cloud, and save the transformed data to the cloud without having to download the entire data?

Yes. that's why this library was built :) But I would need to add GCS support for it ;) I will try to prioritize it.

from litdata.