Document basic_streaming vs basic_external about giftless HOT 6 CLOSED

I did not implement the GCP support, but to answer the question - no, it is not expected. The expected behavior from basic_external implementations is that they provide an external URL (in this case a signed GCP URL) to send the data to via PUT. If you are getting an error instead, please provide it.

Also since this has the "question" tag I will try to clarify the architecture a bit, so this can later go into documentaiton: basic_streaming and basic_external are two transfer adapters. Each one of them can use different storage backend adapters, and sometimes they can both use the same backend adapter class because that backend supports both "streaming" and "direct" access. I will try to clarify what that means:

basic_streaming is suitable for setups where we want the Giftless server itself to also serve files (or more correctly put, "streams of data"). When enabled, it exposes 3 new endpoints in the Giftless HTTP server to upload, download and verify files. It then delegates all the work of actually working with "files" to the selected storage backend object, which must implement the StreamingStorage interface. This can be anything from local file system (implemented by the LocalStoragebackend), Azure, GCP or AWS storage or even anything else, as long as the upload / download / verify requests are streamed through Giftless itself. This adapter has an upside of simplicity, but a downside that resource-intensive upload / download requests are handled by Giftless itself and not by a highly-scalable backend.

basic_external uses storage backends that implement the ExternalStorage interface and is used for setups where Giftless only handles batch requests and verify requests, and leaves resource intensive upload / download requests to be handled by a separate HTTP service - typically, a cloud storage provider (S3, Azure Blobs, GCP Cloud Storage etc.). When asked for an upload URL, ExternalStorage adapters will provide a signed URL, and based on the LFS basic transfer adapter will send a PUT request to that URL with the data. So light-weight "control" requests are sent to Giftless, and intensive "data" requests are send to an external service.

In reality, some backend adapters can implement both ExternalStorage and StreamingStorage and be used by both basic_* transfer adapters. This is true in the case of the Azure backend, and should probably be true for other cloud backends as well.

BTW the reason they are both called basic_* is because both of them implement the same transfer adapter protocol defined by the LFS spec: the "basic" transfer protocol.

from giftless.

Thanks for the clarification! I think part of this comment needs to go to the README; I probably got an error for basic_external for not using it properly. It would be good to re-test it on gcp_integration.py and double-check if it was misuse or if there's a bug on the implementation

from giftless.

Let's split this into two issues:

1. Add more background on the different transfer adapters to the docs - copy from the comments above
2. Let's check that the GCP implementation works properly with basic_external and if not create separate issue(s).

from giftless.

There is no bug with GCP basic_streaming upload.

from giftless.

basic_external is not working. Creating a bug - #34

from giftless.

I'm closing this as the docs have been updated and the other issue is reported in #34.

from giftless.