Ceph and non-S3 S3-compliant storage systems about distribution HOT 16 CLOSED

Huh, I never knew that Ceph also supported the swift API. If the S3 spec is incompatible with Ceph then yes, I could probably get to writing up a driver for v2.

from distribution.

Yeah, I don't know much about it (http://ceph.com/docs/next/radosgw/swift/). Needs to be looked into.
And yes, a swift driver would be swell :)

from distribution.

Hi, just curious, what's S3 features are needed that Ceph does not have?

from distribution.

Invoking @AndreyKostov for the gory details.

from distribution.

It doesn't support the Upload Part Copy api that we use for WriteStream in the s3 driver

http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html

from distribution.

Support of the Ceph Rados Gateway would be nice but I would prefer a pure Rados storage backend.

from distribution.

Can't s3 driver be implemented without Upload Part Copy mechanism?

from distribution.

It looks like that the S3 client in Go doesn't support custom API domains.
https://github.com/AdRoll/goamz/blob/master/s3/s3.go#L853

On 17 Apr 2015, at 10:45, Ian Babrou [email protected] wrote:

Can't s3 driver be implemented without Upload Part Copy mechanism?

—
Reply to this email directly or view it on GitHub.

from distribution.

We can always fork/PR fixes so we can accomplish that, but good to know! I'm currently tied down with some onboarding stuff because we just recently merged with Engine Yard but nobody picks this up then I'd be happy to contribute a driver (assuming that we can make it work)

from distribution.

Hoping this feature returns -- the previous python version supported S3-compatible storage systems by letting you specify the endpoint.
UPDATE: fwiw I don't use Ceph so not worried about its lack of Upload Part Copy, I just miss the ability to specify the S3 endpoint (so it can point to something other than amazonaws.com).

from distribution.

The core of the issue is that the new registry storage driver api requires resumable uploads and the way the s3 driver currently implements that functionality is with the Upload Part Copy mechanism.

If an s3 alternative does support Upload Part Copy, then supporting it is fairly easy, but if it doesn't it requires an s3 driver rewrite (that I am not even sure is possible while adhering to the current storage API)

from distribution.

Ceph does have append support within librados (see
http://ceph.com/docs/master/rados/api/librados/)

There are two possibilities:

1. A Ceph storage driver
2. Add Upload Part Copy to the Ceph S3 API

On Mon, Apr 20, 2015 at 9:50 AM, Andrey Kostov [email protected]
wrote:

The core of the issue is that the new registry storage driver api requires
continuous uploads and the way the s3 driver currently implements that
functionality is with the Upload Part Copy mechanism.

If an s3 alternative does support Upload Part Copy, then supporting it is
fairly easy, but if it doesn't it requires an s3 driver rewrite (that I am
not even sure is possible while adhering to the current storage API)

—
Reply to this email directly or view it on GitHub
#40 (comment).

from distribution.

@AndreyKostov @RichardScothern The better approach might be to have "logical files" in the backend. The main use case is to accept a bunch of offset-fixed chunk writes, then assemble them on "commit" of a blob.

Let's focus on getting resumable upload support fully integrated and then attack this issue. I suspect we need to break from the VFS-style API that we cribbed from V1. It incurs too many requirements on the backend drivers and provides nearly no benefit.

from distribution.

@stevvooe is there a goal to support storages that do not support resumable uploads?

from distribution.

@bobrik That should be a possibility but the current design of the storage drivers doesn't really provide a method for a driver to communicate this. For example, it makes more sense for a storage driver to not implement LayerUpload.Resume rather than have it subtly fail on writing offsets to a file. The former is much more explicit and reliable.

from distribution.

Ceph has been implemented.

from distribution.