This is a zero copy / callback way of sending data rather than using READFUNCTION. In theory it should be faster but we may have to massage headers a little.

We currently use ISO646 conventions internally as outlined here:

https://en.wikipedia.org/wiki/C_alternative_tokens

It is my opinion that they are easier to read. But someone using the API who has read the code said that they don't like them. This is to gather opinion as to whether they should stay or go.

Please comment in this ticket your opinion.

When there are no results it should return NULL.

Delete is supposed to happen if copy is successful, but the opposite is currently true.

We are calling curl_easy_cleanup() on every request. This stops curl from reusing existing connections.

We need a way of keeping the curl handle in a thread safe way so that connections are reused.

Debugging some stuff, I noticed that when ms3_get() is given an empty string to fetch, it returns the response to the list-bucket command instead of an error.

If an error occurs we are freeing using curl_easy_cleanup() and then freeing again a bad pointed on ms3_deinit().

I noticed there was a commit recently to fix building on FreeBSD. The issue affects other POSIX OS's as well (I'm on OpenBSD).

The header in question is a POSIX header and should just be included the same as the others. Here is patch to adjust the code as so.

https://comstyle.com/mariadb/libmarias3-marias3.diff

Currently grows every 1MB, for large files a user may get better performance to increase this. We should have a user option to tune this.

At the moment a failure is causing a bad macro error.

It can really be .amazonaws.. The domain can be prefixed in a few ways with the region (such as s3-us-east-1. or s3.us-east-1.) and the postfix can be country specific (China is the main example of this).

In ticket #33 it was decided that we should remove ISO646 tokens from the codebase and stick to more traditional conventions. This ticket is to track that code change

Look at the else-branch this code:

    if (protocol_version == 1)
    {
      snprintf(uri_buffer, MAX_URI_LENGTH - 1, "%s://%s/%s%s", protocol,
               domain,
               bucket,
               object);
    }
    else
    {
      snprintf(uri_buffer, MAX_URI_LENGTH - 1, "%s://%s.%s%s", protocol,
               bucket, domain,
               object);
    }

it will print:

Protocol://Bucket.DomainObject

which is obviously incorrect, domain and bucket are swapped (see the if-branch for an example of correct printout)

t/basic sometimes fails. I think this is due to running get and status before S3 is ready to serve the file.

We hide the error code symbols, these should be in the public header.

The docs and tests will need updating to reflect this.

Turns out the current policy isn't the long term policy, we need to rollback #78 https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/

i want use this project in windows ,but i donnot konw ,how to compile it in windows? i need some help ,think you

A user may attempt to enter an empty string base_domain instead of NULL. We should assume the user meant NULL in this case.

We currently require linkage of end application to libcurl. I don't think this is required any more.

RPM package building is definitely broken (I haven't added the templates). DEB packaging is untested. Need to fix these.

For large object we should have a streaming async read/write API so we don't have to buffer the entire object in RAM. The hooks already exist in the curl API we are using to do this, just a frontend is needed.

ms3_get()'s callback is called every 16KB of data. We call a realloc every time. This eats up a large amount of CPU. We should create a more intelligent growing buffer.

Google's implementation of S3 API uses key lengths which are out of spec. This causes auth to fail. We need to support variable key lengths.

Needed to support third party S3 implementations.

Need API calls to switch for HTTP and to remove SSL verification.

S3 paginates on 1000 entries, we trim out directory entries so you will get under 1000 results. ms3_list() needs to understand pagination.

"Buckets created in Regions launched after March 20, 2019 are not reachable via the https://bucket.s3.amazonaws.com naming scheme."

I don't think other providers support the DNS method so consider dropping it completely.

The code did at one byte allocate an extra byte but it seems to have been removed somewhere along the way.

Need to come up with a better name / alias. Could maybe reuse ms3_init() now that it is deprecated and removed?

libcurl has the following function:

https://curl.haxx.se/libcurl/c/curl_global_init_mem.html

It allows for a custom allocator to be used. It can be used for instrumentation and a bunch of other things. We should allow for this too in a ms3_library_init() related call and hook into libcurl's implementation.

Removing C99 means we also need to remove bool usage

Curl build with OpenSSL needs some thread locking callback sets (GnuTLS also affected). OpenSSL 1.1 is not affected. If these are not set then crashes in OpenSSL can happen when many threads are using libMariaS3 simultaneously.

Example code to be implemented:

https://curl.haxx.se/libcurl/c/threaded-ssl.html

The problem is going to be detection at build time as to what curl was linked to (can't use curl-config --ssl-backends as it doesn't exists in Debian Jessie). I may make this a manual flag for now.

Should be size * nemb.

To free after get requests. This is due to things such as custom allocators.

The problem is twofold:

1. Some parts of the test suite don't show what the return error code is when an error occurs.

2. We need a way of gaining the error message from S3.

The 3.1.2 tag doesn't build as it uses non-C90 code:

src/xml.c: In function ‘xml_string_equals’:
src/xml.c:207:9: error: ISO C90 forbids mixed declarations and code [-Werror=declaration-after-statement]
  207 |         size_t i = 0; for (; i < a->length; ++i) {
      |         ^~~~~~
src/xml.c: In function ‘xml_string_clone’:
src/xml.c:226:9: error: ISO C90 forbids mixed declarations and code [-Werror=declaration-after-statement]
  226 |         uint8_t* clone = ms3_ccalloc(s->length + 1, sizeof(uint8_t));
      |         ^~~~~~~
src/xml.c: In function ‘xml_node_free’:
src/xml.c:277:9: error: ISO C90 forbids mixed declarations and code [-Werror=declaration-after-statement]
  277 |         struct xml_attribute** at = node->attributes;
      |         ^~~~~~
src/xml.c:284:9: error: ISO C90 forbids mixed declarations and code [-Werror=declaration-after-statement]
  284 |         struct xml_node** it = node->children;
      |         ^~~~~~
src/xml.c: In function ‘xml_parser_error’:
src/xml.c:321:31: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  321 |         #define max(X,Y) ((X) > (Y) ? (X) : (Y))
      |                               ^
src/xml.c:322:28: note: in expansion of macro ‘max’
  322 |         size_t character = max(0, min(parser->length, parser->position + offset));

This problem is fixed by commit 8727cf7 that right after the tag.

Please create a new tag with a commit that works when the documented build instructions are used.