Hi,

We're experiencing an error with Drone artifact uploads to S3 via S3 sync.

The drone build will randomly fail on certain files with the following error:

ERROR: failed to upload /foo/bar/..../baz.svg to foo/img/login.svg: NoSuchKey: The specified key does not exist.
    status code: 404, request id: 123456
[info] build failed (exit code 1)

There are no discernible patterns in the filename, filetype or filesize that we causes the error, nor is there a discernible pattern in the number of files before the upload fails.

Please let me know if I can provide any extra info to help contextualize this issue, and thanks in advance.

I wanted to document this issue even though I haven't had time to dig into the root cause. I was attempting to publish using the native branch and run into the following error. http://beta.drone.io/drone/blog/1

Uploading "/drone/src/github.com/drone/blog/public/post/drone-named-rookie-of-the-year/index.html" with Content-Type "text/html; charset=utf-8" and permissions "public-read"
ERROR: failed to upload /drone/src/github.com/drone/blog/public/post/drone-discussion-site/index.html to post/drone-discussion-site/index.html: : 
    status code: 404, request id: 
[info] build failed (exit code 1)

I switched to the master branch version and the sync executed without error. I'm guessing there is a minor step we're missing in the upload process, but not sure. I'll post back a root cause once I've had the chance to research.

There was an exclude functionality; but it does not appear to exist and is not documented.

It would be good if we could exclude paths like .git

Sync to S3: Done without problems (all files was changed)
Invalidation CloudFront: Error 403 - authorization problem.

AWS user (Key+Secret) has all neeeded permissions.

ERROR: failed to invalidateCloudFront to /*: IncompleteSignature: 'AKIAT7AGGEXXXXXXXXXX/20191010/us-east-1/cloudfront/aws4_request' not a valid key=value pair (missing equal-sign) in Authorization header: 'AWS4-HMAC-SHA256 Credential=********/20191010/us-east-1/cloudfront/aws4_request, SignedHeaders=content-length;host;x-amz-date, Signature=c35c46c9d2f6c5aaa0b7357ec0406c823e7cbc1181624e4f3817************'. status code: 403, request id: 66a36506-eb61-11e9-973d-************

After removing white space from credencials Invalidations started to working.
It was hard to find where is the problem becouse credensials are stored in as secrets, but when i opened database and quered for secrets related to repository i find a bug that i have done during copy + paste.

I think that different behavior when authorizing to S3 and to Cloudfront can be fixed using simple Trim().

The documentation shows bucket parameter provided in two different formats (actually twice for each example):

bucket: my-bucket-name
...
bucket: my-bucket.s3-website-us-east-1.amazonaws.com

Which format is the correct one?

The link to the documentation: http://plugins.drone.io/drone-plugins/drone-s3-sync/

If there be an error or exception throw out to interrupt drone stage when anything unexpected happen during data transfer？
eg. some files transfer successful but left failed.

I get

NotImplemented: A header you provided implies functionality that is not implemented - ERROR 501

whenever I try to upload something. Apparently this is a known issue affecting SDK versions 1.6.2 and earlier.

See: aws/aws-sdk-go#1379

The docs here mention the exclude and include parameter but these parameters are not implemented.

Either implement them or remove them from the docs.
The Rsync Plugin has them implemented for example

Maybe just ignoring the fact that the credentials are empty is enough
(Removing https://github.com/drone-plugins/drone-s3-sync/blob/master/plugin.go#L63)

But basically any plugin communicating with aws should have the ability to work with a role

It happens only sometimes that it deletes files that it shouldn't.

For example just today in the logs we had:

Uploading "/drone/src/github.com/matteosuppo/reference/public/en/index.html" with Content-Type "text/html; charset=utf-8" and permissions "public-read"
[...]
Removing remote file "reference/en/index.html"

where the config was

publish:
    s3_sync:
        acl: public-read
        region: "us-east-1"
        bucket: $$S3_BUCKET
        access_key: $$S3_ACCESS
        secret_key: $$S3_SECRET
        source: public
        delete: true
        target: reference
        when:
            branch: master

Default "*" value for paths is better than nothing, but it'd be nice to have a way to manage it.

Looks like a bug. The x-amz-meta-cache-control metadata key is set instead of Cache-Control.

.drone.yml:

...
  s3_sync:
    acl: public-read
    region: $$AWS_REGION
    bucket: $$AWS_BUCKET
    access_key: $$AWS_ACCESS_KEY_ID
    secret_key: $$AWS_SECRET_ACCESS_KEY
    source: build/
    target: /
    delete: true
    metadata:
      "*.js":
        Cache-Control: "public, max-age: 10000000"
      "*.css":
        Cache-Control: "public, max-age: 10000000"
      "*.jpg":
        Cache-Control: "public, max-age: 10000000"
      "*.gif":
        Cache-Control: "public, max-age: 10000000"
      "*.png":
        Cache-Control: "public, max-age: 10000000"
      "*.html":
        Cache-Control: "public, max-age: 600"

I am working on a PR for this right now, this is just to keep tracking of effort.

Feedback is welcome on the changes on my cloudfront branch.

Using Drone image: drone/drone:1.0.0-rc.3

When configuring acl values like so, it does not work:

settings:
  acl:
    "*": public-read

Bug when configuring it as a JSON string, it seems to work:

settings:
  acl: '{ "*": "public-read" }'

This might be more of an issue with drone and it's yaml to env parsing. Let me know if you want me to open it up in a different repo instead.

As per the documentation, we should be able to pass secrets: [ aws_access_key_id, aws_secret_access_key ]. The secrets are set.

This results in the following error:

Must set access_key, secret_key, and bucket

Currently cache headers can be set globally or on a per file type basis. There is however use cases that require configuration per path/filename, where e.g.

cache_control:
  "*.js": "max-age=31536000"
  "entry.js": "no-cache"
  "assets/**/*.*": "max-age=86400"

should result in all *.js files being cached for a year, everything underneath assets/ for a day and entry.js not at all.

Is it possible to set a "content-encryption" like "gzip" in s3_sync?

Take it from https://github.com/drone-plugins/drone-s3-sync/blob/e07c6190e46fd9093deba811cd0a7af2207998c2/DOCS.md

plugin is dependent it seems on using the s3 fqdn rather than just referencing the bucket. Using it with a private s3 bucket behind cloudfront behind an auth edge lambda.

I'm using the plugin to upload a static Nuxt3 build to S3 in website mode.

I cannot serve the site because the .mjs files that landed on S3 do not have mime type attached.

In the browser:
Failed to load module script: Expected a JavaScript module script but the server responded with a MIME type of "". Strict MIME type checking is enforced for module scripts per HTML spec.

i have aws s3 compatible endpoint, provide creadentials and set endpoint in step to my host:
https://s3.selcdn.ru
but drone after try to execute step says:

RequestError: send request failed
--
2 | caused by: Get https://www.s3.selcdn.ru/?prefix=: x509: certificate is valid for *.selcdn.ru, selcdn.ru, not www.s3.selcdn.ru

i'm recheck and i don't add www prefix to host. why it added?

would like to strip folder prefix for upload to s3 similar to the s3 plugin strip_prefix