Migrate Away from Knox to Official AWS Node SDK about grunt-s3 HOT 15 OPEN

If this solves the ECONNRESET issues with Knox then I'm all for it. I've got a build at the moment that no longer works because the S3 push fails on various file uploads 100% of the time.

from grunt-s3.

As a note here - I am more than willing to contribute to this effort, and I may even start working on it in my fork later this week once I finish a few other elements.

from grunt-s3.

@davidtucker I've got a deadline in about 22 hours and this is completely blocking me, so I'm about to do a quick hack on the S3 upload function to switch it across to the aws-sdk. Hopefully it solves the issue. If it does I'll note it here, but it will be a hack and therefore not really suitable for merging back to the main project.

from grunt-s3.

I did end up doing this work. The aws-sdk codebase is significantly more stable than knox. I haven't had a single failure pushing to S3 with it (where knox consistently failed), although it is much slower as it is pushing a single asset at a time.

Easily fixed by doing HEAD requests and checking the ETag, which should be done anyway to avoid uploading the same content every time.

from grunt-s3.

I've got an implementation for this half-done; download and upload work in my fork. Is this something that people are interested in? Happy to send a pull request when it's more complete if this is something the project maintainer(s) want.

from grunt-s3.

I'd like to see this happen. The Knox implementation is completely flakey. I'm now a week into using the AWS-SDK version and it hasn't had a single issue.

from grunt-s3.

my fork seems to work for downloads, uploads, copies, and deletions. 2 questions before I can submit a pull:

1. What's the deal with the snyc function? I see it in code but it's not in the docs. Is this a planned but incomplete/unready feature?
2. I can't seem to get my tests to run. Does anyone have instructions on how to run the unit tests and pass in the necessary credentials, etc?

from grunt-s3.

@mreinstein the tests are currently broken. Getting fakes3 to work with TravisCI has been a bit of a challenge.

The sync functionality is recently added but undocumented.

from grunt-s3.

@pifantastic can I run the tests without travis? I'm wondering if there's a way I could put in credentials for an s3 account and point it at a test bucket. Just something to verify that I'm not breaking the shit out of existing functionality. :)

What should the configuration block in the gruntfile look like when using the sync feature?

from grunt-s3.

Here's what I've tried:

sync: [
   {
      src: 'index.html',
      dest: 'index.html'
   }
]

It doesn't seem to execute the sync call though.

from grunt-s3.

Same sync issue mentioned in #82

from grunt-s3.

@jchild3rs yeah I'm happy to add documentation but it's not clear to me how the sync feature works, or if @pifantastic plans to merge this. I can't seem to get this sync feature to work. Would love some feedback, happy to follow up and get this finalized.

from grunt-s3.

I'll comment here on the sync features. Since I contributed it, I'll update the docs too.
Basics are this: default behavior is to not upload the file if it already exists, only upload new files. Adding the flag 'verify' forces a hash check, if that's the same, modified time is checked. If local file is newer, upload it.

options: {
            key: cfg.Ss.s3.key,
            secret: cfg.Ss.s3.secret,
            bucket: cfg.Ss.s3.bucket,
            access: 'public-read',
            headers: {
                // Two Year cache policy (1000 * 60 * 60 * 24 * 730)
                "Cache-Control": "max-age=630720000, public",
                "Expires": new Date(Date.now() + 63072000000).toUTCString()
            }
        },
sync: [
   {
      verify: true, // checks md5 hash + mtime before overriding, default only uploads if new
      src: 'index.html',
      dest: 'index.html',
      options: { gzip: true }
   }
]

from grunt-s3.

@geedew How come the sync function doesn't also download files recursively from within folders? It seems "sync" is a very unclear name for the functionality you are trying to achieve here - in reality, all your "sync" is doing is uploaded and also providing the capacity to verify changes, no?

from grunt-s3.

The problem was that looking up folders and files is very expensive and
rate limited. Prior to an official package from Amazon, you would have to
request all files in a bucket... which in many cases is quite large. and
even that was limited to the first few hundred it would give. Second,
automating the removal of files from what is logically a useful CDN source
for cloud front, is an anti pattern of web caching of files. It was simply
not safe for me to implement, nor feasible. But the door was definitely
left open to extend it. (hint, hint)
On Feb 25, 2014 9:31 PM, "Keith Rosenberg" [email protected] wrote:

@geedew https://github.com/geedew How come the sync function doesn't
also download files recursively from within folders? It seems "sync" is a
very unclear name for the functionality you are trying to achieve here - in
reality, all your "sync" is doing is uploaded and also providing the
capacity to verify changes, no?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/83#issuecomment-36087420
.

from grunt-s3.