zchking / s3funnel Goto Github PK

In r44 the LIST op isn't passed config.

What steps will reproduce the problem?

When I give a directory name as an input parameter for PUT operation, the boto 
module crash on 
open() function with error that directory cannot be read.

What is the expected output? What do you see instead?

I would expect that s3funnel will do the walk in the directory tree and upload 
all the files in that 
tree instead...

What version of the product are you using? On what operating system?

0.5.2 on a linux (Debian 5.0)

Please provide any additional information below.

I am attaching a patch which is adding the functionality I am proposing...

I'd like to generate a file through a Unix pipe, then write it to a tag.
Thanks to your comments about pipe support, I was hoping the following line
would work:

program_that_generates_a_lot_of_data_to_stdout | s3funnel bucket put filename

It does not - it appears to look for "filename" as a file, then not be able
to write it. This makes me sad.

I feel like this should really, really work, or something along these lines.

Works great for batch operations on many objects.

I need to update the ACL of many objects already existing in S3.

Please add support for this.

* GETs should not be retried if the error is a 404.
* File system errors should be caught & logged.

Sometimes it's beneficial to transfer without the ssl overhead.

s3funnel does not appear to provide any convenient way to detect errors.

zorba@queston-build64:~$ s3funnel lolbukkit put test && echo hello
ERROR    test: No such file.
CRITICAL Failed starting workers: S3ResponseError: 404 Not Found
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchBucket</Code><Message>The specified bucket does not
exist</Message><RequestId>989359CC0DB82556</RequestId><BucketName>lolbukkit</Buc
ketName><HostId>xonqowJV+wMXxoshyIsIQBBTb3dTyhAH2ZuXLDvm9gWT3FybsdyallNCz8m4SJ8d
</HostId></Error>
hello

Why does it print "hello"? The operation failed horribly, not only was the
file not found but the bucket didn't even exist! There's simply no way to
tell if the data transmission was successful.

What steps will reproduce the problem?
1. Try to get from a bucket a file under a directory 
2.
3.

What is the expected output? What do you see instead?
I except that it would get the file.

Here is what I see:
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/dist-packages/workerpool-0.9.2-py2.7.egg/workerpool/workers.py", line 61, in run
    job.run(toolbox=self.toolbox)
  File "/usr/local/lib/python2.7/dist-packages/s3funnel-0.5.2-py2.7.egg/s3funnel/jobs.py", line 34, in run
    k.get_contents_to_filename(self.key)
  File "/usr/lib/pymodules/python2.7/boto/s3/key.py", line 677, in get_contents_to_filename
    fp = open(filename, 'wb')
IOError: [Errno 2] No such file or directory: '2011Sep04.encrypted/db12/ei.gpg'


What version of the product are you using? On what operating system?
Last version. Ubuntu.

Please provide any additional information below.

Useful to be able to store http headers (e.g. for caching) with a PUT request.