Original author: [email protected] (September 19, 2010 17:40:49)

What steps will reproduce the problem?

Problem 1: Subdomain & ordinary storage calling:

1. Run storage w/o DNS subdomain resolution
2. Try to create new bucket

Problem 2: -h option

1. Try run gsutil -h "Content-Length: 1000" cp file gs://bucket/file

gsutil will substitute 'Content-Length' with fsize( file ) which can be really equal to zero.

it needs when 'file' is really a stream and we know the size of this stream. In this case gsutil can't correctly determine the size and we should specify it by hands.

What version of the product are you using? On what operating system?
gsutil version 2010-09-14 18:56:45-07:00, config file version 2010-09-11 12:28:22-07:00

Please provide any additional information below.
Now we use attached patch on current gsutil version. I think it would be helpful. Bug with -h option is already fixed in this patch.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=37

Original author: [email protected] (June 25, 2010 18:30:56)

What steps will reproduce the problem?

1. gsutil -h "x-goog-meta-name:Mike" cp foo gs://mfsbucket/foo
2. gsutil ls -l gs://mfsbucket/foo

What is the expected output? What do you see instead?

The output should show the metadata field.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=15

Original author: [email protected] (October 03, 2010 18:52:29)

Example:
% gsutil mv ctmp/* gs://c_m
InvalidUriError: Invalid bucket name in URI "gs://c_m".

This should have matched gs://codeontheroad.com

Original issue: http://code.google.com/p/gsutil/issues/detail?id=40

Original author: [email protected] (August 19, 2010 18:41:56)

Use gsutil to access the storage & bandwidth usage will be useful.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=28

Original author: [email protected] (September 25, 2010 00:31:12)

If you do a command like this:

gsutil cp gs://mybucket/obj1 s3://mys3bucket/obj2

the bytes of the object get copied through an intermediate temp file, but any metadata set on the object (Content-Type, Content-Encoding, user-specified x-goog-meta tags, etc.) don't get copied over. This happens because the gsutil cross-provider copy code only copies the bytes over. To fix this that code would need to first get the metadata from the old object, then download to the temp file, the upload that temp file to an object with the given metadata set.

This problem also applies to the gsutil 'mv' command (since it is really just a copy followed by a delete)

Original issue: http://code.google.com/p/gsutil/issues/detail?id=38

Original author: [email protected] (July 10, 2010 08:15:50)

I can't seem to set the Expires header.

~$ gsutil -h "Expires: Fri, 06 Jul 2012 00:00:00 GMT" cp styles.css gs://mybucket
~$ gsutil setacl public-read gs://mybucket/styles.css
~$ curl --head http://commondatastorage.googleapis.com/mybucket/styles.css
HTTP/1.1 200 OK
Expires: Sat, 10 Jul 2010 09:12:24 GMT
Date: Sat, 10 Jul 2010 08:12:24 GMT
Cache-Control: public, max-age=3600
ETag: "9523e4644a3b43461cef552988799c44"
Content-Type: text/css
Last-Modified: Sat, 10 Jul 2010 08:11:59 GMT
Content-Length: 8641
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE

Original issue: http://code.google.com/p/gsutil/issues/detail?id=21

Original author: [email protected] (September 12, 2010 01:56:00)

Be able to download/upload multiple files at the same time to improve speed (like FTP clients do).

Original issue: http://code.google.com/p/gsutil/issues/detail?id=35

Original author: [email protected] (August 18, 2010 15:45:13)

Zip files are easier for Windows users.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=27

Original author: [email protected] (October 15, 2010 16:03:06)

This enhancement was suggested by a question posted on [email protected], included below. This would be useful, for example allowing users to rename "subdirectories" on the cloud server.

from Kevin Postal <[email protected]>
reply-to [email protected]
to gsutil-discuss <[email protected]>
date Thu, Oct 14, 2010 at 5:18 PM
subject Uploading files and directories to folders
mailing list <gsutil-discuss.googlegroups.com>

I was wondering if there was a way to upload Full folders to folder
buckets?

For instance I know I can do:

gsutil cp file/file.txt gs://my_bucket/file/file.txt
gsutil cp file/file.txt gs://my_bucket/file/

Why can't I do?

gsutil cp file/* gs://my_bucket/file/*

or

gsutil cp file/ gs://my_bucket/file/

Original issue: http://code.google.com/p/gsutil/issues/detail?id=45

Original author: [email protected] (September 07, 2010 19:29:34)

copy a file using "gsutil cp filedir/filename gs://bucketname" creates the entire path in the bucket

a number of customers asked to make this command behave like UNIX cp, namely:

gsutil cp dir1/dir2/dir3/filename.txt gs://bucketName
--> copy just the file to the bucket

gsutil cp -r dir1/dir2 gs://bucketName
--> copy dir2 and anything underneath to bucketName

Original issue: http://code.google.com/p/gsutil/issues/detail?id=32

Original author: [email protected] (October 14, 2010 22:09:09)

What steps will reproduce the problem?
$ unset HOME
$ ls -l .boto # it exists
$ gsutil ls
You have no boto config file. This script will create one at
./.boto
containing your credentials, based on your responses to the following questions.

What is the expected output? What do you see instead?
It should find the ./.boto which is there. The above output was
due to it looking for HOME in os.environ only.

What version of the product are you using? On what operating system?

2010-09-20 13:11:11-07:00

linux (ubuntu)

Please provide any additional information below.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=44

Original author: [email protected] (December 15, 2010 08:08:05)

What steps will reproduce the problem?

1. Upload an object with unicode name, like "你.pdf" to somebucket
2. List bucket 'gsutil ls -l gs://somebucket/*

What is the expected output? What do you see instead?
There should be something like:
'224 2010-12-14T18:11:06 gs://somebucket/你.pdf'.
While there is an error:
'Failure: 'ascii' codec can't decode byte 0xef in position 44: ordinal not in range(128).'

What version of the product are you using? On what operating system?
I'm using latest svn gsutil with Python 2.6.6 on Ubuntu 10.10

Please provide any additional information below.
I found that in gslib/command.py, timestamp is unicode while UriStrFor(iterated_uri, obj) is ascii, so I propose a patch to convert timestamp to ascii, please check it out, thanks.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=50

Original author: [email protected] (September 03, 2010 17:51:49)

What steps will reproduce the problem?

1. gsutil cp file gs://mybucket/中文測試
2. gsutil ls -l gs://mybucket/中文測試

This produces the following:

Failure: 'ascii' codec can't decode byte 0xe4 in position 0: ordinal not in range(128).

Original issue: http://code.google.com/p/gsutil/issues/detail?id=31

Original author: [email protected] (May 24, 2010 02:08:10)

As a committer to boto i am pretty excited to see that you found it useful enough to leverage to
build your own utility to Google Storage with.

I am curious however, if there are plans to undo the forked code. I am sure it would be a great
addition to the project (haven't done a diff yet to see what the actual changes were).

Thanks!

Patrick

Original issue: http://code.google.com/p/gsutil/issues/detail?id=2

Original author: [email protected] (June 21, 2010 16:00:17)

What steps will reproduce the problem?

1. $ gsutil cat gs://pub/rose.txt

What is the expected output?
Part of a soliloquy from "Romeo and Juliet"

What do you see instead?
/Users/james/dev/gsutil/gsutil:1154: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
if e.message.find('aws_secret_access_key') != -1:
/Users/james/dev/gsutil/gsutil:1158: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
OutputAndExit(e.message)
'NoneType' object has no attribute 'get_contents_as_string'

What version of the product are you using?
gsutil VERSION file contents:
2010-06-18 17:01:04-07:00

Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin

On what operating system?
Mac OS X 10.6.4

Please provide any additional information below.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=11

Original author: [email protected] (July 02, 2010 16:32:37)

What steps will reproduce the problem?
% gsutil getacl s3://mfsbucket/obj > acl.xml
% gsutil setacl acl.xml s3://mfsbucket/obj

Expected output is empty (success); instead we get:

S3ResponseError:: status=400, code=MalformedACLError, reason=Bad Request.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=20

Original author: [email protected] (September 13, 2010 23:23:32)

example of each:

% gsutil ls -l gs://mfsbucket
2190 2010-09-13T22:11:01 gs://mfsbucket/README

gsutil ls -l gs://mfsbucket/README
2190 Mon, 13 Sep 2010 22 gs://mfsbucket/README

(note that the first is implicitly a wildcard request)

Original issue: http://code.google.com/p/gsutil/issues/detail?id=36

Original author: [email protected] (November 06, 2010 22:55:35)

To repro:

• create object with some ACL's.
• gsutil cp -a public-read gs://x gs://y
  Stubby GetObject on y to see that it has default ACL's.

What is the expected output? What do you see instead?

Please use labels and text to provide additional information.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=47

Original author: [email protected] (October 27, 2010 19:20:11)

If you upload or download a large file it will show status updates every MB until around 1 GB, at which point it just shows "1GB/5GB" until it gets to "2GB/5GB" etc.

It would be better to continue showing status every 1 MB.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=46

Original author: [email protected] (June 18, 2010 18:04:52)

What steps will reproduce the problem?
1.gsutil -d setacl acl.xml gs://chanezon/scan0609.jpg
with an acl.xml file that is malformed
generates a cryptic message saying: Failure: <unknown>:10:16: not well-formed (invalid token).

What is the expected output? What do you see instead?
Specify that the input file was not well formed (how do I know it's not the response)
Maybe provide the stacktrace from the xml parser, to help spot where the issue is.

What version of the product are you using? On what operating system?
eh eh, gsutil has no versioning system yet:-)
Implement my other -v issue and I will tell you.
On Mac OS X.

Please provide any additional information below.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=9

Original author: [email protected] (May 21, 2010 01:27:15)

All I want to do is check flags for using gsutil.

spindle:gsutil polleyj$ ./gsutil help
You have no boto config file. This script will create one at
/Users/polleyj/.boto
containing your credentials, based on your responses to the following questions.

spindle:gsutil polleyj$ ./gsutil
You have no boto config file. This script will create one at
/Users/polleyj/.boto
containing your credentials, based on your responses to the following questions.

spindle6:gsutil polleyj$ ./gsutil -h
You have no boto config file. This script will create one at
/Users/polleyj/.boto
containing your credentials, based on your responses to the following questions.

I should not have to provide any credentials just to get information about the flags the script
accepts.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=1

Original author: [email protected] (September 09, 2010 16:03:52)

Starting in revision 56, gsutil has command code separated into a set of methods inside the Command class. It would be better to make each command a subclass, and also to move header and debug params currently passed to all commands into the constructor.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=33

Original author: [email protected] (August 11, 2010 20:31:41)

What steps will reproduce the problem?

1. upload an object to some bucket, resulting in, say, gs://bucket/object.txt
2. gsutil setacl bucket-owner-read gs://bucket/object.txt

This results in "Failure: ."

Original issue: http://code.google.com/p/gsutil/issues/detail?id=25

Original author: [email protected] (May 25, 2010 21:08:56)

If a user enters an unrecognized flag this results in an uncaught exception
and a stack trace being displayed. I tried gsutil --help and the following
output was dumped to the screen:

gsutil --help
Traceback (most recent call last):
File "/home/mmeade/gsutil/gsutil", line 973, in <module>
main()
File "/home/mmeade/gsutil/gsutil", line 919, in main
command = args[0]
IndexError: list index out of range

Unrecognized command line params should be caught and display the help text
to tell the users what args are valid.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=3

Original author: [email protected] (September 30, 2010 02:18:10)

Some tests fail (false positive failures) because they don't correctly check for object creation: they do bucket listings to do the checks, but since bucket listing results have eventual consistency guarantees it's not guaranteed that checking the listing immediately after creating an object will show that object. Instead the tests with this problem should use HEAD operations on the expected objects to test for their presence.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=39

Original author: [email protected] (August 11, 2010 22:40:36)

% gsutil ls -l gs://somebucket/*

is slow because it retrieves object metadata for each object being listed. In many cases users don't need to see the ACL, and all info other than the ACL can be retrieved from the bucket metadata without touching the objects.

Solution: make gsutil ls -l list everything other than the ACL, and add a new -L option to list that plus the ACL info.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=26

Original author: [email protected] (June 25, 2010 15:17:49)

Since one of the allowable bucket ACLs is public-read-write, it is useful to make gsutil allow unsigned PUT requests for such buckets. Several people (myself included) have needed this.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=13

Original author: [email protected] (August 26, 2010 15:57:30)

The simplest implementation would be to check the local file MD5 against the remote etag.

A more powerful implementation would check for files that have been deleted locally and delete from the object store. If this were added gsutil could be used to synchronize local and hosted storage repositories.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=30

Original author: [email protected] (June 25, 2010 17:51:19)

What steps will reproduce the problem?

1. touch abc.txt
2. gsutil cp abc.txt gs://bucket/abc.txt
3. gsutil -d cp -t gs://bucket/abc.txt s3://bucket/abc.txt

What is the expected output? What do you see instead?

Should see Content-Type header of text/plain on the HTTP PUT.
Instead Content-Type header is application/octet-stream

Note that this only happens for cross-provider copies.
The correct behavior is observed for these two cases:

gsutil -d cp -t abc.txt gs://bucket/abc.txt
gsutil -d cp -t gs://bucket/abc.txt gs://bucket/abc.txt

Original issue: http://code.google.com/p/gsutil/issues/detail?id=14

Original author: [email protected] (September 11, 2010 08:03:10)

vomjom@monad:$ gsutil ls -l gs://vomjom/bar/baz
0 Sat, 11 Sep 2010 08 gs://vomjom/bar/baz
TOTAL: 1 objects, 0 bytes (0.0 B)
vomjom@monad:$ gsutil mv gs://vomjom/bar/baz gs://vomjom/
CommandException: Overlapping source and dest URIs not allowed.
vomjom@monad:~$ gsutil cp gs://vomjom/bar/baz gs://vomjom/
CommandException: Overlapping source and dest URIs not allowed.

Expected behavior:
mv and cp to gs://vomjom/baz

Thanks.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=34

Original author: [email protected] (October 04, 2010 00:27:31)

When you do a command like:
% gsutil cp -z 'txt' abc.txt gs://mybucket
it will create an object gs://mybucket/abc.txt, with its content compressed. If you subsequently do:
% gsutil cp gs://mybucket/abc.txt .
it will copy the compressed data back down to the local file abc.txt. Instead it should uncompress the file, since it has content-encoding:gzip and the file has the extension .txt (not .gz or .gzip etc.)

Original issue: http://code.google.com/p/gsutil/issues/detail?id=42

Original author: [email protected] (December 04, 2010 23:40:15)

Traceback (most recent call last):
File "./gsutil", line 63, in ?
from gslib.command import Command
File "/home2/baron/gsutil/gslib/command.py", line 1340
exp_arg_list.extend(str(u) for u in list(

Original issue: http://code.google.com/p/gsutil/issues/detail?id=48

Original author: [email protected] (June 17, 2010 23:35:40)

What steps will reproduce the problem?

1. gsutil -v
   2.
   3.

What is the expected output? What do you see instead?
version of gsutil

What version of the product are you using? On what operating system?
I don't know, and cannot know, that is the issue:-)

Please provide any additional information below.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=8

Original author: [email protected] (June 30, 2010 18:53:19)

What steps will reproduce the problem?

1. gsutil foo gs://bucket
2. gsutil ls -l gs://bucket
   3.

Similar to issue 15, but simpler (not concerned about Meta-data yet)

What is the expected output? What do you see instead?
Expect to see something like unix ls -l for "foo" (mdate, size, etc)
See:
gs://bucket/:
ACL:
Failure: 0.

What version of the product are you using? On what operating system?
Downloaded and installed to Fedora 8 2010-06-30
No gsutil --version and stdout usage does not show version.

Please provide any additional information below.
work email: [email protected]

Original issue: http://code.google.com/p/gsutil/issues/detail?id=18

Original author: [email protected] (July 15, 2010 23:11:39)

What steps will reproduce the problem?

1. using GS Manager create a folder within a folder - e.g., I created a folder 'def' within folder 'abc' in my bucket "mfsbucket". At this point GS Manager has the following object in my bucket: gs://mfsbucket/abc/def_$folder$
2. If you try to do gsutil ls on this object, you get an error:
   % gsutil ls gs://mfsbucket/abc/def_$folder$
   "gs://mfsbucket/abc/def_$" matches no objects.

What is the expected output? What do you see instead?

It should show the object.

Please use labels and text to provide additional information.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=22

Original author: [email protected] (June 02, 2010 05:24:07)

What steps will reproduce the problem?

1. echo foo > file ; gsutil cp file gs://mybucket/file
2. gsutil ls gs://mybucket/file

What is the expected output? What do you see instead?

MIME type is unconditionally application/octet-stream, regardless of file type or user intention to
the contrary. It's a perfectly sane default, but since one can't alter the MIME type of an object
after creation, it makes gsutil less than useful in pushing files for browser consumption.

What version of the product are you using? On what operating system?

2010-05-25 13:24:19-07:00, MacOS X

Please provide any additional information below.

Patch attached to add a -t sub-option to the gsutil cp command permitting the MIME type to be
set.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=7

Original author: [email protected] (May 26, 2010 21:14:30)

Repro: run this:
% gsutil ls s3:

It produces this response:
'FileStorageUri' object has no attribute 'get_all_buckets'

The code for parsing a URI string into a StorageUri needs to be more robust

Original issue: http://code.google.com/p/gsutil/issues/detail?id=6

Original author: [email protected] (June 25, 2010 18:37:38)

gsutil currently retrieves the full bucket list and handles wildcarding by apply a filter after the fact. It would be more efficient to augment this mechanism with the bucket prefix/delimiter support, when dealing with buckets containing many objects.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=16

Original author: [email protected] (October 03, 2010 18:57:17)

Example:

% gsutil mv ctmp/* gs://codeontheroad.com
prints a number of "Copying" status messages, and then just one "deleting" message. Should delete each of the files after copying.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=41

Original author: [email protected] (May 26, 2010 13:46:06)

For example I did a gsutil copy on a large file and ^C part way through, and was left with a
partial file.

The user probably would expect that behavior, but the case I'm more concerned about is if a
transfer times out part way through. Some possibilities:

1. leave as is
2. warn the user when it happened about the partial files
3. upload/download to a temp name and move once transfer completes to the final name

Original issue: http://code.google.com/p/gsutil/issues/detail?id=5

Original author: [email protected] (August 19, 2010 22:33:59)

Currently if you try to run gsutil on Python v3.x it will get syntax errors at print statements, because Python 3.x made print into a function, so the old syntax:
print 'abcd'
doesn't work, and instead you need:
print('abcd')

There are likely other problems beyond just this to make gsutil work on v3.x. Moreover, at present the boto library only works on Python v2.x.

At some point, when boto moves to support Python 3.x, gsutil should as well.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=29

Original author: [email protected] (August 04, 2010 21:09:05)

What steps will reproduce the problem?

1. gsutil mb gs://somebucket
2. gsutil cp abc.txt gs://somebucket/abc.txt^M
   (use control-V control-M for the final character above)
3. gsutil rm gs://somebucket/*

What is the expected output? What do you see instead?
Expect gsutil to delete the object created above. Instead it gets a 404 "Not found" error.

Please use labels and text to provide additional information.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=24

Original author: [email protected] (December 12, 2010 13:26:52)

Supposing you have a bucket and a file named /mybucket/file.tar.gz with public read ACL. When you perform a "gsutil mv gs://mybucket/file.tar.gz gs://mybucket/myfile.tar.gz" the gsutil application outputs:

Copying gs://mybucket/file.tar.gz...
Removing gs://mybucket/file.tar.gz...

The file is renamed. However, the ACLs from the initial file are not transferred. I mean that in the previous case, the new mybucket/mufile.tar.gz is not public readable. I was expecting that the file is renamed, but also that the ACLs are transferred.

The version I am using is 12-03-2010a.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=49

Original author: [email protected] (October 12, 2010 18:48:00)

For example if you do:

gsutil mv "gs://mybucket/obj" "gs://mybucket/obj"

it detects that the object would overwrite itself (and avoids doing the copy) but then runs the rm anyway. It should just abort at that point instead.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=43

Original author: [email protected] (June 21, 2010 17:39:54)

So that installation is done by:

sudo python setup.py install

See code.google.com/p/googlecl for an example.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=12

Original author: [email protected] (May 25, 2010 21:25:49)

I tried to delete multiple empty buckets in a single shot with gsutil rb
gs://jet*, however this returned an error indicating that bucket names could
not contain a wild card.

Shouldn't the rm and rb both be able to use wild cards?

Actual output:
gsutil rb gs://jet*
Failure: Bucket names cannot contain wildcards.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=4

Original author: [email protected] (June 28, 2010 01:08:11)

What steps will reproduce the problem?

1. create a file bigger than about 10MB
2. gsutil cp file gs://nonexistentbucket

This should result in an error that the bucket doesn't exist, but instead gives a 'Broken pipe' error.

This doesn't happen with small objects (e.g., 8k)

Original issue: http://code.google.com/p/gsutil/issues/detail?id=17

Original author: [email protected] (July 25, 2010 15:40:34)

The 'cat' command uses the boto Key.get_contents_as_string() API, which loads the object into RAM. This should be changed to write to a temp file instead, or possibly adding a new method to the Key object to allow streaming the output directly to stdout.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=23

Original author: [email protected] (June 18, 2010 20:04:27)

repro:

% gsutil mb gs://abcdef
% touch file.txt
% gsutil cp file.txt gs://abcdef//file.txt
% gsutil ls gs://abcdef
gs://abcdef/file.txt

The result should have been
gs://abcdef//file.txt

Original issue: http://code.google.com/p/gsutil/issues/detail?id=10

Original author: [email protected] (July 01, 2010 18:07:25)

What steps will reproduce the problem?
1.
2.
3.

What is the expected output? What do you see instead?

Please use labels and text to provide additional information.

Original issue: http://code.google.com/p/gsutil/issues/detail?id=19