basak / glacier-cli Goto Github PK

I'm getting a TypeError whenever I try to upload a file to a vault:

glacier.py archive upload Backup file

where Backup is the vault name and file is a file on my fs.

Full stack trace:

Traceback (most recent call last):
  File "/Users/mark/bin/glacier.py", line 618, in <module>
    App().main()
  File "/Users/mark/bin/glacier.py", line 604, in main
    args.func(args)
  File "/Users/mark/bin/glacier.py", line 416, in archive_upload
    archive_id = vault.create_archive_from_file(file_obj=args.file, description=name)
TypeError: create_archive_from_file() got an unexpected keyword argument 'description'

from shell script, how to check if the file was successfully uploaded to vault or not? does glacier.py returns something which i can check to see if upload was successful or failed? i am using a shell script to schedule backups to glacier. i want to be notified by e-mail if the upload fails somehow.

See http://git-annex.branchable.com/todo/ctrl_c_handling/

Basically:
When I am transfering a large number of files to glacier with git-annex, it'd be nice to ctrl-c and have the transfers killed cleanly (ie: stop after current transfer is completed, or, if it is a huge file, allow subsequent ctrl-c's to kill it). In the end, a ctrl-c would not result in ugly backtraces but instead useful messages ("All pending transfers stopped" or "All active and pending transfers killed", whatever makes sense).

In issue #50 you said "Packaging and documentation could do with improvement". Markus Hubig has added requirements.txt and setup.py in his branch here. His changes permit installation of glacier-cli with pip, which is always a good thing, using that URL. Will you please consider pulling Markus' changes into the master branch?

1. Install the tool
2. Execute glacier vaults list
3. Exception happens

Traceback (most recent call last):
  File "C:/Python37-32/Scripts/glacier", line 161, in <module>
    main()
  File "C:/Python37-32/Scripts/glacier", line 149, in main
    list_vaults(region, access_key, secret_key)
  File "C:/Python37-32/Scripts/glacier", line 98, in list_vaults
    layer2 = connect(region, access_key = access_key, secret_key = secret_key)
  File "C:/Python37-32/Scripts/glacier", line 90, in connect
    debug=debug_level)
  File "C:\python37-32\lib\site-packages\boto\glacier\__init__.py", line 41, in connect_to_region
    return connect('glacier', region_name, connection_cls=Layer2, **kw_params)
  File "C:\python37-32\lib\site-packages\boto\regioninfo.py", line 220, in connect
    return region.connect(**kw_params)
  File "C:\python37-32\lib\site-packages\boto\regioninfo.py", line 290, in connect
    return self.connection_cls(region=self, **kw_params)
  File "C:\python37-32\lib\site-packages\boto\glacier\layer2.py", line 38, in __init__
    self.layer1 = Layer1(*args, **kwargs)
  File "C:\python37-32\lib\site-packages\boto\glacier\layer1.py", line 98, in __init__
    profile_name=profile_name)
  File "C:\python37-32\lib\site-packages\boto\connection.py", line 569, in __init__
    host, config, self.provider, self._required_auth_capability())
  File "C:\python37-32\lib\site-packages\boto\auth.py", line 1021, in get_auth_handler
    'Check your credentials' % (len(names), str(names)))
boto.exception.NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV4Handler'] Check your credentials

Please add to readme part about storing credentials and add proper error handling to the tool

I can see the vault names and the number of objects in each vault. I can list the vault names in whatever region I choose:

./glacier.py --region us-west-1 vault list
testvault-2
testvault-3
testvault-4
testvault-5
testvault-6
testvault-7
testvault1
testvault2
testvault3
vault_test

vault_test has 1 archive in it according to what I uploaded yesterday and viewed in AWS web.

./glacier.py --region us-west-1 archive list vault_test
glacier-cli [master +0 ~0 -0]$

nothing. Am I missing something? I get the same result back for other region vaults and archives.

latest code as of this writing

Hi

I am using your amazing tool to upload files to aws glacier vaults but every time I upload a new file I get a error traceback

Traceback (most recent call last):
File "/usr/local/bin/glacier", line 730, in
File "/usr/local/bin/glacier", line 716, in main
File "/usr/local/bin/glacier", line 498, in archive_upload
File "/root/scripts/glacier-cli/boto/glacier/vault.py", line 141, in create_archive_from_file
File "/root/scripts/glacier-cli/boto/glacier/writer.py", line 152, in write
File "/root/scripts/glacier-cli/boto/glacier/writer.py", line 141, in send_part
File "/root/scripts/glacier-cli/boto/glacier/layer1.py", line 626, in upload_part
File "/root/scripts/glacier-cli/boto/glacier/layer1.py", line 83, in make_request
File "/root/scripts/glacier-cli/boto/connection.py", line 913, in make_request
File "/root/scripts/glacier-cli/boto/connection.py", line 859, in _mexe
socket.gaierror: [Errno -2] Name or service not known

The point is after some time I check vault with glacier cli and it seems all files were uploaded ok but then, what about this messages? Is something I have to worry about?

Thanks!!

I can supply a PR if there's interest.

Hi!

I've used glacier-cli on Linux and its working great. I'm now trying to use glacier-cli on FreeBSD (FreeNAS 9.2) and glacier-cli will correctly list vaults but when I try to list archives in a vault, I get nothing. I wiresharked traffic and I see no web traffic for 'glacier-cli archive list valuename' but I do capture traffic for 'glacier-cli vault list'.

On my FreeNAS system, I have python 2.7.6, lastest glacier-cli, iso8601 0.1.11, and I've tried various versions of boto downloaded from github.

Is there a way to debug boto and/or glacier-cli to see why its not connecting to Glacier AWS when I ask it to query archives?

Thanks,

Bobby

I'm opening a new issue related to #10, done more tests and got a conclusion that when you upload with another tool (like FastGlacier) you can't download the file back using glacier-cli.

But everything works OK when you upload and download using glacier-cli, I think the files can't be related to the tool that was used to upload.

Please give more information on how to debug this for you.

Thank you

Appears to have a limit of 10 vaults or jobs when doing 'vault list' or 'job list' when there are more than 10.

Glacier allows the uploading of multiple archives with the same name (also archives with no name), so the only way to retrieve or delete the dupes is by id. Currently, glacier-cli provides no way to retrieve the ids from the database.

From the command line, we can access them from the local cache DB:

sqlite3 ~/.cache/glacier-cli/db "select name, id from archive"

From the code, the error condition can be reported when the sqlalchemy.orm.exc.MultipleResultsFound exception is thrown. For example, see http://pastebin.com/Y6WybtJH

It'd be very useful to be able to pipe data into glacier-cli to upload it, as well as allow it to download data and output it to stdout. I'd suggest using - as the filename to enable these modes.

One reason I'd like to be able to do this is because git-annex is encrypting files before sending them. Which means it has to write the encrypted file to a temp file right now, which is a waste of disk space and IO.

The other reason I'd like this is that I want to display progress when uploading & downloading, and if I can pipe data, it's easy to update my progress bar as I send/receive data.

Note: Some of this may already work, but I hesitate to use it unless it's documented. Especially because -o- might send a file to stdout, but for all I know, glacier-cli could write other messages to stdout and corrupt the file.

I've been trying to use glacier-cli with git-annex for a while and currently getting this on sync:

move SHA3_512-s78968--58aa331db254fd53cc422b6b8ba22b492cccec00b888497eb0bab8763d371b6232a854e1419190ab61685f5be82fa
be9b614cafd30f79f09044a3daea1291cfa (checking glacier...) Traceback (most recent call last):
  File "/home/annex/bin/glacier", line 737, in <module>
    main()
  File "/home/annex/bin/glacier", line 733, in main
    App().main()
  File "/home/annex/bin/glacier", line 719, in main
    self.args.func()
  File "/home/annex/bin/glacier", line 600, in archive_checkpresent
    self.args.vault, self.args.name)
  File "/home/annex/bin/glacier", line 161, in get_archive_last_seen
    result = self._get_archive_query_by_ref(vault, ref).one()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2404, in one
    "Multiple rows were found for one()")
sqlalchemy.orm.exc.MultipleResultsFound: Multiple rows were found for one()
(user error (glacier ["--region=us-east-1","archive","checkpresent","glacier-c22e72f3-62a4-4c74-afe1-ad946906e3e1","--quiet","SHA3_512-s78968--58aa331db254fd53cc422b6b8ba22b492cccec00b888497eb0bab8763d371b6232a854e1419190ab61685f5be82fabe9b614cafd30f79f09044a3daea1291cfa"] exited 1)) failed

As noted in #30 , boto also provides a glacier command. Worse, this command exits 0 when passed glacier-cli parameters, without uploading anything. boto/boto#2942

This makes it very unsafe for git-annex to run "glacier" and expect it to do anything sane. I think it could best be dealt with by glacier-cli providing a glacier-cli command, which git-annex could then run. (It could continue providing a glacier command too, or not.. no opinion here.)

Not sure if I have my configuration right, but I'm seeing the stack trace below. Not sure if this is relevant, but I'm on a shared server and have installed everything I needed (including boto and sqlalchemy with pip install --user).

[ djoshea@host ~ ]: glacier vault list
Traceback (most recent call last):
  File "/net/home/djoshea/bin/glacier", line 730, in <module>
    App().main()
  File "/net/home/djoshea/bin/glacier", line 708, in __init__
    cache = Cache(get_connection_account(connection))
  File "/net/home/djoshea/bin/glacier", line 124, in __init__
    self.Base.metadata.create_all(self.engine)
  File "/net/home/djoshea/.local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 3687, in create_all
    tables=tables)
  File "/net/home/djoshea/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1856, in _run_visitor
    conn._run_visitor(visitorcallable, element, **kwargs)
  File "/net/home/djoshea/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1481, in _run_visitor
    **kwargs).traverse_single(element)
  File "/net/home/djoshea/.local/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 121, in traverse_single
    return meth(obj, **kw)
  File "/net/home/djoshea/.local/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 709, in visit_metadata
    [t for t in tables if self._can_create_table(t)])
  File "/net/home/djoshea/.local/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 686, in _can_create_table
    table.name, schema=table.schema)
  File "/net/home/djoshea/.local/lib/python2.7/site-packages/sqlalchemy/dialects/sqlite/base.py", line 1127, in has_table
    connection, "table_info", table_name, schema=schema)
  File "/net/home/djoshea/.local/lib/python2.7/site-packages/sqlalchemy/dialects/sqlite/base.py", line 1468, in _get_table_pragma
    cursor = connection.execute(statement)
  File "/net/home/djoshea/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 906, in execute
    return self._execute_text(object, multiparams, params)
  File "/net/home/djoshea/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1054, in _execute_text
    statement, parameters
  File "/net/home/djoshea/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/net/home/djoshea/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "/net/home/djoshea/.local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/net/home/djoshea/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/net/home/djoshea/.local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked [SQL: u'PRAGMA table_info("archive")']

my config looks like this:

[ djoshea@index ~ ]: cat .boto
[Credentials]
aws_access_key_id = ...
aws_secret_access_key = ...

[s3]
calling_format = boto.s3.connection.OrdinaryCallingFormat
host = s3-us-west-1.amazonaws.com

Hi,

I just uploaded 32GB of data to Glacier with your tool (thanks for it), but was afterwards surprised by the 8500 requests this has caused, which cost extra money. This was caused by a multipart size of 8MB in glacier-ci, it seemed. For some reason I had expected glacier to upload my files in one go, unless I explicitly specify a multipart size.

You should mention this isssue more prominently, e.g. in README.md, and possibly change the default to uploading all in one go.

Thanks,
Joachim

Description of problem:
glacier-cli keeps a local cache of vaults and archives. However, the cache does not seem to distinguish between vaults in different AWS regions. So different AWS vaults with the same name in different AWS regions are assumed to be the same vault.

Version-Release number of selected component (if applicable):
glacier-cli.noarch-0-8.20131113gite8a2536.fc20
Originaly reported to Fedora as
https://bugzilla.redhat.com/show_bug.cgi?id=1167186

How reproducible:
Always

Steps to Reproduce:

$ glacier-cli --region=us-east-1 vault create somevault
$ glacier-cli --region=us-east-1 archive upload somevault somearchive

$ glacier-cli --region=eu-west-1 archive list somevault
(note different region specified)

Actual results:
wrongly returns 'somearchive' as an archive in region eu-west-1

Expected results:
No output (there is no vault or archive in region eu-west-1)

Additional information:
Obvious workaround is to pick one region and stick to it, or to use unique vault names across all regions.

hi

i have export aws credentials to environment variables

and now Im getting

boto.glacier.exceptions.UnexpectedHTTPResponseError: Expected (200,), got (403, {"message":"Signature not yet current: 20131120T113529Z is still later than 20131120T104538Z (20131120T104038Z + 5 min.)","code":"InvalidSignatureException","type":"Client"})

do you know what is happening? I have googled a lot without sucess :S

I get this error when I do a vault sync:

$ glacier vault sync **********
Traceback (most recent call last):
  File "/root/bin/glacier", line 730, in <module>
    App().main()
  File "/root/bin/glacier", line 716, in main
    self.args.func()
  File "/root/bin/glacier", line 469, in vault_sync
    wait=self.args.wait)
  File "/root/bin/glacier", line 448, in _vault_sync
    self._vault_sync_reconcile(vault, complete_job, fix=fix)
  File "/root/bin/glacier", line 435, in _vault_sync_reconcile
    fix=fix)
  File "/root/bin/glacier", line 257, in mark_seen_upstream
    key=self.key, vault=vault, id=id).one()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2184, in one
    ret = list(self)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2226, in __iter__
    self.session._autoflush()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1012, in _autoflush
    self.flush()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1583, in flush
    self._flush(objects)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1654, in _flush
    flush_context.execute()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 331, in execute
    rec.execute(self)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 475, in execute
    uow
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 64, in save_obj
    table, insert)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 530, in _emit_insert_statements
    execute(statement, multiparams)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1449, in execute
    params)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context
    context)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context
    context)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 331, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (IntegrityError) column id is not unique u'INSERT INTO archive (id, name, vault, "key", last_seen_upstream, created_here, deleted_here) VALUES (?, ?, ?, ?, ?, ?, ?)' (u'***********', u'***********', '**********', '**********', 1433336619, 1433423949.613394, None)

How do I get an unique column id again?

Running git annex fsck against the Glacier remote requests files from glacier. Would it be possible to have a "light" fsck which just compares the Glacier inventory hash with what is expected? This would be a really nice way to do the occasional check without having to retrieve all of that data.

I uploaded a 6.6 GiB file and after some time I got this error:

Traceback (most recent call last):
  File "/home/amedee/bin/glacier", line 730, in <module>
    App().main()
  File "/home/amedee/bin/glacier", line 716, in main
    self.args.func()
  File "/home/amedee/bin/glacier", line 498, in archive_upload
    file_obj=self.args.file, description=name)
  File "/home/amedee/bin/glacier-cli/boto/glacier/vault.py", line 141, in create_archive_from_file
    writer.write(data)
  File "/home/amedee/bin/glacier-cli/boto/glacier/writer.py", line 152, in write
    self.send_part()
  File "/home/amedee/bin/glacier-cli/boto/glacier/writer.py", line 141, in send_part
    content_range, part)
  File "/home/amedee/bin/glacier-cli/boto/glacier/layer1.py", line 626, in upload_part
    response_headers=response_headers)
  File "/home/amedee/bin/glacier-cli/boto/glacier/layer1.py", line 83, in make_request
    data=data)
  File "/home/amedee/bin/glacier-cli/boto/connection.py", line 913, in make_request
    return self._mexe(http_request, sender, override_num_retries)
  File "/home/amedee/bin/glacier-cli/boto/connection.py", line 859, in _mexe
    raise e
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

I don't know if the file was uploaded correctly, but the time was long enough to expect an upload to complete. The glacier archive list command doesn't show the file.

I have previously used glacier archive retrieve and glacier vault sync on this system, and they do not return an error.

EDIT: I uploaded a 1.2 GiB file and didn't get an error. The smaller file is listed with glacier archive list.

I uploaded the same file 2 times. i got this error when i want to delete.

Traceback (most recent call last):
File "/usr/local/bin/glacier", line 730, in
App().main()
File "/usr/local/bin/glacier", line 716, in main
self.args.func()
File "/usr/local/bin/glacier", line 587, in archive_delete
self.args.vault, self.args.name)
File "/usr/local/bin/glacier", line 145, in get_archive_id
result = self._get_archive_query_by_ref(vault, ref).one()
File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2023, in one
"Multiple rows were found for one()")
sqlalchemy.orm.exc.MultipleResultsFound: Multiple rows were found for one()

Unfortunately this is a blocker for me, rendering git-annex's glacier special remote useless - the whole point of trying it was to be able to upload big binary files.

$ glacier --region eu-west-1 archive upload music myiso.exe
Traceback (most recent call last):
  File "/home/adam/bin/glacier", line 730, in <module>
    App().main()
  File "/home/adam/bin/glacier", line 716, in main
    self.args.func()
  File "/home/adam/bin/glacier", line 498, in archive_upload
    file_obj=self.args.file, description=name)
  File "/usr/lib/python2.7/site-packages/boto/glacier/vault.py", line 177, in create_archive_from_file
    writer.write(data)
  File "/usr/lib/python2.7/site-packages/boto/glacier/writer.py", line 219, in write
    self.partitioner.write(data)
  File "/usr/lib/python2.7/site-packages/boto/glacier/writer.py", line 61, in write
    self._send_part()
  File "/usr/lib/python2.7/site-packages/boto/glacier/writer.py", line 75, in _send_part
    self.send_fn(part)
  File "/usr/lib/python2.7/site-packages/boto/glacier/writer.py", line 222, in _upload_part
    self.uploader.upload_part(self.next_part_index, part_data)
  File "/usr/lib/python2.7/site-packages/boto/glacier/writer.py", line 129, in upload_part
    content_range, part_data)
  File "/usr/lib/python2.7/site-packages/boto/glacier/layer1.py", line 1279, in upload_part
    response_headers=response_headers)
  File "/usr/lib/python2.7/site-packages/boto/glacier/layer1.py", line 114, in make_request
    data=data)
  File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1068, in make_request
    retry_handler=retry_handler)
  File "/usr/lib/python2.7/site-packages/boto/connection.py", line 942, in _mexe
    request.body, request.headers)
  File "/usr/lib64/python2.7/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/usr/lib64/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 827, in _send_output
    msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 12: ordinal not in range(128)

Thanks for the tool. Here's an idea on how to improve it. Will keep using it with or without :)

Glacier can be very expensive when downloading files too fast, but free when spreading retrievals across a month.

Do you have any plans to allow setting an hourly retrieval limit? User would need to calculate this by

total storage * 5% / 720 hours

Hi.
I've upload all my data using FastGlacier and I can't get an archive list using glacier-cli, I'm using the same access key id and access secret.

Everything goes OK when I create a new vault using glacier-cli and upload files using the command line.

What more information can I give to you?

Installation is working correctly for us-west-1 (and others), but not for ca-central-1:

./glacier.py --region ca-central-1 vault list
Traceback (most recent call last):
  File "./glacier.py", line 736, in <module>
    main()
  File "./glacier.py", line 732, in main
    App().main()
  File "./glacier.py", line 710, in __init__
    cache = Cache(get_connection_account(connection))
  File "./glacier.py", line 330, in get_connection_account
    return connection.layer1.aws_access_key_id
AttributeError: 'NoneType' object has no attribute 'layer1'

OS X 10.8.2 does not include the python modules iso8601 and sqlalchemy by default. Consider adding checks for third party modules.

$ ./glacier.py vault list
Traceback (most recent call last):
  File "./glacier.py", line 36, in <module>
    import iso8601
ImportError: No module named iso8601

Mac users can run the following commands to fix these dependencies:

sudo easy_install iso8601
sudo easy_install sqlalchemy

Thank you for the project.

I have been using it in the context of git-annex. My system has python3 installed and that seems to cause problems when reading from sys.stdin as provided by git-annex:

git annex move text_file.txt --to glacier
move text_file.txt (checking glacier...) (to glacier...)
100%  13 B               27 B/s 0sTraceback (most recent call last):
  File "/Users/username/.local/bin/glacier", line 740, in <module>
    main()
  File "/Users/username/.local/bin/glacier", line 736, in main
    App().main()
  File "/Users/username/.local/bin/glacier", line 722, in main
    self.args.func()
  File "/Users/username/.local/bin/glacier", line 504, in archive_upload
    file_obj=self.args.file, description=name)
  File "/Users/username/.pyenv/versions/3.7.3/lib/python3.7/site-packages/boto/glacier/vault.py", line 178, in create_archive_from_file
    writer.close()
  File "/Users/username/.pyenv/versions/3.7.3/lib/python3.7/site-packages/boto/glacier/writer.py", line 228, in close
    self.partitioner.flush()
  File "/Users/username/.pyenv/versions/3.7.3/lib/python3.7/site-packages/boto/glacier/writer.py", line 79, in flush
    self._send_part()
  File "/Users/username/.pyenv/versions/3.7.3/lib/python3.7/site-packages/boto/glacier/writer.py", line 64, in _send_part
    data = b''.join(self._buffer)
TypeError: sequence item 0: expected a bytes-like object, str found

100%  13 B               17 B/s 0sTraceback (most recent call last):
  File "/Users/username/.local/bin/glacier", line 740, in <module>
    main()
  File "/Users/username/.local/bin/glacier", line 736, in main
    App().main()
  File "/Users/username/.local/bin/glacier", line 722, in main
    self.args.func()
  File "/Users/username/.local/bin/glacier", line 504, in archive_upload
    file_obj=self.args.file, description=name)
  File "/Users/username/.pyenv/versions/3.7.3/lib/python3.7/site-packages/boto/glacier/vault.py", line 178, in create_archive_from_file
    writer.close()
  File "/Users/username/.pyenv/versions/3.7.3/lib/python3.7/site-packages/boto/glacier/writer.py", line 228, in close
    self.partitioner.flush()
  File "/Users/username/.pyenv/versions/3.7.3/lib/python3.7/site-packages/boto/glacier/writer.py", line 79, in flush
    self._send_part()
  File "/Users/username/.pyenv/versions/3.7.3/lib/python3.7/site-packages/boto/glacier/writer.py", line 64, in _send_part
    data = b''.join(self._buffer)
TypeError: sequence item 0: expected a bytes-like object, str found
failed
git-annex: move: 1 failed

When reading from sys.stdin.read(), one needs to use sys.stdin.buffer.read() instead. I got it to work with that change.

git annex move text_file.txt --to glacier
move text_file.txt (checking glacier...) (to glacier...)
ok
(recording state in git...)

I will create a pull request with a patch.

Hi

im just wondering what happen when you uses glacier-cli and you request to delete a file .

Do I have to understand also Glacier will wait some hours to retrieve my request and delete the files?

I have done two tests requesting a deletion of a file and then, when I run another time

$ glacier --region eu-west-1 archive list myvault

the file requested to delete it is not showed anymore. so? :)

what happen? You have to wait hours to retrieve a file but the delete is instantaneous ?

This is important to me because I dont want to archive different versions of the same files. So Im thinking to run a script to delete first all uploaded files and then make the upload of the new ones.

Thanks!

When i try to upload empty file, i'm getting error IndexError: list index out of range
It's affected any automation!

alp ▶ ws207 ▶ ~ ▶ $ ▶ python /opt/glacier-cli/glacier.py --region eu-west-1 archive upload live example-content2222
it's hashes on input: []
len(hashes): 0
[]
Traceback (most recent call last):
File "/opt/glacier-cli/glacier.py", line 730, in
App().main()
File "/opt/glacier-cli/glacier.py", line 716, in main
self.args.func()
File "/opt/glacier-cli/glacier.py", line 498, in archive_upload
file_obj=self.args.file, description=name)
File "/usr/lib/python2.7/dist-packages/boto/glacier/vault.py", line 179, in create_archive_from_file
writer.close()
File "/usr/lib/python2.7/dist-packages/boto/glacier/writer.py", line 229, in close
self.uploader.close()
File "/usr/lib/python2.7/dist-packages/boto/glacier/writer.py", line 156, in close
hex_tree_hash = bytes_to_hex(tree_hash(self._tree_hashes))
File "/usr/lib/python2.7/dist-packages/boto/glacier/utils.py", line 108, in tree_hash
return hashes[0]
IndexError: list index out of range

There have been no commits for almost 2 years, and the installation instructions seem out of date, as reported in #49. This leads me to wonder if the project is still maintained? It seems that other forks are more active. Thanks for any info you can shed.

#30 led me to boto/boto#1548 (comment) which in turn led me to the official aws glacier command. So now I'm a bit confused why glacier-cli exists at all, when there is an officially supported and apparently comprehensive upstream command which does the same thing. Is it simply that glacier-cli was created before aws glacier? Or does glacier-cli offer something which aws glacier doesn't? (besides git-annex support of course, which for me personally is the whole point of using glacier-cli)

I think this should be clarified in the README...

The current --help is not very helpful, would be nice to get a complete list of all possible commands.

I've encountered this issue with glacier-cli failing due to git-annex mistakenly adding things that look like file extension to the key when using the SHA256E backend. Essentially what it means is that certain files will have characters that look like a file extension appended to the key, even when they might not be part of the extension.

Example:

 % ls 12.\ Change\ The\ World\ \(feat.\ 웅산\).mp3 
12. Change The World (feat. 웅산).mp3
 % git annex info 12.\ Change\ The\ World\ \(feat.\ 웅산\).mp3
file: 12. Change The World (feat. 웅산).mp3
size: 7.48 megabytes
key: SHA256E-s7479642--957208748ae03fe4fc8d7877b2c9d82b7f31be0726e4a3dec9063b84cc64cf09.웅산.mp3
present: true
 % git annex calckey 12.\ Change\ The\ World\ \(feat.\ 웅산\).mp3
SHA256E-s7479642--957208748ae03fe4fc8d7877b2c9d82b7f31be0726e4a3dec9063b84cc64cf09.웅산.mp3

I've opened an issue with git-annex here:
https://git-annex.branchable.com/bugs/git-annex_adds_unicode_characters_at_end_of_checksum/

And the will be a fix for the case with brackets, but there are other cases in which a file extension might not be just ASCII. And then this is what happens:

% git annex copy 12.\ Change\ The\ World\ \(feat.\ 웅산\).mp3 --to glacier
copy 12. Change The World (feat. 웅산).mp3 (checking glacier...) Traceback (most recent call last):
  File "/usr/local/bin/glacier", line 737, in <module>
    main() 
  File "/usr/local/bin/glacier", line 733, in main
    App().main()
  File "/usr/local/bin/glacier", line 719, in main
    self.args.func()
  File "/usr/local/bin/glacier", line 600, in archive_checkpresent
    self.args.vault, self.args.name)
  File "/usr/local/bin/glacier", line 161, in get_archive_last_seen
    result = self._get_archive_query_by_ref(vault, ref).one()
  File "/usr/local/bin/glacier", line 136, in _get_archive_query_by_ref
    if ref.startswith('id:'):
UnicodeDecodeError: 'ascii' codec can't decode byte 0xec in position 83: ordinal not in range(128)
(user error (glacier ["--region=eu-west-1","archive","checkpresent","music","--quiet","SHA256E-s7479642--957208748ae03fe4fc8d7877b2c9d82b7f31be0726e4a3dec9063b84cc64cf09.\50885\49328.mp3"] exited 1)) failed
git-annex: copy: 1 failed

Now, As the bug report says, you can avoid this issue by changing your backend from SHA256E to SHA256 to avoid adding extensions. But I think addressing this issue would be good anyway.

Hi i'm trying git annex with glacier archiving but unfortunately I get an strange UnicodeDecodeError when trying to archive a file. First I thought it's related to #16, but even if I rename the file to test.txtI'll get the same error msg:

Traceback (most recent call last):
  File "/usr/local/bin/glacier", line 9, in <module>
    load_entry_point('glacier-cli==0.1.0', 'console_scripts', 'glacier-cli')()
  File "build/bdist.macosx-10.11-x86_64/egg/glacier.py", line 732, in entry_point
  File "build/bdist.macosx-10.11-x86_64/egg/glacier.py", line 718, in main
  File "build/bdist.macosx-10.11-x86_64/egg/glacier.py", line 500, in archive_upload
  File "/Users/markus/.virtualenvs/glacier-cli/lib/python2.7/site-packages/boto-2.38.0-py2.7.egg/boto/glacier/vault.py", line 178, in create_archive_from_file
    writer.close()
  File "/Users/markus/.virtualenvs/glacier-cli/lib/python2.7/site-packages/boto-2.38.0-py2.7.egg/boto/glacier/writer.py", line 228, in close
    self.partitioner.flush()
  File "/Users/markus/.virtualenvs/glacier-cli/lib/python2.7/site-packages/boto-2.38.0-py2.7.egg/boto/glacier/writer.py", line 79, in flush
    self._send_part()
  File "/Users/markus/.virtualenvs/glacier-cli/lib/python2.7/site-packages/boto-2.38.0-py2.7.egg/boto/glacier/writer.py", line 75, in _send_part
    self.send_fn(part)
  File "/Users/markus/.virtualenvs/glacier-cli/lib/python2.7/site-packages/boto-2.38.0-py2.7.egg/boto/glacier/writer.py", line 222, in _upload_part
    self.uploader.upload_part(self.next_part_index, part_data)
  File "/Users/markus/.virtualenvs/glacier-cli/lib/python2.7/site-packages/boto-2.38.0-py2.7.egg/boto/glacier/writer.py", line 129, in upload_part
    content_range, part_data)
  File "/Users/markus/.virtualenvs/glacier-cli/lib/python2.7/site-packages/boto-2.38.0-py2.7.egg/boto/glacier/layer1.py", line 1279, in upload_part
    response_headers=response_headers)
  File "/Users/markus/.virtualenvs/glacier-cli/lib/python2.7/site-packages/boto-2.38.0-py2.7.egg/boto/glacier/layer1.py", line 114, in make_request
    data=data)
  File "/Users/markus/.virtualenvs/glacier-cli/lib/python2.7/site-packages/boto-2.38.0-py2.7.egg/boto/connection.py", line 1071, in make_request
    retry_handler=retry_handler)
  File "/Users/markus/.virtualenvs/glacier-cli/lib/python2.7/site-packages/boto-2.38.0-py2.7.egg/boto/connection.py", line 943, in _mexe
    request.body, request.headers)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1053, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1093, in _send_request
    self.endheaders(body)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1049, in endheaders
    self._send_output(message_body)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 891, in _send_output
    msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0x8c in position 0: ordinal not in range(128)

Hi

It would be nice if you could drop a line about encryption into the readme. Are files encryted when used directly? Are they encrypted when using git annex? Am I right that using git annex makes the backup unreadable by other tools?

Regards, Peter

I think something changed on Amazon side as the command

vault list

only report the first 10 vaults.

Eg:

$binary --region $region vault list | wc -l
10

and I've like 200 vaults remotely.

Same happen on their aws glacier ui, they show you the first 10 and then you've to push "load more".

Can this be fixed somehow?

Thanks
A.

glacier-store-hook = glacier archive upload --name="$ANNEX_KEY" vault-name "$ANNEX_FILE"

The other lines have $ANNEX_KEY as the vault name.

Currently keys need to be specified through environment variables and region by command line argument. Would be nice to be able to do both in both ways.

It would be great to have an option on the "upload" command to delete the local copy of the file after a successful upload. Maybe a "--delete-local"????

I am encrypting my files before uploading to Glacier (like everybody). Therefore, after the file is uploaded I don't need the local file anymore. The deletion of the file also indicates that the file has been uploaded. This means I can interrupt an upload script and restart without duplicating uploads.

I need to be able to interrupt my upload script sometimes because I need my uplink bandwidth for other purposes.

Further info: I am uploading my 80GB of photos as individual encrypted files via a slow 2Mbps link. I am using a basic 'find' to recurse down my directory tree, with an -exec to encrypt, upload, delete encrypted file. Interrupting my simple 'find' and restarting leads to duplicated files. Adding the above option, would allow me to create an encrypted cache of my files, and upload when convenient. If it's deleted then it was successfully uploaded.

checkpresent prints "archive not found" to stderr if a key is not found. This clutters up the git-annex display, in both the hook special remote, and the native glacier git-annex remote that I am currently writing. Could that instead be sent to stdout, or not printed at all?

boto is a requirement to run glacier-cli, but when boto is installed, boto creates a glacier script.

https://github.com/boto/boto/blob/develop/setup.py#L59

In the README examples, glacier.py is referred to as just glacier and in this case it won't work because boto will override it.

Attempts to upload a very large (288.64 GB) file run for about an hour or two, then fail with the following output:

frontier:glacier-cli khagler$ ./glacier.py archive upload Photos ~/Documents/photos.tgz
Traceback (most recent call last):
File "./glacier.py", line 618, in
App().main()
File "./glacier.py", line 604, in main
args.func(args)
File "./glacier.py", line 416, in archive_upload
archive_id = vault.create_archive_from_file(file_obj=args.file, description=name)
File "/Users/khagler/glacier/glacier-cli/boto/glacier/vault.py", line 141, in create_archive_from_file
writer.write(data)
File "/Users/khagler/glacier/glacier-cli/boto/glacier/writer.py", line 152, in write
self.send_part()
File "/Users/khagler/glacier/glacier-cli/boto/glacier/writer.py", line 141, in send_part
content_range, part)
File "/Users/khagler/glacier/glacier-cli/boto/glacier/layer1.py", line 626, in upload_part
response_headers=response_headers)
File "/Users/khagler/glacier/glacier-cli/boto/glacier/layer1.py", line 83, in make_request
data=data)
File "/Users/khagler/glacier/glacier-cli/boto/connection.py", line 913, in make_request
return self._mexe(http_request, sender, override_num_retries)
File "/Users/khagler/glacier/glacier-cli/boto/connection.py", line 859, in _mexe
raise e
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

OS: Mac OS X 10.7.5 Server
Python 2.7.1

I tried uploading the same file using FastGlacier with the part size set to 1 GB. It would upload some of each part before failing with a message about the remote host dropping the connection. After setting the part size to 256 MB, it was able to upload individual parts successfully.

Addendum:

After a bit more investigation, I think I've figured out what might be going on. According to the Amazon documentation, the maximum number of parts for a multi-part upload is 10,000. For this (very large) archive to be split evenly into 10,000 parts, each part would have to be about 27.5 MB--or, given the limits on allowable part sizes, 32 MB. It looks like you're using a default part size (which I didn't realize at the time I could change) of 8 MB. If I'm right about that, then an 80 GB file would be a (marginally less painful) valid test.

I have filename "./2009/Agátka ve školce/PC090374.JPG"
and I'm trying to upload it using:
`glacier archive upload --name "./2009/Agátka ve školce/PC090374.JPG" Photos "./2009/Agátka ve školce/PC090374.JPG"``

I end up with traceback:

Traceback (most recent call last):
  File "/usr/local/bin/glacier", line 618, in <module>
    App().main()
  File "/usr/local/bin/glacier", line 604, in main
    args.func(args)
  File "/usr/local/bin/glacier", line 416, in archive_upload
    archive_id = vault.create_archive_from_file(file_obj=args.file, description=name)
  File "/home/mirek/glacier-cli/boto/glacier/vault.py", line 163, in create_archive_from_file
    part_size=part_size)
  File "/home/mirek/glacier-cli/boto/glacier/vault.py", line 126, in create_archive_writer
    description)
  File "/home/mirek/glacier-cli/boto/glacier/layer1.py", line 479, in initiate_multipart_upload
    response_headers=response_headers)
  File "/home/mirek/glacier-cli/boto/glacier/layer1.py", line 83, in make_request
    raise UnexpectedHTTPResponseError(ok_responses, response)
boto.glacier.exceptions.UnexpectedHTTPResponseError

Not sure if this is problem of boto or glacier-cli.

Will investigate later.

HI

I don't understand how pass aws credentials to glacier-cli

glacier vault list
Traceback (most recent call last):
File "/usr/local/bin/glacier", line 730, in
App().main()
File "/usr/local/bin/glacier", line 705, in init
connection = boto.glacier.connect_to_region(args.region)
File "/root/scripts/glacier-cli/boto/glacier/init.py", line 56, in connect_to_region
return region.connect(*_kw_params)
File "/root/scripts/glacier-cli/boto/regioninfo.py", line 62, in connect
return self.connection_cls(region=self, *_kw_params)
File "/root/scripts/glacier-cli/boto/glacier/layer2.py", line 38, in init
self.layer1 = Layer1(_args, *_kwargs)
File "/root/scripts/glacier-cli/boto/glacier/layer1.py", line 63, in init
suppress_consec_slashes)
File "/root/scripts/glacier-cli/boto/connection.py", line 525, in init
host, config, self.provider, self._required_auth_capability())
File "/root/scripts/glacier-cli/boto/auth.py", line 621, in get_auth_handler
'Check your credentials' % (len(names), str(names)))
boto.exception.NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV4Handler'] Check your credentials

Does anybody any clue?

thanks

"Check out the glacier branch of boto from Github (this branch is not released yet and is still under heavy development)." but this is not true - the branch only has 2 commits which are not in master:

https://github.com/boto/boto/compare/glacier

and even those changes (i.e. adding a description parameter) seem to be already upstream.

I have a 2.7G file that fails to upload every time with the following error:

Traceback (most recent call last):
File "/home/jarno/bin/glacier", line 694, in
App().main()
File "/home/jarno/bin/glacier", line 680, in main
args.func(args)
File "/home/jarno/bin/glacier", line 482, in archive_upload
archive_id = vault.create_archive_from_file(file_obj=args.file, description=name)
File "/home/jarno/bin/glacier-cli/glacier-cli/boto/glacier/vault.py", line 141, in create_archive_from_file
writer.write(data)
File "/home/jarno/bin/glacier-cli/glacier-cli/boto/glacier/writer.py", line 152, in write
self.send_part()
File "/home/jarno/bin/glacier-cli/glacier-cli/boto/glacier/writer.py", line 141, in send_part
content_range, part)
File "/home/jarno/bin/glacier-cli/glacier-cli/boto/glacier/layer1.py", line 626, in upload_part
response_headers=response_headers)
File "/home/jarno/bin/glacier-cli/glacier-cli/boto/glacier/layer1.py", line 88, in make_request
raise UnexpectedHTTPResponseError(ok_responses, response)
boto.glacier.exceptions.UnexpectedHTTPResponseError: Expected (204,), got (403, {"message":"The value passed in as x-amz-content-sha256 does not match the computed payload hash. Computed digest: e5c9cf7330ec9629b5db363c84e0da85e8267ad3b4cc646135fa1450c78c7e7b expected hash: dd4e39e6d6e0dc191af09d36db834dd25026a97b066956a09e2664033142f253","code":"InvalidSignatureException","type":"Client"})

Any ideas what to try?