krobertson / deb-s3 Goto Github PK

View Code? Open in Web Editor NEW

482.0 482.0 149.0 216 KB

Easily create and manage an APT repository on S3 -- NO LONGER MAINTAINED

License: MIT License

Ruby 95.11% HTML 4.89%

deb-s3's People

Contributors

Stargazers

Watchers

Forkers

csizsek micmac musicglue rylon hjrnunes shutej bennullgraham crohr bmoyles scalp42 attili cloudify stk0vrfl0w bazbremner gudmundur raynes raficus pmdumuid cotdsa yannh ministryofjustice janstenpickle jmoberly dwb jmoseley jakedahn magec remh leonardehrenfried zoltrain guilhem madebymany petecheslock fstehle stensonb lsiudut hunglin jnankin whiteley dejo1307 relops thedrow zendeskarchive leedm777 uswitch sammarx kaiyzen anarcher rmldsky pgermishuys datadog isaoshimizu alexmreis evertrue kwatson tools-alexuser01 paddymahoney auvik kleopatra999 jeffw-wherethebitsroam jiakuan intfrr mrmicks ex3cv talevy draconrose lswith luan seomoz clofresh dorianzaccaria georgezhai bndzor dr-strangecode activeprospect jen20 gengmao pombredanne gdm jhart-r7 ruland billups alxschwarz russmac bliskner rtloeffler cnicolov vmlintu-nosto noriordan ar45 dlovitch armory kusabisensei blamarvt njd-siteimprove agimenez thusoy ctompkinson mtanda spaceapegames

deb-s3's Issues

apt-get does not find the uploaded deb package

More of a question than an actual issue, but when I upload a package apt-get does not seem to find it. The repository is http://repository.homeki.com/, and I use --prefix packages so all files are located at http://repository.homeki.com/packages. The GPG public key can be found at http://repository.homeki.com/homeki-public.key.

http://repository.homeki.com/packages/dists/unstable/Release contains the following:

Codename: unstable
Date: Wed, 10 Sep 2014 21:57:16 UTC
Architectures: all amd64 i386 armhf
Components: main
MD5Sum:
 4f3cf5178b0dd03424da2fd14b71e1ae              744 main/binary-all/Packages
 96e0f20dfecc7a5632874b86d9399305              519 main/binary-all/Packages.gz
 d41d8cd98f00b204e9800998ecf8427e                0 main/binary-amd64/Packages
 8400fa1de14b9d32e31b1cea5c4ff83e               20 main/binary-amd64/Packages.gz
 309336f2c4b3735122f113733cae3d09              593 main/binary-armhf/Packages
 80e16a7549bc9a101050e40a6bae9149              439 main/binary-armhf/Packages.gz
 d41d8cd98f00b204e9800998ecf8427e                0 main/binary-i386/Packages
 e9cc6202fcb07bf0c3b566b58a992687               20 main/binary-i386/Packages.gz
SHA1:
 d430bd1513c071e60c0c5fe2eec74701c88b43b0              744 main/binary-all/Packages
 c1fcc0929684ec545217c35631a3a044fff6db52              519 main/binary-all/Packages.gz
 da39a3ee5e6b4b0d3255bfef95601890afd80709                0 main/binary-amd64/Packages
 57d33b0adb18bc672fa3f1830ebba6564ee59cbd               20 main/binary-amd64/Packages.gz
 67178b66271827e6bd290aaf302b6aca90016200              593 main/binary-armhf/Packages
 4d8ad1038830f5ec3449d4e8d91407b918f318ed              439 main/binary-armhf/Packages.gz
 da39a3ee5e6b4b0d3255bfef95601890afd80709                0 main/binary-i386/Packages
 22b71702958a741fbaf314ee05c92e70640f0b08               20 main/binary-i386/Packages.gz
SHA256:
 942c526090e1bfa2b85d13cbcc97c5edb45441fb1902b7afd85b8dfc7829a7b0              744 main/binary-all/Packages
 3aaebeef47f6d20db4fce2121d420f41b11ed9f8b51fdf1dcb37ec71aa5a227b              519 main/binary-all/Packages.gz
 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855                0 main/binary-amd64/Packages
 056d24401881a17aff07d43a6ea546039a65c56ead73ce85f9074199134c2898               20 main/binary-amd64/Packages.gz
 8f4c5ed1ad2d2a7c4c1c45e005120ece7e5b0e49fcfd9acfa0b87a187908ea13              593 main/binary-armhf/Packages
 d5cf6598b1017217da8c9499a4a95284ff6be66ddae7ee4344f3e8781af29bcc              439 main/binary-armhf/Packages.gz
 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855                0 main/binary-i386/Packages
 894f125c15a2c1bd529dad459981745141e3318f3c313c92c5cba78e1672f343               20 main/binary-i386/Packages.gz

And if we follow the path into Packages by going to http://repository.homeki.com/packages/dists/unstable/main/binary-all/Packages, we find:

Package: homeki
Version: 3.1410383081.8a4d4ac
License: unknown
Vendor: none
Architecture: all
Maintainer: Homeki Development Team <[email protected]>
Installed-Size: 24240
Depends: openjdk-7-jre-headless, postgresql, telldus, owfs, libjna-java
Section: misc
Priority: extra
Homepage: http://nourlgiven.example.com/
Filename: pool/h/ho/homeki.deb
Size: 23156866
SHA1: 4f9ab1b0fa19998422a447da6b9491da452f880f
SHA256: 996124535d21d1cc99ae3cd7c07bf13a325d2580bbda690f8592be99aa3e5f6d
MD5sum: 2de33dcdcd24b2578a0552117fabf795
Description: Unified interface for all your sensors and devices connected to your computer.
 Provides a REST HTTP API for accessing, logging, modifying and managing various devices and sensors connected to your computer.

So it really seems the package is there and available. I then go ahead and add the below to /etc/apt/sources.list, as well as add the GPG public key.

# Homeki
deb http://repository.homeki.com/packages unstable main

If I now run

sudo apt-get update
sudo apt-get install homeki

I get

E: Unable to locate package homeki

even though the sudo apt-get update returns no error and it specified it downloaded the Release from unstable.

Does anyone have any idea why my package is not picked up by apt-get?

Thanks in advance!

deb-s3 delete fails with TypeError

I’m trying to delete a package from my S3 APT repo.

delete my_package -v authenticated -b my-apt-repo -c stage --arch amd64 --versions 0.90.1

>> Versions to delete: 0.90.1
>> Retrieving existing manifests
/var/lib/gems/1.9.1/gems/deb-s3-0.7.0/lib/deb/s3/manifest.rb:72:in `+': can't convert nil into String (TypeError)
    from /var/lib/gems/1.9.1/gems/deb-s3-0.7.0/lib/deb/s3/manifest.rb:72:in `block in delete_package'
    from /var/lib/gems/1.9.1/gems/deb-s3-0.7.0/lib/deb/s3/manifest.rb:67:in `select'
    from /var/lib/gems/1.9.1/gems/deb-s3-0.7.0/lib/deb/s3/manifest.rb:67:in `delete_package'
    from /var/lib/gems/1.9.1/gems/deb-s3-0.7.0/lib/deb/s3/cli.rb:390:in `delete'
    from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor/command.rb:27:in `run'
    from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor/invocation.rb:120:in `invoke_command'
    from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor.rb:363:in `dispatch'
    from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor/base.rb:439:in `start'
    from /var/lib/gems/1.9.1/gems/deb-s3-0.7.0/bin/deb-s3:8:in `<top (required)>'
    from /usr/local/bin/deb-s3:19:in `load'
    from /usr/local/bin/deb-s3:19:in `<main>'

Add support to choose different aws region

Hi Ken,

Here is the error messge when I'm trying use deb-s3 to upload my deb.
My apt repo on s3 was initialized using reprepro, and it is functional correctly.
BTW, my desktop is running ubuntu 12.04

seems like errors when parsing the aws response, would you pls take some time to look into it, thanks in advance:-)

Examining package file mydeb.deb
Retrieving existing manifests
/usr/lib/ruby/1.9.1/rexml/parsers/baseparser.rb:410:in block in pull': Undefined prefix Net: found (REXML::UndefinedNamespaceException) from /usr/lib/ruby/1.9.1/set.rb:221:inblock in each'
from /usr/lib/ruby/1.9.1/set.rb:221:in each_key' from /usr/lib/ruby/1.9.1/set.rb:221:ineach'
from /usr/lib/ruby/1.9.1/rexml/parsers/baseparser.rb:408:in pull' from /usr/lib/ruby/1.9.1/rexml/parsers/treeparser.rb:22:inparse'
from /usr/lib/ruby/1.9.1/rexml/document.rb:243:in build' from /usr/lib/ruby/1.9.1/rexml/document.rb:43:ininitialize'
from /var/lib/gems/1.9.1/gems/xml-simple-1.1.2/lib/xmlsimple.rb:965:in new' from /var/lib/gems/1.9.1/gems/xml-simple-1.1.2/lib/xmlsimple.rb:965:inparse'
from /var/lib/gems/1.9.1/gems/xml-simple-1.1.2/lib/xmlsimple.rb:164:in xml_in' from /var/lib/gems/1.9.1/gems/xml-simple-1.1.2/lib/xmlsimple.rb:203:inxml_in'
from /var/lib/gems/1.9.1/gems/aws-s3-0.6.3/lib/aws/s3/parsing.rb:64:in parse' from /var/lib/gems/1.9.1/gems/aws-s3-0.6.3/lib/aws/s3/parsing.rb:55:ininitialize'
from /var/lib/gems/1.9.1/gems/aws-s3-0.6.3/lib/aws/s3/response.rb:55:in new' from /var/lib/gems/1.9.1/gems/aws-s3-0.6.3/lib/aws/s3/response.rb:55:inparsed'
from /var/lib/gems/1.9.1/gems/aws-s3-0.6.3/lib/aws/s3/extensions.rb:183:in block in parsed' from /var/lib/gems/1.9.1/gems/aws-s3-0.6.3/lib/aws/s3/extensions.rb:152:inexpirable_memoize'
from /var/lib/gems/1.9.1/gems/aws-s3-0.6.3/lib/aws/s3/extensions.rb:182:in parsed' from /var/lib/gems/1.9.1/gems/aws-s3-0.6.3/lib/aws/s3/response.rb:45:inerror?'
from /var/lib/gems/1.9.1/gems/aws-s3-0.6.3/lib/aws/s3/base.rb:72:in request' from /var/lib/gems/1.9.1/gems/aws-s3-0.6.3/lib/aws/s3/base.rb:88:inget'
from /var/lib/gems/1.9.1/gems/aws-s3-0.6.3/lib/aws/s3/object.rb:134:in value' from /var/lib/gems/1.9.1/gems/aws-s3-0.6.3/lib/aws/s3/object.rb:138:instream'
from /var/lib/gems/1.9.1/gems/deb-s3-0.2.0/lib/deb/s3/utils.rb:47:in s3_read' from /var/lib/gems/1.9.1/gems/deb-s3-0.2.0/lib/deb/s3/release.rb:23:inretrieve'
from /var/lib/gems/1.9.1/gems/deb-s3-0.2.0/lib/deb/s3/cli.rb:114:in upload' from /var/lib/gems/1.9.1/gems/thor-0.17.0/lib/thor/task.rb:27:inrun'
from /var/lib/gems/1.9.1/gems/thor-0.17.0/lib/thor/invocation.rb:120:in invoke_task' from /var/lib/gems/1.9.1/gems/thor-0.17.0/lib/thor.rb:344:indispatch'
from /var/lib/gems/1.9.1/gems/thor-0.17.0/lib/thor/base.rb:434:in start' from /var/lib/gems/1.9.1/gems/deb-s3-0.2.0/bin/deb-s3:8:in<top (required)>'
from /usr/local/bin/deb-s3:19:in load' from /usr/local/bin/deb-s3:19:in
'

Go with newer(-est) version of aws-sdk in dependencies?

Hi,
Currently the gem depends on "aws-sdk", "~> 1.18" is there a chance to bump it up to recent version of aws-sdk? I could see in the commit history you did change the dependency once already but reverted it back the next day. Where there any problems with it? Does it involve a lot of code changes of deb-s3 itself? Is this planned anytime soon?

Thanks!

preserve-versions flag always obstructs FILE argument

First, this is a super-handy gem, thank you 👍

I'm interested in the preserve-versions option offered but have not been able to make the command run when including this flag. I tried all kinds of yes/true/no-arg formatting until looking at the code and seeing that it seemed to indeed function as a flag (at least, since changing how it worked in 73a1e45, after Pull Request #1 got merged)

Steps to Reproduce

deb-s3 upload my-package_0.8_amd64.deb --bucket=my-packages.skyebook.net --visibility=authenticated --arch=amd64 --access-key=foo --secret-key=bar --preserve-versions=yes
deb-s3 upload requires at least 1 argument: "deb-s3 upload FILE -b, --bucket=BUCKET".

This is the consistent result of putting the --preserve-versions (or -p) flag anywhere in the order of commands. Removing it lets uploads proceed as normally.

I apologize for not being able to bring a Pull Request with me, but my level of Ruby knowledge is NaN when measured from 1-10

Creating a new repo

Is it a goal of this project to be able to initialise a new repository up on S3, or is that still the job of reprepro (or something else?)

Thank you

Despite specifying arch type ALL manifests get uploaded

$ ./bin/deb-s3 upload -a amd64 -p -b joel-test-xyz -v private joel_2-1_amd64.deb 
>> Retrieving existing manifests
>> Examining package file joel_2-1_amd64.deb
/usr/bin/dpkg
>> Uploading packages and new manifests to S3
   -- Transferring pool/j/jo/joel_2-1_amd64.deb
   -- Transferring dists/stable/main/binary-amd64/Packages
   -- Transferring dists/stable/main/binary-amd64/Packages.gz
   -- Transferring dists/stable/main/binary-i386/Packages
   -- Transferring dists/stable/main/binary-i386/Packages.gz
   -- Transferring dists/stable/Release
>> Update complete.

OS X: No such file or directory

When doing

deb-s3 upload --bucket cbox-pkg-repo-test.s3.amazonaws.com cbox-webapp_1.0.5_all.deb --access-key=mykey --secret-key=mysecret

on OSX 10.8 I get:

>> Retrieving existing manifests
>> Examining package file cbox-webapp_1.0.5_all.deb
ar: control.tar.gz: not found in archive
/usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/gems/1.9.1/gems/deb-s3-0.5.1/lib/deb/s3/package.rb:65:in `read': No such file or directory - /var/folders/97/1jdjqycx60xf5vcmc3l2lh740000gn/T/d20130815-37872-fxbt4o/control (Errno::ENOENT)
    from /usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/gems/1.9.1/gems/deb-s3-0.5.1/lib/deb/s3/package.rb:65:in `block in extract_control'
    from /usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/1.9.1/tmpdir.rb:83:in `mktmpdir'
    from /usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/gems/1.9.1/gems/deb-s3-0.5.1/lib/deb/s3/package.rb:63:in `extract_control'
    from /usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/gems/1.9.1/gems/deb-s3-0.5.1/lib/deb/s3/package.rb:47:in `parse_file'
    from /usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/gems/1.9.1/gems/deb-s3-0.5.1/lib/deb/s3/cli.rb:108:in `block in upload'
    from /usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/gems/1.9.1/gems/deb-s3-0.5.1/lib/deb/s3/cli.rb:106:in `each'
    from /usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/gems/1.9.1/gems/deb-s3-0.5.1/lib/deb/s3/cli.rb:106:in `upload'
    from /usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/gems/1.9.1/gems/thor-0.18.1/lib/thor/command.rb:27:in `run'
    from /usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/gems/1.9.1/gems/thor-0.18.1/lib/thor/invocation.rb:120:in `invoke_command'
    from /usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/gems/1.9.1/gems/thor-0.18.1/lib/thor.rb:363:in `dispatch'
    from /usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/gems/1.9.1/gems/thor-0.18.1/lib/thor/base.rb:439:in `start'
    from /usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/gems/1.9.1/gems/deb-s3-0.5.1/bin/deb-s3:8:in `<top (required)>'
    from /usr/local/Cellar/ruby/1.9.3-p362/bin/deb-s3:23:in `load'
    from /usr/local/Cellar/ruby/1.9.3-p362/bin/deb-s3:23:in `<main>'

Also, not sure how and if it's related, but the package does contain control.tar.gz (it's built with fpm).

This doesn't seem to happen on Ubuntu 12.04LTS.

Copy Command

Something that we've been needing for a while is package promotion. Being able to move it from one distribution/component to another.

This way we can build against a commit, and promote that package up to different repositories vs having to publish it to multiple repositories. Since the GPG, file management has already been done in deb-s3 it makes sense to me to add this feature here.

It's also something that would save on upload time, since you can do an internal S3 copy that doesn't leave the AWS network.

Happy to start plugging away at it, just thought I'd put it up for discussion here, in terms of how the command should look.

deb-s3 cp "package_name/path" "package_name/path" where path = distribution/component elements.
e.g.
deb-s3 cp redis2.8.1 stable/main/staging stable/main/production

sysread_nonblock: end of file reached (EOFError)

I'm trying to get a new repo setup with a package built via FPM and when pushing into a fresh s3 bucket I'm seeing the below error. This error is from the tip of master, and I see the same issue from the most recent gem.

Ubuntu 12.04
Ruby 1.9.3

Is there something I need to do to get an initial repo setup? Are there any debug flags I can set to see what it is doing further?

root@f3ed34bc5efa:/deb-s3# bundle exec deb-s3 upload --bucket=charliek-apt --access-key-id=XXXXX --secret-access-key=XXXX jre1.7.0-45_1.0_amd64.deb
>> Retrieving existing manifests
>> Examining package file jre1.7.0-45_1.0_amd64.deb
/usr/bin/dpkg
/usr/lib/ruby/1.9.1/openssl/buffering.rb:174:in `sysread_nonblock': end of file reached (EOFError)
    from /usr/lib/ruby/1.9.1/openssl/buffering.rb:174:in `read_nonblock'
    from /usr/lib/ruby/1.9.1/net/protocol.rb:141:in `rbuf_fill'
    from /usr/lib/ruby/1.9.1/net/protocol.rb:122:in `readuntil'
    from /usr/lib/ruby/1.9.1/net/protocol.rb:132:in `readline'
    from /usr/lib/ruby/1.9.1/net/http.rb:2562:in `read_status_line'
    from /usr/lib/ruby/1.9.1/net/http.rb:2551:in `read_new'
    from /usr/lib/ruby/1.9.1/net/http.rb:1319:in `block in transport_request'
    from /usr/lib/ruby/1.9.1/net/http.rb:1316:in `catch'
    from /usr/lib/ruby/1.9.1/net/http.rb:1316:in `transport_request'
    from /usr/lib/ruby/1.9.1/net/http.rb:1293:in `request'
    from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/core/http/connection_pool.rb:330:in `request'
    from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/core/http/net_http_handler.rb:61:in `block in handle'
    from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/core/http/connection_pool.rb:129:in `session_for'
    from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/core/http/net_http_handler.rb:55:in `handle'
    from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/core/client.rb:244:in `block in make_sync_request'
    from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/core/client.rb:280:in `retry_server_errors'
    from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/core/client.rb:240:in `make_sync_request'
    from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/core/client.rb:502:in `block (2 levels) in client_request'
    from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/core/client.rb:382:in `log_client_request'
    from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/core/client.rb:468:in `block in client_request'
    from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/core/client.rb:364:in `return_or_raise'
    from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/core/client.rb:467:in `client_request'
    from (eval):3:in `head_object'
    from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/s3/s3_object.rb:293:in `head'
    from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/s3/s3_object.rb:270:in `exists?'
    from /deb-s3/lib/deb/s3/utils.rb:55:in `s3_exists?'
    from /deb-s3/lib/deb/s3/utils.rb:59:in `s3_read'
    from /deb-s3/lib/deb/s3/manifest.rb:22:in `retrieve'
    from /deb-s3/lib/deb/s3/cli.rb:123:in `block in upload'
    from /deb-s3/lib/deb/s3/cli.rb:111:in `each'
    from /deb-s3/lib/deb/s3/cli.rb:111:in `upload'
    from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor/command.rb:27:in `run'
    from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor/invocation.rb:120:in `invoke_command'
    from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor.rb:363:in `dispatch'
    from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor/base.rb:439:in `start'
    from /deb-s3/bin/deb-s3:8:in `<top (required)>'
    from /var/lib/gems/1.9.1/bin/deb-s3:19:in `load'
    from /var/lib/gems/1.9.1/bin/deb-s3:19:in `<main>'

bucket names that are URLs result in errors

With s3, if you want to setup an apt repo that people access via apt.example.com, you need the bucket in s3 to be called apt.example.com

however, if you do this, and then you try to do:
deb-s3 upload ---bucket="apt.example.com" examplefile.deb

it will throw an error

Retrieving existing manifests
/var/lib/gems/1.9.1/gems/aws-sdk-1.42.0/lib/aws/core/client.rb:375:in return_or_raise': AWS::Errors::Base (AWS::Errors::Base) from /var/lib/gems/1.9.1/gems/aws-sdk-1.42.0/lib/aws/core/client.rb:476:inclient_request'
from (eval):3:in head_object' from /var/lib/gems/1.9.1/gems/aws-sdk-1.42.0/lib/aws/s3/s3_object.rb:293:inhead'
from /var/lib/gems/1.9.1/gems/aws-sdk-1.42.0/lib/aws/s3/s3_object.rb:270:in exists?' from /var/lib/gems/1.9.1/gems/deb-s3-0.6.1/lib/deb/s3/utils.rb:55:ins3_exists?'
from /var/lib/gems/1.9.1/gems/deb-s3-0.6.1/lib/deb/s3/utils.rb:59:in s3_read' from /var/lib/gems/1.9.1/gems/deb-s3-0.6.1/lib/deb/s3/release.rb:23:inretrieve'
from /var/lib/gems/1.9.1/gems/deb-s3-0.6.1/lib/deb/s3/cli.rb:107:in upload' from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor/command.rb:27:inrun'
from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor/invocation.rb:120:in invoke_command' from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor.rb:363:indispatch'
from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor/base.rb:439:in start' from /var/lib/gems/1.9.1/gems/deb-s3-0.6.1/bin/deb-s3:8:in<top (required)>'
from /usr/local/bin/deb-s3:23:in load' from /usr/local/bin/deb-s3:23:in
'

update index without upload when already in pool

Sometimes I want to promote a package to a different component to support staging environments. So I use the components unstable, releasecandidate and stable.
Currently I have to re-upload the package in order to install them in releasecandidate or stable but it is actually already in the pool so it would be faster if we could skip this step.

Handle architecture independent packages correctly

deb-s3 puts arch-independent (*_all.deb) files in a binary-all when they should, AFAICT, be placed in the manifests for both binary-amd64 and binary-i386.

I'm working around this for now by setting the arch flag (-a amd64).

binary-all packages seem to be ignored by apt-get

I'm trying to debug a problem even though the Releases file looks ok. Any binary-all package in my repo generated by deb-s3 isn't picked up

[root@mysql1 /etc/apt]# apt-get update
Ign http://company-apt-trusty.s3.amazonaws.com stable InRelease
Hit http://company-apt-trusty.s3.amazonaws.com stable Release.gpg
Hit http://company-apt-trusty.s3.amazonaws.com stable Release
Hit http://company-apt-trusty.s3.amazonaws.com stable/main amd64 Packages
Hit http://company-apt-trusty.s3.amazonaws.com stable/main i386 Packages
Ign http://company-apt-trusty.s3.amazonaws.com stable/main Translation-en_US
Ign http://company-apt-trusty.s3.amazonaws.com stable/main Translation-en

My Releases file looks like

Mikes-MacBook-Pro:~ mike$ cat Release
Codename: stable
Date: Thu, 17 Jul 2014 02:29:05 UTC
Architectures: all amd64 i386
Components: main
MD5Sum:
 96dc9836059769df3751601db3106016            22376 main/binary-all/Packages
 70d286c2c73dee5efefc79806f59dc7b             6029 main/binary-all/Packages.gz
 315f477e0d604b618a6aa66f69c9701a            17520 main/binary-amd64/Packages
 7a21a6b4a08311743f3c9e39758c19be             4870 main/binary-amd64/Packages.gz
 d41d8cd98f00b204e9800998ecf8427e                0 main/binary-i386/Packages
 5216f264cc0ec4d7f87846910a9e79dc               20 main/binary-i386/Packages.gz
SHA1:
 65062c1f86eb76adc0c70ced023faa94700b7dce            22376 main/binary-all/Packages
 48a4e212fc2d0e9b23227b4762acc078a5eaee4c             6029 main/binary-all/Packages.gz
 59d93bc40d61fbb425f45f267812a7cde78c1e4c            17520 main/binary-amd64/Packages
 6c73cf9843aafe9a21e20619849c294c85b2061d             4870 main/binary-amd64/Packages.gz
 da39a3ee5e6b4b0d3255bfef95601890afd80709                0 main/binary-i386/Packages
 6359d0d0a7927fabc2c7407a5084e82c3e514863               20 main/binary-i386/Packages.gz
SHA256:
 c49cec29c78e104a8b765d552edb4d5e5ed172f3b62b1a6144d1fcd80d27351d            22376 main/binary-all/Packages
 7b7880a8edb631bfa87ad237d2f6814263486fb038d61063b7a07e585c4991bf             6029 main/binary-all/Packages.gz
 1923a7b095259caff3e8c76c17c5d129cb788d10233ff74741907bd9216f3b4b            17520 main/binary-amd64/Packages
 4a0839c6a3b1839c03694e0490b1cfab692b043cf52cb3009eaf69b421c79453             4870 main/binary-amd64/Packages.gz
 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855                0 main/binary-i386/Packages
 6e10437fdf39929b91cff0097600c7e3e76e0545a4320f518ad15c125bf7f7b6               20 main/binary-i386/Packages.gz

Also might want to add this to the help as it is a hacky way to fix my problem. I just told deb-s3 that the packages were amd64

     # validate we have them
      error("No architcture given and unable to determine one for #{file}. " +
            "Please specify one with --arch [i386,amd64].") unless arch

That isn't in the --help at all

Feel free to close this if you think this isn't a deb-s3 issue

IAM Role support

It would be awesome if deb-s3 could use IAM roles for credentials.

Install these packages without verification [y/N]

I created a PPA with deb-s3, and it worked very well. However, when users try to install the package hosted in the PPA, they are asked if they want to install without verification. How can I get rid of this message?

$ sudo add-apt-repository 'deb http://mslinn-ppa.s3.amazonaws.com stable main'
$ sudo apt-get update
$ sudo apt-get install pound
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  pound
0 upgraded, 1 newly installed, 0 to remove and 6 not upgraded.
Need to get 105 kB of archives.
After this operation, 239 kB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  pound
Install these packages without verification? [y/N] y
E: Some packages could not be authenticated

Use SHA256 digest when signing

APT has deprecated SHA1 signatures on repositories since version 1.2.7. Right now it's only a warning when downloading packages from a SHA1-signed repository, but starting in January 2017 it will become an error.

GPG signs using SHA1 by default, but can be configured to sign using SHA256. This should be the default on deb-s3 to avoid issues in the future. SHA256 has been supported in APT since version 0.7.7 (October 2007) so it should be safe to set as default.

To sign with SHA256 right now, the following option can be used: --gpg-options='--digest-algo SHA256'

AWS::S3::Errors::Forbidden

I use deb-s3 got the message in below. What happen about this?

I don't change any setting , it can be work normal before, but can not work now

Retrieving existing manifests
/var/lib/gems/1.9.1/gems/aws-sdk-1.40.2/lib/aws/core/client.rb:375:in return_or_raise': AWS::S3::Errors::Forbidden (AWS::S3::Errors::Forbidden) from /var/lib/gems/1.9.1/gems/aws-sdk-1.40.2/lib/aws/core/client.rb:478:inclient_request'
from (eval):3:in head_object' from /var/lib/gems/1.9.1/gems/aws-sdk-1.40.2/lib/aws/s3/s3_object.rb:293:inhead'
from /var/lib/gems/1.9.1/gems/aws-sdk-1.40.2/lib/aws/s3/s3_object.rb:270:in exists?' from /var/lib/gems/1.9.1/gems/deb-s3-0.6.2/lib/deb/s3/utils.rb:55:ins3_exists?'
from /var/lib/gems/1.9.1/gems/deb-s3-0.6.2/lib/deb/s3/utils.rb:59:in s3_read' from /var/lib/gems/1.9.1/gems/deb-s3-0.6.2/lib/deb/s3/release.rb:23:inretrieve'
from /var/lib/gems/1.9.1/gems/deb-s3-0.6.2/lib/deb/s3/cli.rb:116:in upload' from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor/command.rb:27:inrun'
from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor/invocation.rb:120:in invoke_command' from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor.rb:363:indispatch'
from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor/base.rb:439:in start' from /var/lib/gems/1.9.1/gems/deb-s3-0.6.2/bin/deb-s3:8:in<top (required)>'
from /usr/local/bin/deb-s3:19:in load' from /usr/local/bin/deb-s3:19:in

Packages now download when trying to view in a browser

It used to be you could go to the url for a package, say
dists/master/debian/binary-armhf/Packages and the browser would load the XML for you to view in the browser.

Instead, as of version 7.1, the system now causes the browser to download the file content. Is this change by design? If not, I assume the mime type got changes somewhere, if you can give a hint on where to look I can try fixing it.

Large manifest file causes deb-s3 not to work

I've got a 60MB Packages file that prevents deb-s3 from completing. We upload upload upload and never delete packages. I'm currently investigating more and will update this ticket with more details as I dig.

Silently fails on EU buckets

I created a bucket on eu-west-1 and tried using deb-s3 with it. Everything seemed to be doing just fine, but the bucket always remained empty.

After digging around and trying different things, it seems I must provide a different endpoint for the EU S3 endpoint.

deb-s3 upload --endpoint=s3-eu-west-1.amazonaws.com -b my_bucket my_package.deb

The weird thing is that no errors are printed out or anything.

Origin would be nice to have!

I'd like to enable unattended-upgrades, per this:

http://plusbryan.com/my-first-5-minutes-on-a-server-or-essential-security-for-linux-servers

However, to follow those instructions, the release.rb file would need to be changed to do "Origin" properly, per these instructions:

http://askubuntu.com/questions/87849/how-to-enable-silent-automatic-updates-for-any-repository

I opened it and started looking into how to do it, but I'm not a Ruby wizard and so it didn't seem obvious.

It works spectacularly, otherwise! :-)

deb-s3 seems to hang on upload

Hi.

When trying to upload a package to an empty bucket, deb-s3 seems to hang.

Here's the command:

deb-s3 upload --bucket cbox-pkg-repo-test.s3.amazonaws.com cbox-webapp_1.0.5_all.deb --access-key=mykey --secret-key=mysecret

here's what is shown:

>> Retrieving existing manifests
>> Examining package file cbox-webapp_1.0.5_all.deb
>> Uploading packages and new manifests to S3
   -- Transferring pool/c/cb/cbox-webapp_1.0.5_all.deb

Nothing happens after this. It seems to hang. I monitored traffic, and while initially it seems to send data to AWS, it stops after a few seconds, and hangs indefinitely.
Is there a way to further diagnose this? Or is something wrong with my command?

When I interrupt it, here's the stack trace:

^C/home/vagrant/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/net/protocol.rb:199:in `write': Interrupt
    from /home/vagrant/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/net/protocol.rb:199:in `write0'
    from /home/vagrant/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/net/protocol.rb:173:in `block in write'
    from /home/vagrant/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/net/protocol.rb:190:in `writing'
    from /home/vagrant/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/net/protocol.rb:172:in `write'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/extensions.rb:330:in `send_request_with_body_stream'
    from /home/vagrant/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/net/http.rb:1922:in `exec'
    from /home/vagrant/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/net/http.rb:1318:in `block in transport_request'
    from /home/vagrant/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/net/http.rb:1317:in `catch'
    from /home/vagrant/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/net/http.rb:1317:in `transport_request'
    from /home/vagrant/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/net/http.rb:1294:in `request'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/connection.rb:45:in `block in request'
    from /home/vagrant/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/net/http.rb:746:in `start'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/connection.rb:52:in `request'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/base.rb:69:in `request'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/base.rb:88:in `put'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/object.rb:241:in `store'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/lib/deb/s3/utils.rb:58:in `block in s3_store'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/lib/deb/s3/utils.rb:57:in `open'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/lib/deb/s3/utils.rb:57:in `s3_store'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/lib/deb/s3/manifest.rb:69:in `block in write_to_s3'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/lib/deb/s3/manifest.rb:66:in `each'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/lib/deb/s3/manifest.rb:66:in `write_to_s3'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/lib/deb/s3/cli.rb:127:in `block in upload'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/lib/deb/s3/cli.rb:126:in `each_value'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/lib/deb/s3/cli.rb:126:in `upload'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/thor-0.18.1/lib/thor/command.rb:27:in `run'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/thor-0.18.1/lib/thor/invocation.rb:120:in `invoke_command'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/thor-0.18.1/lib/thor.rb:363:in `dispatch'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/thor-0.18.1/lib/thor/base.rb:439:in `start'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/bin/deb-s3:8:in `<top (required)>'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/bin/deb-s3:23:in `load'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/bin/deb-s3:23:in `<main>'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/bin/ruby_noexec_wrapper:14:in `eval'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/bin/ruby_noexec_wrapper:14:in `<main>'

Remove the ability to specify arch type?

This ticket is mostly a discussion.

What is the case where you want to allow a user to specify an arch type?
Is it valid to have a package of a same name in all, and specific arch types?
How does a system behave when an i386 package is placed in an amd64 repository?

At the least I think adjusting https://github.com/krobertson/deb-s3/blob/master/lib/deb/s3/cli.rb#L177 to be arch = pkg.architecture || options[:arch] would be more correct, as long as it throws a warning.

Thoughts on this @krobertson?

setting fail-if-exists flag caused index corruption if the uploaded file exists

The following upload command with the --fail-if-exists and the same filename will upload some of the Packages file to s3, then exit because of the file exists. Since some of the Packages file got uploaded but not the Release file. It causes the repository index corruption:

deb-s3 upload
--fail-if-exists
--preserve-versions
--bucket ABC
--codename precise
--component main
--arch all
--sign ABC
graphicsmagick_1.3.26~ABC_all.deb

Output:
00:09:11 >> Retrieving existing manifests
00:09:11 >> Examining package file graphicsmagick_1.3.26ABC_all.deb
00:09:18 !! Uploading manifest failed because: file https://s3.amazonaws.com/ABC/apt/pool/g/gr/graphicsmagick_1.3.26~ABC_all.deb already exists with different contents
00:09:18 >> Uploading packages and new manifests to S3
00:09:18 -- Transferring dists/precise/main/binary-amd64/Packages
00:09:18 -- Transferring dists/precise/main/binary-amd64/Packages.gz
00:09:18 -- Transferring dists/precise/main/binary-i386/Packages
00:09:18 -- Transferring dists/precise/main/binary-i386/Packages.gz
00:09:18 -- Transferring pool/g/gr/graphicsmagick_1.3.26ABC_all.deb

Seems like one solution is to upload the graphicsmagick_1.3.26~ABC_all.deb first and if it exists and exit without uploading other Packages files.

Pinned Thor version 0.18

Is there any reason for locking Thor version on ~> 0.18.0?

I have conflicts with other deb in my Gemfile and I was thinking about allowing using version 0.19.
Could we change it to ~> 0.18?

--gpg-options does not work

Trying to use --gpg-options produces an error like this:

$ deb-s3 upload --bucket redacted --arch amd64 --component redacted --codename production --access-key redacted --secret-key redacted --sign=redacted --gpg-options="--no-tty" redacted_162_all.deb 
!! File '--no-tty' doesn't exist

Inserting a little debug output shows that function upload is being passed an incorrect set of values for *files; is Thor incorrectly parsing the command-line?

["--no-tty", "redacted_all.deb"]

support for Openstack Swift?

First off...LOVE the idea of this gem.

I run an OpenStack Swift cluster that has the s3-proxy plugin which is supposed to emulate the S3 api...the only problem is, the aws-sdk gem (dependent upon here) has a hard time with non-AWS URLs.

What about native OpenStack Swift support? Of course, Swift is just another flavor of the week...

Actually, what I'm proposing is a pluggable design where any class satisfying a ObjectStore interface will work here.

I'm happy to do the abstraction, just wanting some upstream "approval" that this might be accepted.

Upload only working in one region (us-east-1), --endpoint not working as expect

Here is what I tried. my.oregon.bucket is hosted in Oregon. I tried the upload command with and without the endpoint "--endpoint=s3-website-us-west-2.amazonaws.com". Doesn't work for me.
Then I setup a bucket in US standard (us-east-1) with the exactly setting as the one in Oregon except the region. It works.

Here's the error:

deb-s3 upload my.deb --codename=stable --component=main  --bucket=my.oregon.bucket
>> Retrieving existing manifests
/Library/Ruby/Gems/2.0.0/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:375:in `return_or_raise': AWS::Errors::Base (AWS::Errors::Base)
    from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:476:in `client_request'
    from (eval):3:in `head_object'
    from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-v1-1.66.0/lib/aws/s3/s3_object.rb:296:in `head'
    from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-v1-1.66.0/lib/aws/s3/s3_object.rb:273:in `exists?'
    from /Library/Ruby/Gems/2.0.0/gems/deb-s3-0.7.1/lib/deb/s3/utils.rb:58:in `s3_exists?'
    from /Library/Ruby/Gems/2.0.0/gems/deb-s3-0.7.1/lib/deb/s3/utils.rb:62:in `s3_read'
    from /Library/Ruby/Gems/2.0.0/gems/deb-s3-0.7.1/lib/deb/s3/release.rb:30:in `retrieve'
    from /Library/Ruby/Gems/2.0.0/gems/deb-s3-0.7.1/lib/deb/s3/cli.rb:136:in `upload'
    from /Library/Ruby/Gems/2.0.0/gems/thor-0.18.1/lib/thor/command.rb:27:in `run'
    from /Library/Ruby/Gems/2.0.0/gems/thor-0.18.1/lib/thor/invocation.rb:120:in `invoke_command'
    from /Library/Ruby/Gems/2.0.0/gems/thor-0.18.1/lib/thor.rb:363:in `dispatch'
    from /Library/Ruby/Gems/2.0.0/gems/thor-0.18.1/lib/thor/base.rb:439:in `start'
    from /Library/Ruby/Gems/2.0.0/gems/deb-s3-0.7.1/bin/deb-s3:8:in `<top (required)>'
    from /usr/bin/deb-s3:23:in `load'
    from /usr/bin/deb-s3:23:in `<main>'

Concurrency issue

Is there some locking when multiple deb-s3 upload debs to the same repo?
I recently ended with a corrupt repo due hash-sum errors from accidently runnning multiple deb-s3 instances.

Maybe a temporary .lock file on the s3 would be enought to defer other deb-s3.

config file

Be nice is deb-s3 had a config file with the default bucket name, aws keys, etc.

Filenames with versions are wrong

If the filename has a + in it which some packages use in their versions deb-s3 writes the packages wrong since it look like it switches the + with the urlencoded version

see the following

Package: salt-cloud
Version: 2014.1.7+ds-1trusty1
License: unknown
Vendor: none
Architecture: all
Maintainer: Joe Healy <[email protected]>
Installed-Size: 99
Depends: python:any, salt-common (= 2014.1.7+ds-1trusty1), sshpass
Section: admin
Priority: extra
Homepage: http://saltstack.org/
Filename: pool/s/sa/salt-cloud_2014.1.7%2Bds-1trusty1_all.deb

It was causing apt-get to 403 trying to get the file

Err http://company-apt-trusty.s3.amazonaws.com/ stable/main salt-common all 2014.1.7+ds-1trusty1
  403  Forbidden
Fetched 6,041 kB in 4s (1,311 kB/s)
E: Failed to fetch http://company-apt-trusty.s3.amazonaws.com/pool/s/sa/salt-minion_2014.1.7%2Bds-1trusty1_all.deb  403  Forbidden

E: Failed to fetch http://company-apt-trusty.s3.amazonaws.com/pool/s/sa/salt-common_2014.1.7%2Bds-1trusty1_all.deb  403  Forbidden

The file on the s3 bucket had the + in there

If I renamed the file to remove the + things worked

Add support for uploading debian packages to S3 using AWS CodeDeploy

CodeDeploy triggers more actions when deploying a build artifact. It uploads them to S3 which means it is within the scope of the project.
If you could provide an option to use it to upload to debian packages to S3 I'd be happy to donate a 100$ to you or to a charity of your choosing.

Sources.gz

I don't know if this is the right place for this (I'm new at managing repo's), but my machine complains when I use the repo this generated, because it can't find a Sources.gz. How do you make a Sources.gz and would this benefit from having it generate one?

Upload with preserve-versions not behaving as expected at preserving older version of a package

I am trying to create private repo in S3. So starting with just basic stuff and trying to add two debs in an empty S3 bucket. Both of the debs have the same package name but different versions.

I first issue this command ...

deb-s3 upload sample-deb-app_20131113-165611-864050148_amd64.deb --preserve-versions --arch=amd64 --access-key-id=MYKEY --prefix=smalik --secret-access-key=MYSECRET --codename=precise --component=main --visibility=private --bucket=my-s3-repo

This creates the appropriate structure in my bucket ... and using apt-get and https://github.com/krobertson/apt-s3, I can even access that repo and install that package. Then I run this next command ...

deb-s3 upload sample-deb-app_20131113-165709-087212215_amd64.deb --preserve-versions --arch=amd64 --access-key-id=MYKEY --prefix=smalik --secret-access-key=MYSECRET --codename=precise --component=main --visibility=private --bucket=my-s3-repo

This ends up over writing the single package entry in Packages file that it generates. The old deb file is still present in S3 but just no entry for it in Packages file. I was expecting that the preserve-versions flag would allow me to keep the old version.

The listing in S3 look like this after the 2nd call ...

Object  1,227   Wed Nov 13 13:29:54 EST 2013    smalik/dists/precise/Release
Object  540     Wed Nov 13 13:29:54 EST 2013    smalik/dists/precise/main/binary-amd64/Packages
Object  387     Wed Nov 13 13:29:54 EST 2013    smalik/dists/precise/main/binary-amd64/Packages.gz
Object  0       Wed Nov 13 12:48:49 EST 2013    smalik/dists/precise/main/binary-i386/Packages
Object  20      Wed Nov 13 12:48:49 EST 2013    smalik/dists/precise/main/binary-i386/Packages.gz
Object  854     Wed Nov 13 12:48:49 EST 2013    smalik/pool/115/sa/sample-deb-app_20131113-165611-864050148_amd64.deb
Object  854     Wed Nov 13 13:29:54 EST 2013    smalik/pool/115/sa/sample-deb-app_20131113-165709-087212215_amd64.deb

The DEBIAN/control file in the first package was ...

Package: sample-deb-app
Version: 20131113-165611-864050148
Maintainer: smalik
Architecture: amd64
Section: main
Priority: optional
Depends: varnish
Description: Sample Debian Service Package

In the second package it was ...

Package: sample-deb-app
Version: 20131113-165709-087212215
Maintainer: smalik
Architecture: amd64
Section: main
Priority: optional
Depends: varnish
Description: Sample Debian Service Package

Any insight/help would be appreciated

Packages larger than 5MB (default multipart chunk) are reuploaded even when unchanged

The current way of checking if a file is already present in the repository works by getting the value from the etag, and comparing it to the md5 of the file.

For files uploaded in multipart, the etag doesn't contain the md5 but should contain the md5sum of the concatenated md5sums of all parts, followed by a dash and the number of parts. [1]

The AWS SDK seems to uses a default multipart size of 5MB, and seems to use multipart uploads for files larger than 5MB.

[1] http://permalink.gmane.org/gmane.comp.file-systems.s3.s3tools/583

Add additional arch type amdhf

We are developing software that runs on amdhf as well as i386/amd64. It would be good if this was supported.

How to get the previous version using deb-s3?

deb-s3 generates Packages.gz while doing deb-s3 upload.

Suppose I am using s3 as my custom apt-repository with the package: alpha-pkg

Now, when I do sudo apt-get update, then only the latest version of alpha-pkg comes. All the previous versions are gone. Is there any way to keep the previous versions too?

This way, I can quickly switch back to any previous version of alpha-pkg

Support for Pre-Depends dependencies.

Some of our packages have Pre-Depends dependencies in the control file, these dependencies are unfortunately not picked up by deb-s3.

For an explanation about Pre-Depends vs Depends:
http://askubuntu.com/questions/83553/what-is-the-difference-between-dependencies-and-pre-depends

'deb-s3 delete' argument parsing issue?

Hi, I'm running into an issue with deb-s3's argument parsing for the delete operation. I'm not quite sure what's going on, haven't inserted a pry binding to try to dig around yet, but I wanted to at least bring this up. Here, I'm using delete with the syntax in your readme.

Just for reference, I check the version of golang-amqp from the Packages file in my s3 bucket.

┌[igneous☮localhost]-(~/chef/orc-agent-build)
└> curl -o- https://s3.amazonaws.com/famc-debian-packages/dists/stable/main/binary-amd64/Packages
Package: golang-amqp
Version: 1.0
License: unknown
Vendor: none
Architecture: amd64
Maintainer: Bucky Wolfe <[email protected]>
Installed-Size: 9237
Depends: libc6 (>= 2.4), dpkg-dev
Section: net
Priority: extra
Homepage: https://github.com/streadway/amqp
Filename: pool/g/go/golang-amqp_1.0_amd64.deb
Size: 1175260
SHA1: e8cbaaa12524afac390db8f50de9715413f27904
SHA256: f44ea5f945d7b505975dfb807cbf384b6c210c1999f73e8b00eb86ead3bb9ffb
MD5sum: b48ff7a9620adcba0b824c1d41789e3e
Description: streadway's amqp library for go,

Package: orc-agent
Version: 0.1
License: unknown
Vendor: none
Architecture: amd64
Maintainer: Bucky Wolfe <[email protected]>
Installed-Size: 4922
Depends: libc6 (>= 2.4), dpkg-dev
Section: net
Priority: extra
Homepage: https://github.com/Igneous/orc-agent
Filename: pool/o/or/orc-agent_0.1_amd64.deb
Size: 919738
SHA1: d71b45f3ce3836fc67e74b27ffc524467fde3419
SHA256: c5b39b6ee072067861a53f718e9398a3bf647456069140d96b53278075569979
MD5sum: 36b43f477572bd6582382f54e1b7bd0a
Description: amqp consumer, generic message handler for orchestration tasks

Trying to delete the golang-amqp package with deb-s3 delete.

┌[igneous☮localhost]-(~/chef/orc-agent-build)
└> deb-s3 delete --arch amd64 --bucket famc-debian-packages --versions 1.0 golang-amqp
ERROR: deb-s3 delete was called with no arguments
Usage: "deb-s3 delete PACKAGE".

Trying without the rbenv shim here, just in case it's doing something goofy.

┌[igneous☮localhost]-(~/chef/orc-agent-build)
└> ~/.rbenv/versions/2.0.0-p195/bin/deb-s3 delete --arch amd64 --bucket famc-debian-packages --versions 1.0 golang-amqp
ERROR: deb-s3 delete was called with no arguments
Usage: "deb-s3 delete PACKAGE".

Trying some other deb-s3 operations.

┌[igneous☮localhost]-(~/chef/orc-agent-build)
└> deb-s3 verify -b famc-debian-packages
>> Retrieving existing manifests
>> Checking for missing packages in: stable/main amd64
>> Checking for missing packages in: stable/main armel
>> Checking for missing packages in: stable/main i386
>> Checking for missing packages in: stable/main all

Successfully uploading a package.

┌[igneous☮localhost]-(~/chef/orc-agent-build)
└> deb-s3 upload -b famc-debian-packages orc-agent_0.1_amd64.deb
>> Retrieving existing manifests
>> Examining package file orc-agent_0.1_amd64.deb
>> Uploading packages and new manifests to S3
   -- Transferring pool/o/or/orc-agent_0.1_amd64.deb
   -- Transferring dists/stable/main/binary-amd64/Packages
   -- Transferring dists/stable/main/binary-amd64/Packages.gz
   -- Transferring dists/stable/Release
>> Update complete.

Trying to remove the same package.

┌[igneous☮localhost]-(~/chef/orc-agent-build)
└> deb-s3 delete --arch amd64 --bucket famc-debian-packages --versions 0.1 orc-agent
ERROR: deb-s3 delete was called with no arguments
Usage: "deb-s3 delete PACKAGE".

`syswrite': Connection reset by peer (Errno::ECONNRESET)

>> Retrieving existing manifests
>> Examining package file checkout-universal-checkout-web-3861-2014-07-02-22-10-10.deb_3861_amd64.deb
/usr/bin/dpkg
>> Uploading packages and new manifests to S3
   -- Transferring pool/c/ch/checkout-universal-checkout-web-3861-2014-07-02-22-10-10.deb_3861_amd64.deb
/usr/lib/ruby/1.9.1/openssl/buffering.rb:317:in `syswrite': Connection reset by peer (Errno::ECONNRESET)
    from /usr/lib/ruby/1.9.1/openssl/buffering.rb:317:in `do_write'
    from /usr/lib/ruby/1.9.1/openssl/buffering.rb:335:in `write'
    from /usr/lib/ruby/1.9.1/net/protocol.rb:199:in `write0'
    from /usr/lib/ruby/1.9.1/net/protocol.rb:173:in `block in write'
    from /usr/lib/ruby/1.9.1/net/protocol.rb:190:in `writing'
    from /usr/lib/ruby/1.9.1/net/protocol.rb:172:in `write'
    from /usr/lib/ruby/1.9.1/net/http.rb:1955:in `send_request_with_body_stream'
    from /usr/lib/ruby/1.9.1/net/http.rb:1921:in `exec'
    from /usr/lib/ruby/1.9.1/net/http.rb:1317:in `block in transport_request'
    from /usr/lib/ruby/1.9.1/net/http.rb:1316:in `catch'
    from /usr/lib/ruby/1.9.1/net/http.rb:1316:in `transport_request'
    from /usr/lib/ruby/1.9.1/net/http.rb:1293:in `request'
    from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/core/http/connection_pool.rb:340:in `request'
    from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/core/http/net_http_handler.rb:61:in `block in handle'
    from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/core/http/connection_pool.rb:129:in `session_for'
    from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/core/http/net_http_handler.rb:55:in `handle'
    from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/core/client.rb:252:in `block in make_sync_request'
    from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/core/client.rb:288:in `retry_server_errors'
    from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/core/client.rb:248:in `make_sync_request'
    from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/core/client.rb:510:in `block (2 levels) in client_request'
    from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/core/client.rb:390:in `log_client_request'
    from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/core/client.rb:476:in `block in client_request'
    from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/core/client.rb:372:in `return_or_raise'
    from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/core/client.rb:475:in `client_request'
    from (eval):3:in `upload_part'
    from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/s3/multipart_upload.rb:204:in `add_part'
    from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/s3/s3_object.rb:1722:in `block in write_with_multipart'
    from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/s3/s3_object.rb:724:in `multipart_upload'
    from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/s3/s3_object.rb:1721:in `write_with_multipart'
    from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/s3/s3_object.rb:605:in `write'
    from /usr/lib/ruby/gems/1.9.1/gems/deb-s3-0.6.2/lib/deb/s3/utils.rb:75:in `s3_store'
    from /usr/lib/ruby/gems/1.9.1/gems/deb-s3-0.6.2/lib/deb/s3/manifest.rb:85:in `block in write_to_s3'
    from /usr/lib/ruby/gems/1.9.1/gems/deb-s3-0.6.2/lib/deb/s3/manifest.rb:82:in `each'
    from /usr/lib/ruby/gems/1.9.1/gems/deb-s3-0.6.2/lib/deb/s3/manifest.rb:82:in `write_to_s3'
    from /usr/lib/ruby/gems/1.9.1/gems/deb-s3-0.6.2/lib/deb/s3/cli.rb:141:in `block in upload'
    from /usr/lib/ruby/gems/1.9.1/gems/deb-s3-0.6.2/lib/deb/s3/cli.rb:140:in `each_value'
    from /usr/lib/ruby/gems/1.9.1/gems/deb-s3-0.6.2/lib/deb/s3/cli.rb:140:in `upload'
    from /usr/lib/ruby/gems/1.9.1/gems/thor-0.18.1/lib/thor/command.rb:27:in `run'
    from /usr/lib/ruby/gems/1.9.1/gems/thor-0.18.1/lib/thor/invocation.rb:120:in `invoke_command'
    from /usr/lib/ruby/gems/1.9.1/gems/thor-0.18.1/lib/thor.rb:363:in `dispatch'
    from /usr/lib/ruby/gems/1.9.1/gems/thor-0.18.1/lib/thor/base.rb:439:in `start'
    from /usr/lib/ruby/gems/1.9.1/gems/deb-s3-0.6.2/bin/deb-s3:8:in `<top (required)>'
    from /usr/bin/deb-s3:23:in `load'
    from /usr/bin/deb-s3:23:in `<main>'

i18n: package info with utf8 chars prevents upload

/var/lib/gems/1.9.1/gems/deb-s3-0.6.1/lib/deb/s3/package.rb:237:in `[]': invalid byte sequence in US-ASCII (ArgumentError)

I'm trying to upload the latest nodejs deb in my s3 repo.
In the Maintainer field there are some utf8 chars.

Less output:

nodejs_0.10.28-1chl1~quantal1_i386.deb:
 new debian package, version 2.0.
 size 5619768 bytes: control archive=34984 bytes.
      25 bytes,     1 lines      conffiles            
     861 bytes,    22 lines      control              
  122444 bytes,  1283 lines      md5sums              
     282 bytes,    17 lines   *  postinst             #!/bin/sh
     240 bytes,    13 lines   *  prerm                #!/bin/sh
 Package: nodejs
 Version: 0.10.28-1chl1~quantal1
 Architecture: i386
 Maintainer: J<C3><A9>r<C3><A9>my Lal <[email protected]>
 Installed-Size: 16654
 Depends: libc6 (>= 2.15), libgcc1 (>= 1:4.1.1), libstdc++6 (>= 4.1.1), rlwrap
 Conflicts: nodejs-dev, npm
 Replaces: nodejs-dev, npm
 Provides: nodejs-dev, npm
 Section: web
 Priority: extra
 Homepage: http://nodejs.org/

Fatal error: Start tag expected, '<' not found at :1.

Hi,

I get this when I try to upload a package to my empty bucket.

On the blogpost it says that if it's the first package, all the necessary files will be created. Is this accurate? Or does a repository structure need be created before hand (e.g. via reprepro)?

Authenticate to AWS using custom profile

Allow users to provide a profile name to use when authenticating against AWS

Locking fails when uploading without an architecture type

Steps to reproduce:

Add a sleep(60) here: https://github.com/krobertson/deb-s3/blob/master/lib/deb/s3/cli.rb#L233
Run deb-s3 upload -l -p -b your-bucket -v private your-package.deb
While it's sleeping in your-bucket/dists/stable/main you'll discover a binary- folder holding the lockfile

The fix should be as simple as getting a list of all arch types up front, but this is complicated. Reasons:

A user can specify an arch type, this over rides the arch type on the package. #95
Manifests with no packages to upload, are still being uploaded. #96

`block in pull_event': Undefined prefix Net: found (REXML::UndefinedNamespaceException)

I'm trying to upload my first package to brand new bucket and I get the following error:

>> Retrieving existing manifests
/home/timfall/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/rexml/parsers/baseparser.rb:406:in `block in pull_event': Undefined prefix Net: found (REXML::UndefinedNamespaceException)
    from /home/timfall/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/set.rb:222:in `block in each'
    from /home/timfall/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/set.rb:222:in `each_key'
    from /home/timfall/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/set.rb:222:in `each'
    from /home/timfall/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/rexml/parsers/baseparser.rb:404:in `pull_event'
    from /home/timfall/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/rexml/parsers/baseparser.rb:183:in `pull'
    from /home/timfall/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/rexml/parsers/treeparser.rb:22:in `parse'
    from /home/timfall/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/rexml/document.rb:245:in `build'
    from /home/timfall/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/rexml/document.rb:43:in `initialize'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/xml-simple-1.1.2/lib/xmlsimple.rb:965:in `new'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/xml-simple-1.1.2/lib/xmlsimple.rb:965:in `parse'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/xml-simple-1.1.2/lib/xmlsimple.rb:164:in `xml_in'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/xml-simple-1.1.2/lib/xmlsimple.rb:203:in `xml_in'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/parsing.rb:64:in `parse'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/parsing.rb:55:in `initialize'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/response.rb:55:in `new'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/response.rb:55:in `parsed'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/extensions.rb:183:in `block in parsed'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/extensions.rb:152:in `expirable_memoize'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/extensions.rb:182:in `parsed'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/response.rb:45:in `error?'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/base.rb:72:in `request'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/base.rb:88:in `get'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/object.rb:134:in `value'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/object.rb:138:in `stream'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/lib/deb/s3/utils.rb:49:in `s3_read'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/lib/deb/s3/release.rb:23:in `retrieve'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/lib/deb/s3/cli.rb:102:in `upload'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/thor-0.18.1/lib/thor/command.rb:27:in `run'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/thor-0.18.1/lib/thor/invocation.rb:120:in `invoke_command'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/thor-0.18.1/lib/thor.rb:363:in `dispatch'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/thor-0.18.1/lib/thor/base.rb:439:in `start'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/bin/deb-s3:8:in `<top (required)>'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/bin/deb-s3:23:in `load'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/bin/deb-s3:23:in `<main>'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/bin/ruby_executable_hooks:15:in `eval'
    from /home/timfall/.rvm/gems/ruby-1.9.3-p448/bin/ruby_executable_hooks:15:in `<main>'

Support concurrent uploads

deb-s3 upload currently bottlenecks our CI builds while uploading our 2.1 GB of packages to S3. (Yes, we're working on reducing that.)

It can't be run concurrently (uploading manifests clobber each other), so we've throttled the uploads to go one at a time. However, at 10 minutes per upload, that caps us to 6 builds per hour. Add in 40 developers, and Jenkins routinely gets backed up like whoa.

It sure would be nice if we could parallelize the 2.1 GB uploads to /pool (which should be safe for unique package versions), but single-track the mere MB of manifest uploads to /dists!

Here's a proof-of-concept that seems to be working well for us: https://github.com/htmldoug/deb-s3/commit/24338da032b77c63e5617d49c882f4b8bdb68631. Running deb-s3 upload --manifest_path /workspace is safe to be done concurrently and uploads /pool, but dumps the manifest additions to /workspace/dists/... instead of S3. A second deb-s3 append_manifest /workspace call can be run when under mutual exclusion to pull the fresh manifest and add the updates. That drops our bottleneck to 3 minutes which is a nice throughput increase from 6/hr to 20/hr.

No pull request yet because my ruby's rusty, I'm time crunched, and my changes have probably introduced bugs in the features we don't use (particularly --arch all).

I may find some time later to spruce this up to your standards if you're interested in adopting our approach. Or if you can come up with something even better, we'll switch to that!

delete a package

Doesn't look like you can delete a package from the repo. Be nice to add.

Package overwrite in pool between distribution

In our case we are building package for 3 distributions: wheezy, precise, trusty.

Packages have same version.

When we are uploading them, everything is ok in every "package" file.. but bad things happen in pool directory: only last file uploaded take place and every distribution other distributions point to this one...

Avoid overwriting old versions

It would be useful if you needed to pass --force-overwrite (or similar) to overwrite existing versions of packages.

(Alternatively, have a --no-overwrite if you disagree about the default.)