Hey,
I just tried using middelman-sync and it struck me as odd that sync.after_build needs to be true to work. Is this intentional? Am I missing something?

Currently it will sync your build to /build on S3. Not ideal.

When syncing to S3, all files are copied each time. Deleted files are not cleaned up. I'm not sure if the fix needs to be in middleman-sync or asset_sync.

The problem is get_remote_files() in lib/asset_sync/storage.rb always returns a blank list of files. This can be fixed by commenting out the :prefix option in bucket()

def bucket
  # fixes: https://github.com/rumblelabs/asset_sync/issues/18
  @bucket ||= connection.directories.get(self.config.fog_directory)#, :prefix => self.config.assets_prefix)
end

The :prefix option is obviously needed in other situations, so a solution upstream is needed. Maybe something with middleman-sync options.prefix = "**"? I'm getting over my head at this point.

I'm happy to dig further and submit a pull request, but I need some guidance. Thanks.

I am getting an openssl error - specifically:

hostname "typkit.rwboyer.com.s3.amazonaws.com" does not match the server certificate

when attempting to use middleman sync and the same ID/KEYS as everything else I use that works including s3cmd, s3_website, etc...

Would be great if middleman sync worked for S3

I've included gem "middleman-sync", "~> 3.0.12" in config.rb but when attempting to bundle install I receive the following error:

Bundler could not find compatible versions for gem "builder":
  In Gemfile:
    middleman-sync (~> 3.0.12) ruby depends on
      builder (~> 2.1.2) ruby

    builder (3.2.2)

It's telling me that I require Builder 2.1.2 or above, but I actually have Builder 3.2.2. Anyone know what the problem is?

Hi @karlfreeman,

I'd like to start deploying my site in a two-stage process: First to a staging server for others within my organization to review, and then to a production server. Do you have any suggestions on how to do this?

Thanks!

Any hints on getting this to work? Using the README info a run into problems.

Unknown Extension: sync_source

Same with sync_target. I'm using the latest github sources and already saw that it should be: aws_target?

I appreciate this is an Asset Sync issue, but it will effect people using S3 buckets to host static sites with custom domain names.

Amazon requires that if you are setting up a bucket for a static site, it is named off your domain, including sub domains:

example.com
www.example.com

However it seems that Asset Sync and therefore Middleman Sync will fail if the bucket name contains a period with the following error:

/Users/me/.rbenv/versions/1.9.3-p392/lib/ruby/1.9.1/openssl/ssl-internal.rb:121:in `post_connection_check': hostname does not match the server certificate (OpenSSL::SSL::SSLError) (Excon::Errors::SocketError)

It seems SSL related - something to do with Amazon's SSL certificate not allowing for subdomains.

This would appear to mean that using Middleman Sync is no longer an option if the site is hosted on S3 with a custom domain name.

This would appear to be a relatively recent change. and hopefully I am jumping to the wrong conclusion.

For anyone encountering this issue and needing a quick fix, S3 Sync has no issues with periods.

Middleman 3.1.1 adds support for proxies with custom content-types: bhollis/middleman@7a4aa10

In our case we have an extensionless proxy that points to an html file:

proxy 'faq', 'faq.html'

When going to that URL on s3 the browser downloads the file because the default content type for files without an extension is application/octet=stream. I realized that this has to do with mime-type lookup coming from asset_sync:
https://github.com/rumblelabs/asset_sync/blob/master/lib/asset_sync/multi_mime.rb

We considered setting the content-type manually on s3 for the files in question, but that's not sustainable.

Since middleman uses Rack::Mime our workaround is adding a mime-type for files without an extension (which also works on middleman 3.1):

Rack::Mime::MIME_TYPES.merge!('.' => 'text/html')

When using version 0.5.2 of the asset_sync dependency, the following error occurs when syncing

uninitialized constant AssetSync::Storage::Mime (NameError)

I was able to work around this issue by adding the following to my Gemfile

# use a specific version of the asset_sync dependency of middleman-sync
gem 'asset_sync', '0.5.0'

When I run middleman build, it syncs regardless of the after_build setting in my config. I've set it to false, but it continues to sync on each build. Any ideas?

activate :sync do |sync|
  sync.fog_provider = 'AWS' # Your storage provider
  ...
  sync.after_build = false
end

I simply appended this config block to the bottom of the config.rb file. Does it need to be nested under configure :build do?

Filled in S3 credentials in config.rb and manually running middleman sync, and am getting a "You need to activate the sync extension in config.rb".

activate :sync do |sync|
  sync.fog_provider = ENV['FOG_DIRECTORY'] # Your storage provider
  sync.fog_directory = 'my_awesome_bucket' # Your bucket name
  sync.fog_region = 'us-east-1' # The region your storage bucket is in (eg us-east-1, us-west-1, eu-west-1, ap-southeast-1 )
  sync.aws_access_key_id = ENV['AWS_ACCESS_KEY_ID'] # Your Amazon S3 access key
  sync.aws_secret_access_key = ENV['AWS_SECRET_ACCESS_KEY'] # Your Amazon S3 access secret
  sync.existing_remote_files = 'delete' # What to do with your existing remote files? ( keep or delete )
  # sync.gzip_compression = false # Automatically replace files with their equivalent gzip compressed version
  # sync.after_build = false # Disable sync to run after Middleman build ( defaults to true )
end

Using Foreman and .env file to fill in the secret/key/provider.

It seems like it only syncs directories but not any files in the build directory (such as index.html, error.html, or maintenance.html). For an example see: https://github.com/heroku/custom-error-pages

Can we get a configuration flag to set S3 to use reduced redundancy storage for our files?

I want to use Sourcegraph for middleman-sync code search, browsing, and usage examples. Can an admin enable Sourcegraph for this repository? Just go to https://sourcegraph.com/github.com/middleman-contrib/middleman-sync. (It should only take 30 seconds.)

Thank you!

Syncing or building Throws the following:

/Users/username/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/asset_sync-0.5.4/lib/asset_sync/railtie.rb:1:in<top (required)>': uninitialized constant Rails::Railtie (NameError)`

asset_sync (0.5.4) 
middleman-sync (3.0.9)

Tried including Railties explicitly in the Gemfile, but this leads to a tangle of gem dependency errors.

activate :sync do |sync|
  sync.fog_provider = 'AWS' # Your storage provider
  sync.fog_directory = ENV['AWS_BUCKET'] # Your bucket name
  sync.fog_region = ENV['AWS_REGION'] # The region your storage bucket is in (eg us-east-1, us-west-1, eu-west-1, ap-southeast-1 )
  sync.aws_access_key_id = ENV['AWS_ACCESS_KEY_ID'] # Your Amazon S3 access key
  sync.aws_secret_access_key = ENV['AWS_SECRET_ACCESS_KEY'] # Your Amazon S3 access secret
  sync.existing_remote_files = 'keep' # What to do with your existing remote files? ( keep or delete )
  sync.after_build = true # Disable sync to run after Middleman build ( defaults to true )
end

Upon middleman build with the following config:

activate :sync do |sync|
  sync.fog_provider = 'AWS' 
  sync.fog_directory = ENV['BUCKET_NAME'] 
  sync.fog_region = 'us-east-1'
  sync.aws_access_key_id = ENV['AWS_ACCESS_KEY'] 
  sync.aws_secret_access_key = ENV['AWS_SECRET_KEY']
  sync.existing_remote_files = 'delete'
  sync.gzip_compression = true 
  sync.after_build = true
end

using latest middleman release (3.0.12) and latest middleman_sync, the follow error occurs:

$ middleman build
/Users/david/.rvm/gems/ruby-1.9.3-p392/gems/asset_sync-0.5.4/lib/asset_sync/config.rb:99:in `yml_path': undefined method `root' for Rails:Module (NoMethodError)

Full trace here: https://gist.github.com/dpfranklin/5052057

I know there is some work being done for this in another fork. This issue is just a reminder of it until it gets merged.

Cheers!

Would you mind pushing the gem to rubygems.org for others to use?

Thanks for this project. It looks like a perfect solution to the Middleman-to-S3 problem I've been trying to solve.

I would like to suggest you add the correct values to set the region variable to the README file. It took me a little longer than I would have liked to figure this out. Here are the ones I know:

US Standard: us-east-1
US West (Northern California): us-west-1
EU (Ireland): eu-west-1
Asia Pacific (Singapore): ap-southeast-1

RubyGems.org doesn't report a license for your gem. This is because it is not specified in the gemspec of your last release.

via e.g.

spec.license = 'MIT'
# or
spec.licenses = ['MIT', 'GPL-2']

Including a license in your gemspec is an easy way for rubygems.org and other tools to check how your gem is licensed. As you can imagine, scanning your repository for a LICENSE file or parsing the README, and then attempting to identify the license or licenses is much more difficult and more error prone. So, even for projects that already specify a license, including a license in your gemspec is a good practice. See, for example, how rubygems.org uses the gemspec to display the rails gem license.

There is even a License Finder gem to help companies/individuals ensure all gems they use meet their licensing needs. This tool depends on license information being available in the gemspec. This is an important enough issue that even Bundler now generates gems with a default 'MIT' license.

I hope you'll consider specifying a license in your gemspec. If not, please just close the issue with a nice message. In either case, I'll follow up. Thanks for your time!

Appendix:

If you need help choosing a license (sorry, I haven't checked your readme or looked for a license file), GitHub has created a license picker tool. Code without a license specified defaults to 'All rights reserved'-- denying others all rights to use of the code.
Here's a list of the license names I've found and their frequencies

p.s. In case you're wondering how I found you and why I made this issue, it's because I'm collecting stats on gems (I was originally looking for download data) and decided to collect license metadata,too, and make issues for gemspecs not specifying a license as a public service :). See the previous link or my blog post about this project for more information.

Hi Karl,

I'm getting the following error using middleman-sync:

/Users/pedr/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/middleman-sync-3.0.6/lib/middleman-sync/commands.rb:24:in sync': undefined method options' for #<Middleman::Application:0x70156794690600> (NoMethodError) from /Users/pedr/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/thor-0.15.4/lib/thor/task.rb:27:in run'
from /Users/pedr/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/thor-0.15.4/lib/thor/invocation.rb:120:in invoke_task' from /Users/pedr/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/thor-0.15.4/lib/thor.rb:275:in dispatch'
from /Users/pedr/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/thor-0.15.4/lib/thor/base.rb:425:in start' from /Users/pedr/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/middleman-core-3.0.6/lib/middleman-core/cli.rb:77:in method_missing'
from /Users/pedr/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/thor-0.15.4/lib/thor/task.rb:29:in run' from /Users/pedr/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/thor-0.15.4/lib/thor/task.rb:126:in run'
from /Users/pedr/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/thor-0.15.4/lib/thor/invocation.rb:120:in invoke_task' from /Users/pedr/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/thor-0.15.4/lib/thor.rb:275:in dispatch'
from /Users/pedr/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/thor-0.15.4/lib/thor/base.rb:425:in start' from /Users/pedr/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/middleman-core-3.0.6/lib/middleman-core/cli.rb:22:in start'
from /Users/pedr/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/middleman-core-3.0.6/bin/middleman:18:in <top (required)>' from /Users/pedr/.rbenv/versions/1.9.3-p194/bin/middleman:23:in load'
from /Users/pedr/.rbenv/versions/1.9.3-p194/bin/middleman:23:in <main>'

Since putting middleman-sync out there its been downloaded over 15,000+ times and has had spawned from it many other libs due mainly to missing features / functionality.

Considering that middleman-sync is essentially a shim for asset_sync I've been quietly impressed by how much use it's been getting. However because of its shimy' nature though its difficult to support features requested which is why middleman-sync 3.1 will be a complete re-write, sans asset_sync.

If your using / wanted to use middleman-sync please chime in on what you would like to see / whats been pain points.

Questions

• Is anyone actually using Rackspace / Google storage?
• Would globbing dir's inside build to 'sync' allow for more flexibility? Eg does anyone not sync the whole of /build to :target ?

Feature list / Requirements

• Dropping Asset_Sync, it's had a good run but maintainability and future development is too closely tied to what is an excellant sprockets / rails asset synchronisation library.
• S3 sync strategy with RRS and CORS support
• Strategy independent sync options ( existing_remote_files / gzip_compression / ignore_files ).
• Testing, robust testing. Cucumber seems to be the middleman choice, not sure myself. Perhaps a combination so that strategies are tested fully too.

Wishlist

• Perhaps change up the idea of syncing the whole /build but instead support multiple build directories by specifying a glob for files within /build instead.
• Multiple :sync configurations, say your fonts are Cloudfronted on S3 but the rest live somewhere else.
• Actually use Middleman's thor logger
• Other fog storage providers sync strategies.
• Allow for more strategies beyond fog

I'm getting an error when running middleman sync to sync with S3:

AssetSync: Syncing.
Invalid Excon request keys: :scheme, :host

Files are uploading, but the error still shows up.