rainforestapp / fourchette Goto Github PK

Doc: http://ruby-doc.org/stdlib-2.0/libdoc/observer/rdoc/Observable.html

Replace how before and after are done right now which is very primitive and not flexible enough.

That would be cool to be able to "talk" to Fourchette and ask things:

• "Fourchette: restart fork"
• "Fourchette: delete fork"
• "Fourchette: scale web=1 worker=3"

Idea stolen from @ukd1 from a couple weeks ago.

Hi, I followed the easy instruction path to setup Fourchette through Heroku directly. All of my config looks correct—repo, base Heroku app to fork from, API keys, etc.

I committed some code and opened a PR in a branch. First problem, the webhook setup on Github pointed to the Heroku app to fork from, not the Fourchette app. I changed the webhook to point to my Fourchette app, made another commit, and a new app was forked. However:

• it didn't copy dyno and worker settings. In fact, there are no dynos and I can't add any after initial creation it appears.
• there appears to be no code there.

Anything obvious I'm missing?

Looks like trying to add pgbackups now results in No such add-on plan preventing db copy. I'm not sure if there's another way to do it, the Platform API doesn't seem to have anything.

Question about isolating forked databases

First and foremost, excellent work on this gem. I've been thinking about different ways to isolate + automate our testing environments recently and this implementation was spot on. Very excited to have found the project.

I've got a general question related to the forked app's postgres database though. It looks like the database is cloned ok; pgbackups copies over the correct data from the sourceapp's database to the newly forked app without issue. However what I've noticed is that Heroku copies over the sourceapp's DATABASE_URL config variable as well, resulting in the forked app sharing the same database as the source app. I'm aware that copying over the config vars is a part of heroku's fork command, so nothing specific to fourchette there. What i'm curious about is if/how you are handling this to isolate the forked apps database? What we've done is manually pg:promote the forked apps database so it replaces the cloned DATABASE_URL, and that seems to solve the issue, though it feels off. So few questions..

• Should fourchette automatically promote the forked database as well (using pg:promote)?
• Is it expected to have to pg:promote the forked db?
• Is anyone else running into this? If so, how are you handling it?

Again, great job on this gem! Looking forward to using it more!

Thanks!

Hey all,

I'm running into the following error when trying to run both the manual and the 'easy' method of installing.
I, [2014-10-30T19:33:53.383927 #3] INFO -- : Enabling the hooks for your app... rake aborted! Octokit::NotFound: GET https://api.github.com/repos/rainforestapp/fourchette/hooks: 404 - Not Found // See: https://developer.github.com/v3 /app/vendor/bundle/ruby/2.0.0/gems/octokit-3.3.1/lib/octokit/response/raise_error.rb:16:in on_complete'
/app/vendor/bundle/ruby/2.0.0/gems/faraday-0.9.0/lib/faraday/response.rb:9:in block in call' /app/vendor/bundle/ruby/2.0.0/gems/faraday-0.9.0/lib/faraday/response.rb:57:in on_complete'
/app/vendor/bundle/ruby/2.0.0/gems/faraday-0.9.0/lib/faraday/response.rb:8:in call' /app/vendor/bundle/ruby/2.0.0/gems/faraday-0.9.0/lib/faraday/rack_builder.rb:139:in build_response'
/app/vendor/bundle/ruby/2.0.0/gems/faraday-0.9.0/lib/faraday/connection.rb:377:in run_request' /app/vendor/bundle/ruby/2.0.0/gems/faraday-0.9.0/lib/faraday/connection.rb:140:in get'
/app/vendor/bundle/ruby/2.0.0/gems/sawyer-0.5.5/lib/sawyer/agent.rb:94:in call' /app/vendor/bundle/ruby/2.0.0/gems/octokit-3.3.1/lib/octokit/client.rb:339:in request'
/app/vendor/bundle/ruby/2.0.0/gems/octokit-3.3.1/lib/octokit/client.rb:200:in paginate' /app/vendor/bundle/ruby/2.0.0/gems/octokit-3.3.1/lib/octokit/client/repositories.rb:494:in hooks'
/app/vendor/bundle/ruby/2.0.0/gems/fourchette-0.1.1/lib/fourchette/github.rb:60:in hooks' /app/vendor/bundle/ruby/2.0.0/gems/fourchette-0.1.1/lib/fourchette/github.rb:66:in fourchette_hook'
/app/vendor/bundle/ruby/2.0.0/gems/fourchette-0.1.1/lib/fourchette/github.rb:6:in enable_hook' /app/vendor/bundle/ruby/2.0.0/gems/fourchette-0.1.1/lib/fourchette/rake_tasks.rb:6:in block (2 levels) in <top (required)>'
Tasks: TOP => fourchette:enable
(See full trace by running task with --trace)`

This happens during the Run scripts & scale dynos step in the easy section or when I try to run bundle exec rake fourchette:enable after manual install.

Is this something that I've misconfigured or is this a bug with Fourchette? Thanks in advance for any tips!

In fact, we should make it so that we do it once, on enabling Fourchette.

Right now, Fourchette uses Heroics 0.0.2. It does not work with the latest one (probably not a big deal, just did not try for more than 30 seconds).

Also, the fine folks at Heroku worked on https://github.com/heroku/platform-api which should be almost a drop in replacement, besides the use of OAuth. We should look at what is involved and do the move.

It looks like fourchette currently pulls down the GitHub repo and then uploads it to the Heroku Build API, but the build API supports grabbing the tarball directly from GitHub. For private repos, you could use GitHub's temporary authenticated archive link:

https://developer.github.com/v3/repos/contents/#get-archive-link

/cc @friism

This should have been done before, but I think Fourchette should have been 1.0+ for a "long time" now...

Thoughts?

I started to use the GitHub API to get a temporary link to a tarball of the branch we want a Heroku fork for. The downside is that if you use submodules, you're screwed as it's not part of the tarball (which makes sense I think).

Possible solutions:

• do a clone, make a tarball of it and publicly available for Heroku for a few seconds at some random an long URL + with Heroku token or something like that
  or
• have the same thing, but as a different and optional app, a github-to-tarball as a service sort of thing.
  or
• have Heroku support Git + submodules while cloning it.

The fine folks @ Heroku are aware of that, waiting on them but I am not sure I'll wait for them before implementing something better.

There was a lot of exploration to do at the beginning, but it is now starting to take a better and more stable form, so it's time to add more testing here.

I think it would make more sense (at least to us) not to fork an app but instead use app.json to create a new app, and seed our database ourselves.

I am not sure how many people rely on Fourchette and the fact that it is forking apps vs the number of people who would be fine using app.json?

Yes, we could support both, maybe it's not worth it though.

Thoughts?

/cc @smathieu @ukd1 @markpundsack @friism @bdarfler

Supporting multiple apps per fourchette instance would be cool. Thoughts?

Right now, we have the full output of the tar command, this is annoying and polluting the logs.

We had a push to Heroku failing due to a yanked gem and it did not raise any exception AFAIK. Needs to be double checked then fixed.

Maybe just not thread-safe?

Needs investigation.

See later posts in #10, https://github.com/heroku/heroku/blob/master/lib/heroku/command/pgbackups.rb#L266 and https://github.com/heroku/heroku/blob/master/lib/heroku/client/pgbackups.rb.

Right now, if you have "deployhooks:http" as an addon, it is not copied over as we get this error:

, [2014-03-31T11:21:14.418651 #17330] ERROR -- : Expected([200, 201, 202, 206, 304]) <=> Actual(422 Unprocessable Entity) (Excon::Errors::UnprocessableEntity)
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/excon-0.31.0/lib/excon/middlewares/expects.rb:6:in `response_call'
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/excon-0.31.0/lib/excon/middlewares/response_parser.rb:26:in `response_call'
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/excon-0.31.0/lib/excon/connection.rb:398:in `response'
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/excon-0.31.0/lib/excon/connection.rb:268:in `request'
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/heroics-0.0.2/lib/heroics/link.rb:60:in `run'
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/heroics-0.0.2/lib/heroics/resource.rb:27:in `method_missing'
/Users/jipiboily/code/fourchette/lib/fourchette/heroku.rb:65:in `block in copy_add_ons'
/Users/jipiboily/code/fourchette/lib/fourchette/heroku.rb:60:in `each'
/Users/jipiboily/code/fourchette/lib/fourchette/heroku.rb:60:in `copy_add_ons'
/Users/jipiboily/code/fourchette/lib/fourchette/heroku.rb:11:in `fork'
/Users/jipiboily/code/fourchette/lib/fourchette/fork.rb:82:in `create_unless_exists'
/Users/jipiboily/code/fourchette/lib/fourchette/fork.rb:64:in `create'
/Users/jipiboily/code/fourchette/lib/fourchette/pull_request.rb:16:in `perform'
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25:in `public_send'
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25:in `dispatch'
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:67:in `dispatch'
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/celluloid-0.15.2/lib/celluloid/actor.rb:322:in `block in handle_message'
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/celluloid-0.15.2/lib/celluloid/actor.rb:416:in `block in task'
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/celluloid-0.15.2/lib/celluloid/tasks.rb:55:in `block in initialize'
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/celluloid-0.15.2/lib/celluloid/tasks/task_fiber.rb:13:in `block in create'

It is crashing on:

# in heroku.rb
client.addon.create(to, { plan: name })

While catched, this is not ideal.

Let's find a how to create it with the correct configuration.

• implement some minimal styling
• implement some sort of asset management (possibly https://github.com/rstacruz/sinatra-assetpack)
• implement usage of OAuth for GitHub
• implement usage of OAuth for Heroku
• profit?

I swapped in RSpec 3 and only one test actually fails, but there are deprecation warnings everywhere due to "should" syntax.

I can work on this if you'd like.

Please consider using the new (prototype) build interface in the API (https://devcenter.heroku.com/articles/platform-api-reference#build-create) rather than the SSH/Git dance.

If seems that killing the fork can sometime kills the DB before the web processes and dynos. This causes a bunch on exceptions to be raised, which pollutes our exception tracker.

A solution would be to first scale down all the dynos to 0, then kill the fork.

Heroku's database transfer hasn't been super reliable for us recently. I reached out to Heroku and they suggested to me to use the new way of doing that, which is a new (at least different) API from what we were using. I'll be looking into fix this. If anyone has already done this or has insights, feel free to ping me. Otherwise this should be fixed or worked around soon-ish.

Conditional check for Postgres seems to be incorrect?

It looks like this commit which adds a pg_enabled? check assumes that the addon name will be "heroku-postgresql" however, I'm seeing it return "Heroku Postgres" for the addon name resulting in the check failing and data not being copied over. I wanted to check on this before submitting a PR. Can someone else confirm?

Generates at least those files:

• config.ru
• Rakefile
• Procfile
• Gemfile

Trying to fork Node.js app. Here is the log:

2014-07-21T17:20:13.830949+00:00 heroku[router]: at=info method=POST path="//hooks" host=stg-delta-web-fourchette.herokuapp.com request_id=ff6385b9-4731-4168-b936-c948cff4f656 fwd="192.30.252.35" dyno=web.1 connect=2 service=6 status=200 bytes=20943
2014-07-21T17:20:13.832214+00:00 app[web.1]: I, [2014-07-21T17:20:13.832140 #16]  INFO -- : Placeholder for before steps... (see callbacks.rb to override)
2014-07-21T17:20:13.830693+00:00 app[web.1]: 192.30.252.35 - - [21/Jul/2014 17:20:13] "POST //hooks HTTP/1.1" 200 15 0.0013
2014-07-21T17:20:13.928398+00:00 app[web.1]: I, [2014-07-21T17:20:13.928265 #16]  INFO -- : Creating delta-web-fourchette-pr-58
2014-07-21T17:20:14.226966+00:00 app[web.1]: I, [2014-07-21T17:20:14.226829 #16]  INFO -- : Copying configs from stg-delta-web-pr to delta-web-fourchette-pr-58
2014-07-21T17:20:14.585446+00:00 app[web.1]: I, [2014-07-21T17:20:14.585332 #16]  INFO -- : Copying addons from stg-delta-web-pr to delta-web-fourchette-pr-58
2014-07-21T17:20:14.685569+00:00 app[web.1]: I, [2014-07-21T17:20:14.685462 #16]  INFO -- : Adding newrelic:wayne to delta-web-fourchette-pr-58
2014-07-21T17:20:16.559465+00:00 app[web.1]: I, [2014-07-21T17:20:16.559329 #16]  INFO -- : Adding pgbackups:plus to delta-web-fourchette-pr-58
2014-07-21T17:20:17.017844+00:00 app[web.1]: I, [2014-07-21T17:20:17.017711 #16]  INFO -- : Copying Postgres's data from stg-delta-web-pr to delta-web-fourchette-pr-58
2014-07-21T17:20:18.168417+00:00 app[web.1]: I, [2014-07-21T17:20:18.168303 #16]  INFO -- : Cloning repository...
2014-07-21T17:20:19.033223+00:00 app[web.1]: I, [2014-07-21T17:20:19.033027 #16]  INFO -- : Preparing tarball...
2014-07-21T17:20:19.148040+00:00 app[web.1]: I, [2014-07-21T17:20:19.147799 #16]  INFO -- : Tarball to URL as a service in progress...
2014-07-21T17:20:19.141613+00:00 app[web.1]: tar: .: file changed as we read it
2014-07-21T17:20:20.035939+00:00 heroku[router]: at=info method=GET path="//Datavail/delta-front-end/791540cf-194e-4150-93c6-cde451eee640/1405963519" host=stg-delta-web-fourchette.herokuapp.com request_id=b55d3ea4-29ac-4805-a637-27a4f79575dd fwd="54.83.111.8" dyno=web.1 connect=2 service=57 status=200 bytes=505
2014-07-21T17:20:19.936448+00:00 app[web.1]: I, [2014-07-21T17:20:19.936289 #16]  INFO -- : Start of the build process on Heroku...
2014-07-21T17:20:20.035214+00:00 app[web.1]: 54.83.111.8 - - [21/Jul/2014 17:20:20] "GET //Datavail/delta-front-end/791540cf-194e-4150-93c6-cde451eee640/1405963519 HTTP/1.1" 200 1071959 0.0539
2014-07-21T17:20:19.981817+00:00 app[web.1]: I, [2014-07-21T17:20:19.981707 #25]  INFO -- : Serving a tarball!
2014-07-21T17:20:50.171814+00:00 app[web.1]: I, [2014-07-21T17:20:50.171627 #16]  INFO -- : Placeholder for after steps... (see callbacks.rb to override)

And nothing is built at the end. Also it always add PGBackups addon to forked app and to fork, when it should not do it for node.js app.

We have some (all?) of our forks as production environment while they should be qa.

First off: excited to use this in our org—turns out we solved this same problem for our team at almost the same time you did, although yours is better in many ways! :) The @circleci folks tipped me off about fourchette.

As alluded to in #44, I need to run an after callback to copy the Postgres DB from the source Heroku app to the newly created app. Forking the DB is not an option. Looks like fourchette already does this but it fails (#42)?

In any case, in my app generated with fourchette my-app-name, the callbacks don't appear to be running. Here's my callbacks file:

class FourchetteCallbacks
  include Fourchette::Logger

  def initialize(params)
    @params = params
    @heroku = Fourchette::Heroku.new
  end

  def before_all
    logger.info 'Placeholder for before steps... (see callbacks.rb to override)'
  end

  def after_all
    logger.info "Copying database from #{source_app_name} to #{new_app_name}..."
    pg_backup.copy(source_app_name, new_app_name)
  end

  private

  def heroku_fork_obj
    @heroku_fork ||= Fourchette::Fork.new(@params)
  end

  def new_app_name
    @fork_name ||= heroku_fork.fork_name
  end

  def source_app_name
    ENV.fetch('FOURCHETTE_HEROKU_APP_TO_FORK')
  end

  def pg_backup
    Fourchette::Pgbackups.new
  end
end

I pushed my code to Heroku and opened a PR to cue creation of an app, but the logs show:

2014-11-14T02:34:05.657332+00:00 app[web.1]: I, [2014-11-14T02:34:05.657210 #23]  INFO -- : Cloning repository...
2014-11-14T02:34:09.216835+00:00 app[web.1]: I, [2014-11-14T02:34:09.216679 #23]  INFO -- : Preparing tarball...
2014-11-14T02:34:10.148992+00:00 app[web.1]: tar: .: file changed as we read it
2014-11-14T02:34:10.153502+00:00 app[web.1]: I, [2014-11-14T02:34:10.153329 #23]  INFO -- : Tarball to URL as a service in progress...
2014-11-14T02:34:11.032727+00:00 app[web.1]: I, [2014-11-14T02:34:11.032622 #23]  INFO -- : Start of the build process on Heroku...
2014-11-14T02:34:11.069947+00:00 app[web.1]: I, [2014-11-14T02:34:11.069797 #7]  INFO -- : Serving a tarball!
2014-11-14T02:34:41.254423+00:00 app[web.1]: I, [2014-11-14T02:34:41.254280 #23]  INFO -- : Placeholder for after steps...
...

Looks like the default after callback in the gem is running. Assuming the callbacks defined in my app should override the gem's, but that doesn't appear to be the case.

While trying to roll out fourchette solution for a company i work in, i've found out that heroku has released Review App functionality (currently in Beta).
https://blog.heroku.com/archives/2015/5/19/heroku_review_apps_beta

It appears, that this project will become obsolete once this functionality will become stable?

I'm about to follow the deployment instructions for the first time, and I found myself pining for a Deploy to Heroku button :)
https://devcenter.heroku.com/articles/heroku-button

Fourchette is successfully a building a copies of a Heroku app, but one thing it misses is scaling dynos to match the source app. Our Github project's Procfile specifies multiple web and worker dynos, but it seems dynos are setup after the slug is built. For example, here is the dyno list of an app which is still building:

2.1.4 (main)> client.formation.list('my-new-app)
=> []

Whereas here's the dyno list on a built app:

2.1.4 (main)> client.formation.list('source-app')
=> [
  [0] {
       "command" => "bundle exec sidekiq -e $RACK_ENV -c $SIDEKIQ_CONCURRENCY",
    "created_at" => "2014-11-19T00:48:41Z",
            "id" => "da5db655-d09c-427b-ba90-9bc5e19191ba",
          "type" => "worker",
      "quantity" => 2,
          "size" => "1X",
    "updated_at" => "2014-11-19T00:48:41Z"
  },
  [1] {
       "command" => "bundle exec unicorn -p $PORT -E $RACK_ENV -c ./config/unicorn.rb",
    "created_at" => "2014-11-19T00:48:41Z",
            "id" => "218b2b07-4d25-4059-8e4e-fe74989e1f36",
          "type" => "web",
      "quantity" => 2,
          "size" => "1X",
    "updated_at" => "2014-11-19T00:48:41Z"
  },
  [2] {
       "command" => "bin/rails console",
    "created_at" => "2014-11-19T00:48:41Z",
            "id" => "ec43e3ba-de4a-474b-99d9-3a8be50923fd",
          "type" => "console",
      "quantity" => 0,
          "size" => "1X",
    "updated_at" => "2014-11-19T00:48:41Z"
  },
  [3] {
       "command" => "bundle exec rake",
    "created_at" => "2014-11-19T00:48:41Z",
            "id" => "955369fd-0c1b-4529-a700-8ced3d382a55",
          "type" => "rake",
      "quantity" => 0,
          "size" => "1X",
    "updated_at" => "2014-11-19T00:48:41Z"
  }
]

Is it possible to setup these values before/during the build using the Platform API?