rainforestapp / fourchette Goto Github PK
View Code? Open in Web Editor NEWDEPRECATED - Your new best friend for isolated testing environments on Heroku.
License: MIT License
DEPRECATED - Your new best friend for isolated testing environments on Heroku.
License: MIT License
Doc: http://ruby-doc.org/stdlib-2.0/libdoc/observer/rdoc/Observable.html
Replace how before and after are done right now which is very primitive and not flexible enough.
That would be cool to be able to "talk" to Fourchette and ask things:
Idea stolen from @ukd1 from a couple weeks ago.
Hi, I followed the easy instruction path to setup Fourchette through Heroku directly. All of my config looks correct—repo, base Heroku app to fork from, API keys, etc.
I committed some code and opened a PR in a branch. First problem, the webhook setup on Github pointed to the Heroku app to fork from, not the Fourchette app. I changed the webhook to point to my Fourchette app, made another commit, and a new app was forked. However:
Anything obvious I'm missing?
Looks like trying to add pgbackups
now results in No such add-on plan
preventing db copy. I'm not sure if there's another way to do it, the Platform API doesn't seem to have anything.
First and foremost, excellent work on this gem. I've been thinking about different ways to isolate + automate our testing environments recently and this implementation was spot on. Very excited to have found the project.
I've got a general question related to the forked app's postgres database though. It looks like the database is cloned ok; pgbackups
copies over the correct data from the sourceapp
's database to the newly forked app without issue. However what I've noticed is that Heroku copies over the sourceapp
's DATABASE_URL
config variable as well, resulting in the forked app sharing the same database as the source app. I'm aware that copying over the config vars is a part of heroku's fork command, so nothing specific to fourchette there. What i'm curious about is if/how you are handling this to isolate the forked apps database? What we've done is manually pg:promote
the forked apps database so it replaces the cloned DATABASE_URL
, and that seems to solve the issue, though it feels off. So few questions..
pg:promote
)?pg:promote
the forked db?Again, great job on this gem! Looking forward to using it more!
Thanks!
Hey all,
I'm running into the following error when trying to run both the manual and the 'easy' method of installing.
I, [2014-10-30T19:33:53.383927 #3] INFO -- : Enabling the hooks for your app... rake aborted! Octokit::NotFound: GET https://api.github.com/repos/rainforestapp/fourchette/hooks: 404 - Not Found // See: https://developer.github.com/v3 /app/vendor/bundle/ruby/2.0.0/gems/octokit-3.3.1/lib/octokit/response/raise_error.rb:16:in
on_complete'
/app/vendor/bundle/ruby/2.0.0/gems/faraday-0.9.0/lib/faraday/response.rb:9:in block in call' /app/vendor/bundle/ruby/2.0.0/gems/faraday-0.9.0/lib/faraday/response.rb:57:in
on_complete'
/app/vendor/bundle/ruby/2.0.0/gems/faraday-0.9.0/lib/faraday/response.rb:8:in call' /app/vendor/bundle/ruby/2.0.0/gems/faraday-0.9.0/lib/faraday/rack_builder.rb:139:in
build_response'
/app/vendor/bundle/ruby/2.0.0/gems/faraday-0.9.0/lib/faraday/connection.rb:377:in run_request' /app/vendor/bundle/ruby/2.0.0/gems/faraday-0.9.0/lib/faraday/connection.rb:140:in
get'
/app/vendor/bundle/ruby/2.0.0/gems/sawyer-0.5.5/lib/sawyer/agent.rb:94:in call' /app/vendor/bundle/ruby/2.0.0/gems/octokit-3.3.1/lib/octokit/client.rb:339:in
request'
/app/vendor/bundle/ruby/2.0.0/gems/octokit-3.3.1/lib/octokit/client.rb:200:in paginate' /app/vendor/bundle/ruby/2.0.0/gems/octokit-3.3.1/lib/octokit/client/repositories.rb:494:in
hooks'
/app/vendor/bundle/ruby/2.0.0/gems/fourchette-0.1.1/lib/fourchette/github.rb:60:in hooks' /app/vendor/bundle/ruby/2.0.0/gems/fourchette-0.1.1/lib/fourchette/github.rb:66:in
fourchette_hook'
/app/vendor/bundle/ruby/2.0.0/gems/fourchette-0.1.1/lib/fourchette/github.rb:6:in enable_hook' /app/vendor/bundle/ruby/2.0.0/gems/fourchette-0.1.1/lib/fourchette/rake_tasks.rb:6:in
block (2 levels) in <top (required)>'
Tasks: TOP => fourchette:enable
(See full trace by running task with --trace)`
This happens during the Run scripts & scale dynos
step in the easy section or when I try to run bundle exec rake fourchette:enable
after manual install.
Is this something that I've misconfigured or is this a bug with Fourchette? Thanks in advance for any tips!
In fact, we should make it so that we do it once, on enabling Fourchette.
Right now, Fourchette uses Heroics 0.0.2. It does not work with the latest one (probably not a big deal, just did not try for more than 30 seconds).
Also, the fine folks at Heroku worked on https://github.com/heroku/platform-api which should be almost a drop in replacement, besides the use of OAuth. We should look at what is involved and do the move.
It looks like fourchette currently pulls down the GitHub repo and then uploads it to the Heroku Build API, but the build API supports grabbing the tarball directly from GitHub. For private repos, you could use GitHub's temporary authenticated archive link:
https://developer.github.com/v3/repos/contents/#get-archive-link
/cc @friism
This should have been done before, but I think Fourchette should have been 1.0+ for a "long time" now...
Thoughts?
I started to use the GitHub API to get a temporary link to a tarball of the branch we want a Heroku fork for. The downside is that if you use submodules, you're screwed as it's not part of the tarball (which makes sense I think).
Possible solutions:
The fine folks @ Heroku are aware of that, waiting on them but I am not sure I'll wait for them before implementing something better.
There was a lot of exploration to do at the beginning, but it is now starting to take a better and more stable form, so it's time to add more testing here.
I think it would make more sense (at least to us) not to fork an app but instead use app.json to create a new app, and seed our database ourselves.
I am not sure how many people rely on Fourchette and the fact that it is forking apps vs the number of people who would be fine using app.json?
Yes, we could support both, maybe it's not worth it though.
Thoughts?
Supporting multiple apps per fourchette instance would be cool. Thoughts?
Right now, we have the full output of the tar command, this is annoying and polluting the logs.
We had a push to Heroku failing due to a yanked gem and it did not raise any exception AFAIK. Needs to be double checked then fixed.
Maybe just not thread-safe?
Needs investigation.
Right now, if you have "deployhooks:http" as an addon, it is not copied over as we get this error:
, [2014-03-31T11:21:14.418651 #17330] ERROR -- : Expected([200, 201, 202, 206, 304]) <=> Actual(422 Unprocessable Entity) (Excon::Errors::UnprocessableEntity)
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/excon-0.31.0/lib/excon/middlewares/expects.rb:6:in `response_call'
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/excon-0.31.0/lib/excon/middlewares/response_parser.rb:26:in `response_call'
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/excon-0.31.0/lib/excon/connection.rb:398:in `response'
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/excon-0.31.0/lib/excon/connection.rb:268:in `request'
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/heroics-0.0.2/lib/heroics/link.rb:60:in `run'
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/heroics-0.0.2/lib/heroics/resource.rb:27:in `method_missing'
/Users/jipiboily/code/fourchette/lib/fourchette/heroku.rb:65:in `block in copy_add_ons'
/Users/jipiboily/code/fourchette/lib/fourchette/heroku.rb:60:in `each'
/Users/jipiboily/code/fourchette/lib/fourchette/heroku.rb:60:in `copy_add_ons'
/Users/jipiboily/code/fourchette/lib/fourchette/heroku.rb:11:in `fork'
/Users/jipiboily/code/fourchette/lib/fourchette/fork.rb:82:in `create_unless_exists'
/Users/jipiboily/code/fourchette/lib/fourchette/fork.rb:64:in `create'
/Users/jipiboily/code/fourchette/lib/fourchette/pull_request.rb:16:in `perform'
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25:in `public_send'
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25:in `dispatch'
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:67:in `dispatch'
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/celluloid-0.15.2/lib/celluloid/actor.rb:322:in `block in handle_message'
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/celluloid-0.15.2/lib/celluloid/actor.rb:416:in `block in task'
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/celluloid-0.15.2/lib/celluloid/tasks.rb:55:in `block in initialize'
/usr/local/opt/rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/celluloid-0.15.2/lib/celluloid/tasks/task_fiber.rb:13:in `block in create'
It is crashing on:
# in heroku.rb
client.addon.create(to, { plan: name })
While catched, this is not ideal.
Let's find a how to create it with the correct configuration.
I swapped in RSpec 3 and only one test actually fails, but there are deprecation warnings everywhere due to "should" syntax.
I can work on this if you'd like.
Please consider using the new (prototype) build interface in the API (https://devcenter.heroku.com/articles/platform-api-reference#build-create) rather than the SSH/Git dance.
If seems that killing the fork can sometime kills the DB before the web processes and dynos. This causes a bunch on exceptions to be raised, which pollutes our exception tracker.
A solution would be to first scale down all the dynos to 0, then kill the fork.
Heroku's database transfer hasn't been super reliable for us recently. I reached out to Heroku and they suggested to me to use the new way of doing that, which is a new (at least different) API from what we were using. I'll be looking into fix this. If anyone has already done this or has insights, feel free to ping me. Otherwise this should be fixed or worked around soon-ish.
It looks like this commit which adds a pg_enabled? check
assumes that the addon name will be "heroku-postgresql" however, I'm seeing it return "Heroku Postgres" for the addon name resulting in the check failing and data not being copied over. I wanted to check on this before submitting a PR. Can someone else confirm?
Generates at least those files:
Trying to fork Node.js app. Here is the log:
2014-07-21T17:20:13.830949+00:00 heroku[router]: at=info method=POST path="//hooks" host=stg-delta-web-fourchette.herokuapp.com request_id=ff6385b9-4731-4168-b936-c948cff4f656 fwd="192.30.252.35" dyno=web.1 connect=2 service=6 status=200 bytes=20943
2014-07-21T17:20:13.832214+00:00 app[web.1]: I, [2014-07-21T17:20:13.832140 #16] INFO -- : Placeholder for before steps... (see callbacks.rb to override)
2014-07-21T17:20:13.830693+00:00 app[web.1]: 192.30.252.35 - - [21/Jul/2014 17:20:13] "POST //hooks HTTP/1.1" 200 15 0.0013
2014-07-21T17:20:13.928398+00:00 app[web.1]: I, [2014-07-21T17:20:13.928265 #16] INFO -- : Creating delta-web-fourchette-pr-58
2014-07-21T17:20:14.226966+00:00 app[web.1]: I, [2014-07-21T17:20:14.226829 #16] INFO -- : Copying configs from stg-delta-web-pr to delta-web-fourchette-pr-58
2014-07-21T17:20:14.585446+00:00 app[web.1]: I, [2014-07-21T17:20:14.585332 #16] INFO -- : Copying addons from stg-delta-web-pr to delta-web-fourchette-pr-58
2014-07-21T17:20:14.685569+00:00 app[web.1]: I, [2014-07-21T17:20:14.685462 #16] INFO -- : Adding newrelic:wayne to delta-web-fourchette-pr-58
2014-07-21T17:20:16.559465+00:00 app[web.1]: I, [2014-07-21T17:20:16.559329 #16] INFO -- : Adding pgbackups:plus to delta-web-fourchette-pr-58
2014-07-21T17:20:17.017844+00:00 app[web.1]: I, [2014-07-21T17:20:17.017711 #16] INFO -- : Copying Postgres's data from stg-delta-web-pr to delta-web-fourchette-pr-58
2014-07-21T17:20:18.168417+00:00 app[web.1]: I, [2014-07-21T17:20:18.168303 #16] INFO -- : Cloning repository...
2014-07-21T17:20:19.033223+00:00 app[web.1]: I, [2014-07-21T17:20:19.033027 #16] INFO -- : Preparing tarball...
2014-07-21T17:20:19.148040+00:00 app[web.1]: I, [2014-07-21T17:20:19.147799 #16] INFO -- : Tarball to URL as a service in progress...
2014-07-21T17:20:19.141613+00:00 app[web.1]: tar: .: file changed as we read it
2014-07-21T17:20:20.035939+00:00 heroku[router]: at=info method=GET path="//Datavail/delta-front-end/791540cf-194e-4150-93c6-cde451eee640/1405963519" host=stg-delta-web-fourchette.herokuapp.com request_id=b55d3ea4-29ac-4805-a637-27a4f79575dd fwd="54.83.111.8" dyno=web.1 connect=2 service=57 status=200 bytes=505
2014-07-21T17:20:19.936448+00:00 app[web.1]: I, [2014-07-21T17:20:19.936289 #16] INFO -- : Start of the build process on Heroku...
2014-07-21T17:20:20.035214+00:00 app[web.1]: 54.83.111.8 - - [21/Jul/2014 17:20:20] "GET //Datavail/delta-front-end/791540cf-194e-4150-93c6-cde451eee640/1405963519 HTTP/1.1" 200 1071959 0.0539
2014-07-21T17:20:19.981817+00:00 app[web.1]: I, [2014-07-21T17:20:19.981707 #25] INFO -- : Serving a tarball!
2014-07-21T17:20:50.171814+00:00 app[web.1]: I, [2014-07-21T17:20:50.171627 #16] INFO -- : Placeholder for after steps... (see callbacks.rb to override)
And nothing is built at the end. Also it always add PGBackups addon to forked app and to fork, when it should not do it for node.js app.
We have some (all?) of our forks as production environment while they should be qa.
First off: excited to use this in our org—turns out we solved this same problem for our team at almost the same time you did, although yours is better in many ways! :) The @circleci folks tipped me off about fourchette.
As alluded to in #44, I need to run an after callback to copy the Postgres DB from the source Heroku app to the newly created app. Forking the DB is not an option. Looks like fourchette already does this but it fails (#42)?
In any case, in my app generated with fourchette my-app-name
, the callbacks don't appear to be running. Here's my callbacks file:
class FourchetteCallbacks
include Fourchette::Logger
def initialize(params)
@params = params
@heroku = Fourchette::Heroku.new
end
def before_all
logger.info 'Placeholder for before steps... (see callbacks.rb to override)'
end
def after_all
logger.info "Copying database from #{source_app_name} to #{new_app_name}..."
pg_backup.copy(source_app_name, new_app_name)
end
private
def heroku_fork_obj
@heroku_fork ||= Fourchette::Fork.new(@params)
end
def new_app_name
@fork_name ||= heroku_fork.fork_name
end
def source_app_name
ENV.fetch('FOURCHETTE_HEROKU_APP_TO_FORK')
end
def pg_backup
Fourchette::Pgbackups.new
end
end
I pushed my code to Heroku and opened a PR to cue creation of an app, but the logs show:
2014-11-14T02:34:05.657332+00:00 app[web.1]: I, [2014-11-14T02:34:05.657210 #23] INFO -- : Cloning repository...
2014-11-14T02:34:09.216835+00:00 app[web.1]: I, [2014-11-14T02:34:09.216679 #23] INFO -- : Preparing tarball...
2014-11-14T02:34:10.148992+00:00 app[web.1]: tar: .: file changed as we read it
2014-11-14T02:34:10.153502+00:00 app[web.1]: I, [2014-11-14T02:34:10.153329 #23] INFO -- : Tarball to URL as a service in progress...
2014-11-14T02:34:11.032727+00:00 app[web.1]: I, [2014-11-14T02:34:11.032622 #23] INFO -- : Start of the build process on Heroku...
2014-11-14T02:34:11.069947+00:00 app[web.1]: I, [2014-11-14T02:34:11.069797 #7] INFO -- : Serving a tarball!
2014-11-14T02:34:41.254423+00:00 app[web.1]: I, [2014-11-14T02:34:41.254280 #23] INFO -- : Placeholder for after steps...
...
Looks like the default after callback in the gem is running. Assuming the callbacks defined in my app should override the gem's, but that doesn't appear to be the case.
While trying to roll out fourchette solution for a company i work in, i've found out that heroku has released Review App functionality (currently in Beta).
https://blog.heroku.com/archives/2015/5/19/heroku_review_apps_beta
It appears, that this project will become obsolete once this functionality will become stable?
I'm about to follow the deployment instructions for the first time, and I found myself pining for a Deploy to Heroku button :)
https://devcenter.heroku.com/articles/heroku-button
Fourchette is successfully a building a copies of a Heroku app, but one thing it misses is scaling dynos to match the source app. Our Github project's Procfile
specifies multiple web and worker dynos, but it seems dynos are setup after the slug is built. For example, here is the dyno list of an app which is still building:
2.1.4 (main)> client.formation.list('my-new-app)
=> []
Whereas here's the dyno list on a built app:
2.1.4 (main)> client.formation.list('source-app')
=> [
[0] {
"command" => "bundle exec sidekiq -e $RACK_ENV -c $SIDEKIQ_CONCURRENCY",
"created_at" => "2014-11-19T00:48:41Z",
"id" => "da5db655-d09c-427b-ba90-9bc5e19191ba",
"type" => "worker",
"quantity" => 2,
"size" => "1X",
"updated_at" => "2014-11-19T00:48:41Z"
},
[1] {
"command" => "bundle exec unicorn -p $PORT -E $RACK_ENV -c ./config/unicorn.rb",
"created_at" => "2014-11-19T00:48:41Z",
"id" => "218b2b07-4d25-4059-8e4e-fe74989e1f36",
"type" => "web",
"quantity" => 2,
"size" => "1X",
"updated_at" => "2014-11-19T00:48:41Z"
},
[2] {
"command" => "bin/rails console",
"created_at" => "2014-11-19T00:48:41Z",
"id" => "ec43e3ba-de4a-474b-99d9-3a8be50923fd",
"type" => "console",
"quantity" => 0,
"size" => "1X",
"updated_at" => "2014-11-19T00:48:41Z"
},
[3] {
"command" => "bundle exec rake",
"created_at" => "2014-11-19T00:48:41Z",
"id" => "955369fd-0c1b-4529-a700-8ced3d382a55",
"type" => "rake",
"quantity" => 0,
"size" => "1X",
"updated_at" => "2014-11-19T00:48:41Z"
}
]
Is it possible to setup these values before/during the build using the Platform API?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.