splitrb / split Goto Github PK

I couldn't get the alternative override to work consistently, and think I found why; when Split::Helper#ab_test looks for an override, it looks for a URL param corresponding to the experiment key, rather than experiment name, and the key sometimes contains the version:

https://github.com/jasonm/split/blob/edc42130ff7c64d9e5c0761a8e5abe4fa839a14b/lib/split/experiment.rb#L51-57

I've changed the override code to look for the experiment name instead of key:

jasonm@edc4213

I just spiked this out, and would like to provide test coverage, but wanted to make sure I understood the issue correctly first.

Thanks!

At the moment every alternative is given an equal share of the users, if there are two alternatives then each should receive 50% of users, if there are 3 then 33.3% each etc

If you are introducing a new feature and want to test that it improves conversion, but you are unsure of the effects, maybe you only want 10% of the traffic to use that alternative whilst the other 90% go through the control, you should be able to specify the share when defining the test.

in recent rails a session is created lazily - that is it's created when the session is accessed.
if there is an ab_test in a view and no session is present, then the same user could be presented with different ab tests in the same "session".
(i.e. e-commerce site and basket is empty)

Hi,

this is a bug that I have not been able to reproduce because it does not happen every time. Basically when I reset and experiment the version number keeps incrementing and never stops...
To fix the problem I had to delete the experiment and recreate it.

Any idea where this could come from?
Greg

Hi,
first of all, thank you for split, it is great and looks very promising!

We have been trying to use it in our startup but we have found a stupid problem (I hope...). We have defined the a/b test in a view:

= ab_test('request_text', t('home.public_overlay.join'), t('home.public_overlay.join_alternative'))

and out finished test is marked in a controller action, reachable by a button near the previous ab text:

class RequestsController < ApplicationController
  def new
    finished 'request_text'
  end
end

Now, everything is working fine except the dashboard. In that we see people entering the experiment but it never finishes.
What it appears to be happening is that split is creating a new version of the experiment each time a user connects and/or finishes the experiment.

Would that be possible? Is there any more information I can offer you?

You may wish to store the users session somewhere other than the users session (the rack type).

For example: store the a/b session against the current logged in users database record or a cookie shared between sub domains

If you reset an experiment it rolls the alternative counters back to zero but it doesn't empty the session for users who already have the alternative set in their cookie then the can finish the, now reset, experiment, without their participation being measured.

This could, in the extreme case, cause an infinite conversion rate if 0 users participated but 1 finished!

I suggest we put the version in the cookie and increment a version when resetting the experiment. The more extreme alternative would be to store the users session in redis but that add more overhead.

If you run an ab_test with two alternatives and then change the alternatives it doesn't remove the old alternatives.

If I don't want to enable to the dashboard then I should not need my project to depend upon Sinatra.

If the dashboard were moved into a separate gem e.g. split-dashboard then it could be an optional install

I'm using Redis 3.0.1. When a redis connection is refused, it throws a Redis::CannotConnectError exception, not a ECONNREFUSED exception, therefore it still bubbles upwards and kills the app.

I suggest using a catch-all exception handler when db_failover is enabled, since it's quite important for production systems to "keep calm and carry on" and I don't want to check all possible exceptions this and future Redis could throw (Timeout exceptions? Peer disconnected exceptions?).

@livingsocial have created a pretty cool gem for doing statistical analysis on A/B test results: https://github.com/livingsocial/abanalyzer

It may be worth bringing in some of the tests to the dashboard or at least proving the option to.

When you add a new experiment to your code and deploy there is a currently a race condition.

If one of your visitors triggers the finish helper for the experiment before one of your visitors triggers the creation of the experiment then the finish method raises an exception.

Split should handle this without raising an exception.

Hello,

I have the following scenario:

• Previously I was testing experiment "landing_page" with variants "original" and "new_copy".
• We decided to run a new test.
• We replaced this test with a new one, same experiment name "landing_page" with variants "original" and "screencast_and_testimonials".
• Split will now only serve variant "original"
• If you visit http://domain.com/?landing_page=screencast_and_testimonials, the screencast_and_testimonials variant is displayed to the user. However, we use the following code to record the AB variant into KISSmetrics, and when we manually request screencast_and_testimonials, it was being recorded as new_copy (variant #2 from the prior experiment):

  <% Split::Experiment.all.each do |experiment| %>
    <% km.set(experiment.name, ab_test(experiment.name, *experiment.alternative_names)) %>
  <% end %>

Is this expected behavior? e.g should we not reuse an experiment name with new sets of variants?

(As an aside, is there a better way to record the variants into KM other than that snippet?)

Thanks,
-Jason

When erroneously trying to use symbols or ints instead of strings as alternatives, I got

irb(main):017:0> ab_test('xyz', 1, 2, 3)
NameError: uninitialized constant Split::Experiment::InvalidArgument
    from /Users/przemek/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/split-0.4.6/lib/split/experiment.rb:167:in `initialize_alternatives'
    from /Users/przemek/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/split-0.4.6/lib/split/experiment.rb:143:in `find_or_create'
    from /Users/przemek/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/split-0.4.6/lib/split/helper.rb:108:in `experiment_variable'
    from /Users/przemek/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/split-0.4.6/lib/split/helper.rb:10:in `ab_test'
    from (irb):17
...

It looks like it's supposed to raise an InvalidArgument exception with a meaningful message, but instead it's a NameError. There are two places where InvalidArgument is used in experiment.rb, lines 139 & 167.

Passing non-string alternatives to a test results results in constantly resetting tests

Its great that we can filter on bot user agents, but most bots forge their user agent. :(

One feature I liked in ABingo was their human detection support. It would be great to track participants and conversions, but not really include the data in the totals until the participant is proven to be human. With ABingo I would create a javascript callback that would indicate that the js was executed (which was inferred as a human) and it worked great.

It would be good to add support for something like this:

Split.is_human?
Split.human!

I also may be able to help with this if others also feel like its useful.

Cheers!

in recent rails a session is created lazily - that is it's created when the session is accessed.
if there is an ab_test in a view and no session is present, then the same user could be presented with different ab tests in the same "session".
(i.e. e-commerce site and basket is empty)

I'm setting up A/B testing for the first time and I'm wondering if I am misunderstanding how it works.

I have a page set up where the current option is weighted 1 and the new option is weighted 10. If I reload the page repeatedly, I should see the new option 9 times of out 10, right? Currently, it seems to be picking one option on initial load and sticking with it forever. Also, as I reload, the number of non-finished experiments doesn't increment. Should it?

Say that I run an ecommerce site and the conversion is a purchase. Now assume that I'm using split to test a whole host of things, from which landing page to give the site visitor to which checkout form layout to which promotion to display.

The problem is that right now I need to do this on completion in my checkout controller:

 finished('select_lander');
 finished('checkout_form);
 finished('promotion_pick);

which is a bit insane. The DSL would be much cleaner if we can specify a NAME for an experiement and a GOAL, so, for example, we could do this:

ab_test('select_lander','purchase',option1, option2, etc) 

or

ab_test(name: 'select_lander',goal: 'purchase', [option1, option2, etc]) 

and then simply

finished('purchase')

or better

finished!('purchase') to indicate that it was an obtained goal.

Unless this functionality already exists and I'm missing it. Thanks.

Hi!
First, thank you very much for this really useful tool!
I'm currently evaluating whether I should choose Vanity or Split for a Rails 3 project. You mention that Split is heavily inspired by Vanity, but what exactly would be the reason to go with Split instead of Vanity? Is there any documentation describing the differences and advantages?

Thanks a lot!
Michael

I would like to be able to provide adapters for different databases, such as mongodb, mysql.

But first that will require extracting the Split.redis method and it's calls out into a database api of some sort.

can I use split with mongoid odm?

Thank you!

for a more important feature than the color of a button the ab test could intervene in more than one place (several controllers/views). I could be useful to have the same ab_test operate the same switch throughout the whole session.

It would be cool if ab_test would by default return true/false if no other arguments are given and thus a code block, which defines the experiment, is run or not.

I currently have 2 experiments running localy to test them.

What I did:

1. I used one alternative for each one of them
2. I kept using my application as a user
3. I reset both of them
4. I deleted both of them

Then when I reran my application only one of the experiments ran and the other appeared to be closed no matter how many times I visited it.

Has anybody encountered this before?

Record when the experiment has started and show on dashboard

It doesn't look possible to use symbols as experiment names; is this correct? Would it be useful to be able to do that (perhaps to take advantage of terse Ruby 1.9 hash syntax)?

Hello,

I'm getting the following error when I try to access the dashboard at /split:

RangeError at /split
can't convert -8.659560562354933e-17+1.414213562373095i into Float
file: dashboard.rb location: to_f line: 32

Does this mean I've somehow setup the experiment wrong?

Thanks,
Ryan

These are a couple of things I found out (or so I may think) by using split for ab-testing. Please feel free to add or comment anything.

• The old version should be given as the first attribute in tha abTesting
  E.g abtest( AbTestName, OldVersion, NewVersion1, NewVersion2, etc)
  • This will help to set it as a fallback (called control) when the redis connection fails
  • Also this means that any other alternative is compared statistically with this in the dashboard
• How to stop an experiment
  • Stop the experiment but have the experiment code in place without it running?
    Simply select the alternative you want from the dashboard. This will only show the alternative you have selected, while the experiment won’t count participants any more.
  • Stop the experiment permanently?
    You will have to remove all the ab_test code and finalization that was used for this experiment only and then delete the experiment (to remove all the data from redis) from the dashboard.
  • Stop the experiment temporarily?
    There is no such function currently (v 0.4.1) so as to PAUSE an experiment, but you can always select an option to show temporarily (probably the one you have set as the control) without removing the ab_test code and the experiment.
    When you want to retake the code simply click “Reset Data” or “Delete” from the dashboard. It will remove all the data and start the experiment from scratch with a new version number
• Split Dashboard
  • Delete Button
    • It erases everything from the current expirement concerning the data assembled
    • Whenever it is clicked (either when an option is selected or not) if the ab_test code is in use the test will begin again with new data and a new version number.
    • In case you want to change the control attribute you should deploy your code with this change and then delete the experiment from the dashboard. When the first user runs the test you will see the experiment with the correct order.
  • Reset Data Button
    • Pretty much what Delete does but without deleting the experiment board.
  • Use This Button
    • With this button you can select the option to be used and finalize the test
  • Control Attribute
    • The control alternative is the version that all the other alternatives are compared to
  • Version Attribute
    • It refers to how many times the experiment has been reset or deleted
  • Participants Column
    • Total users that have been given that alternative
  • Non-Finished Column
    • Users that have run the test but haven’t finalized it yet. (Completed - Participants)
  • Completed Column
    • Total users that have run the test and followed every step till the test finalized (was completed)
  • Conversion Rate Column
    • The conversion rate has 2 attributes
      • The percentage that appears in black letters that is calculated by the percentage of those who took the test and finalized.
      • The percentage that appears in red or green color that is the percentage of comparison to the control alternative.
        This is calculated by: ( Alternatives’ Conversion / Controls’ Conversion) - 1

While testing split I've come to a error when opening the dashboard because the Complex class was not able to convert the number to float. The number is 2.222670197524858e-18-0.03629895899899249i.

If you try on irb

n = Complex "2.222670197524858e-18-0.03629895899899249i"
n.to_f #error

this erros will be shown RangeError: can't convert 2.222670197524858e-18-0.03629895899899249i into Float

although it doesn't happen if you try to convert it to a float from a string like

n = "2.222670197524858e-18-0.03629895899899249i"
n.to_f #2.222670197524858e-18

Usage is quite simple:

some_signin_view_file.erb:

<% signin_mode = ab_test( 'log in style','LogIn_ATest','LogIn_BTest' )%>
.
.
Do something according to signin_mode...
and
some_post_signin_controller_file.rb:

finished("log in style", :reset=> FALSE)
Did one simple test with no problems. However, my second test yield negative numbers on the non-finished columns, and only on the first experiment (marked as control).

How can it be negative...? Am i missing something?

[EDIT: remove a non working workaround that was here...]

see also http://stackoverflow.com/questions/9964357/rails-ab-testing-with-split-gem-negative-numbers-on-non-finished/9972546#9972546

The test suite is breaking on jruby-19mode, not sure if it's split, a bug in jRuby or a problem with travis.

https://travis-ci.org/andrew/split/builds/3112928

see comment here: #43 (comment)

I'm putting a patch together for submissions, just wanted to generate an issue number for my branch

This is just a suggestion, not necessarily a bug. For example, say I have a header that I ab_test on the home page, then on the signup page I have a finished action. If the user reloads the signup page then it gets counted twice as finished and messes up the numbers. Personally I would rather track the finish events for each user so that this wouldn't happen, but maybe thats just me. I realize this adds a much higher data requirement.

Thanks

I may be clueless about how weighting works, but this is what I have

    enable = ab_test( "enable_jbc", {'false' => 0.99}, {'true' => 0.01} )

I would expect enable to be 99% false, but it's always about 50%. Did some digging and found that the following line of code will return alternative with equals weights, e.g. 1 after it's created in redis.

        experiment = Split::Experiment.find_or_create(experiment_name, *alternatives)

So only the first user will have the alternative with the highest weight, after that all alternatives will have equals weights

Following the instructions from the README when I visit "/split" in my app I get an error:

Environment

ruby 1.9.2p318
Rails 3.2.1
split (0.4.5)
Redis server version 2.4.15 (00000000:0)

View

<%= f.submit ab_test('csn_submit_button', 'Submit', 'Submit!'), data:{disable_with:'...'}, class:'btn btn-primary' %>

Controller

finished('csn_submit_button') if URI.parse(request.referrer).path == root_path

Error message

argument out of range

Backtrace

/Users/gabeodess/.rvm/rubies/ruby-1.9.2-p318/lib/ruby/1.9.1/time.rb in local
        self.local(year, mon, day, hour, min, sec, usec)
/Users/gabeodess/.rvm/rubies/ruby-1.9.2-p318/lib/ruby/1.9.1/time.rb in make_time
        self.local(year, mon, day, hour, min, sec, usec)
/Users/gabeodess/.rvm/rubies/ruby-1.9.2-p318/lib/ruby/1.9.1/time.rb in parse
      make_time(year, d[:mon], d[:mday], d[:hour], d[:min], d[:sec], d[:sec_fraction], d[:zone], now)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/split-0.4.5/lib/split/experiment.rb in start_time
      Time.parse(t) if t
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/split-0.4.5/lib/split/dashboard/views/_experiment.erb in evaluate_source
      <small><%= experiment.start_time ? experiment.start_time.strftime('%Y-%m-%d') : 'Unknown' %></small>
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/tilt-1.3.3/lib/tilt/template.rb in instance_eval
      scope.instance_eval(source, eval_file, line - offset)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/tilt-1.3.3/lib/tilt/template.rb in evaluate_source
      scope.instance_eval(source, eval_file, line - offset)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/tilt-1.3.3/lib/tilt/template.rb in cached_evaluate
      evaluate_source(scope, locals, &block)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/tilt-1.3.3/lib/tilt/template.rb in evaluate
      cached_evaluate(scope, locals, &block)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/tilt-1.3.3/lib/tilt/template.rb in render
      evaluate scope, locals || {}, &block
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in render
        output          = template.render(scope, locals, &block)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in erb
      render :erb, template, options, locals
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/split-0.4.5/lib/split/dashboard/views/index.erb in block in evaluate_source
    <%= erb :_experiment, :locals => {:experiment => experiment} %>
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/split-0.4.5/lib/split/dashboard/views/index.erb in each
  <% @experiments.each do |experiment| %>
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/split-0.4.5/lib/split/dashboard/views/index.erb in evaluate_source
  <% @experiments.each do |experiment| %>
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/tilt-1.3.3/lib/tilt/template.rb in instance_eval
      scope.instance_eval(source, eval_file, line - offset)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/tilt-1.3.3/lib/tilt/template.rb in evaluate_source
      scope.instance_eval(source, eval_file, line - offset)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/tilt-1.3.3/lib/tilt/template.rb in cached_evaluate
      evaluate_source(scope, locals, &block)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/tilt-1.3.3/lib/tilt/template.rb in evaluate
      cached_evaluate(scope, locals, &block)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/tilt-1.3.3/lib/tilt/template.rb in render
      evaluate scope, locals || {}, &block
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in render
        output          = template.render(scope, locals, &block)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in erb
      render :erb, template, options, locals
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/split-0.4.5/lib/split/dashboard.rb in block in <class:Dashboard>
      erb :index
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in call
            proc { |a,p| unbound_method.bind(a).call } ]
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in block in compile!
            proc { |a,p| unbound_method.bind(a).call } ]
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in []
            route_eval { block[*args] }
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in block (3 levels) in route!
            route_eval { block[*args] }
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in route_eval
      throw :halt, yield
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in block (2 levels) in route!
            route_eval { block[*args] }
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in block in process_route
        block ? block[self, values] : yield(self, values)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in catch
      catch(:pass) do
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in process_route
      catch(:pass) do
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in block in route!
          pass_block = process_route(pattern, keys, conditions) do |*args|
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in each
        routes.each do |pattern, keys, conditions, block|
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in route!
        routes.each do |pattern, keys, conditions, block|
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in dispatch!
      route!
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in block in call!
      invoke { dispatch! }
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in block in invoke
      res = catch(:halt) { yield }
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in catch
      res = catch(:halt) { yield }
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in invoke
      res = catch(:halt) { yield }
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in call!
      invoke { dispatch! }
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in call
      dup.call!(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/rack-1.4.1/lib/rack/auth/basic.rb in call
          return @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/rack-protection-1.2.0/lib/rack/protection/xss_header.rb in call
        status, headers, body = @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/rack-protection-1.2.0/lib/rack/protection/path_traversal.rb in call
        app.call env
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/rack-protection-1.2.0/lib/rack/protection/json_csrf.rb in call
        status, headers, body = app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/rack-protection-1.2.0/lib/rack/protection/base.rb in call
        result or app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/rack-protection-1.2.0/lib/rack/protection/xss_header.rb in call
        status, headers, body = @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/rack-1.4.1/lib/rack/nulllogger.rb in call
      @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/rack-1.4.1/lib/rack/head.rb in call
    status, headers, body = @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/rack-1.4.1/lib/rack/methodoverride.rb in call
      @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/showexceptions.rb in call
      @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in call
      result, callback = app.call(env), env['async.callback']
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in block in call
        synchronize { prototype.call(env) }
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in synchronize
          yield
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sinatra-1.3.3/lib/sinatra/base.rb in call
        synchronize { prototype.call(env) }
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/journey-1.0.3/lib/journey/router.rb in block in call
        status, headers, body = route.app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/journey-1.0.3/lib/journey/router.rb in each
      find_routes(env).each do |match, parameters, route|
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/journey-1.0.3/lib/journey/router.rb in call
      find_routes(env).each do |match, parameters, route|
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/actionpack-3.2.1/lib/action_dispatch/routing/route_set.rb in call
        @router.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/omniauth-1.1.0/lib/omniauth/strategy.rb in call!
      @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/omniauth-1.1.0/lib/omniauth/strategy.rb in call
      dup.call!(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/omniauth-1.1.0/lib/omniauth/builder.rb in call
      to_app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/sass-3.1.15/lib/sass/plugin/rack.rb in call
        @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/warden-1.1.1/lib/warden/manager.rb in block in call
        @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/warden-1.1.1/lib/warden/manager.rb in catch
      result = catch(:warden) do
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/warden-1.1.1/lib/warden/manager.rb in call
      result = catch(:warden) do
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/actionpack-3.2.1/lib/action_dispatch/middleware/best_standards_support.rb in call
      status, headers, body = @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/rack-1.4.1/lib/rack/etag.rb in call
      status, headers, body = @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/rack-1.4.1/lib/rack/conditionalget.rb in call
        status, headers, body = @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/actionpack-3.2.1/lib/action_dispatch/middleware/head.rb in call
        @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/actionpack-3.2.1/lib/action_dispatch/middleware/params_parser.rb in call
      @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/actionpack-3.2.1/lib/action_dispatch/middleware/flash.rb in call
      @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/rack-1.4.1/lib/rack/session/abstract/id.rb in context
          status, headers, body = app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/rack-1.4.1/lib/rack/session/abstract/id.rb in call
          context(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/actionpack-3.2.1/lib/action_dispatch/middleware/cookies.rb in call
      status, headers, body = @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/activerecord-3.2.1/lib/active_record/query_cache.rb in call
      status, headers, body = @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/activerecord-3.2.1/lib/active_record/connection_adapters/abstract/connection_pool.rb in call
        status, headers, body = @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/actionpack-3.2.1/lib/action_dispatch/middleware/callbacks.rb in block in call
        @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/activesupport-3.2.1/lib/active_support/callbacks.rb in _run__810018673512189422__call__2626213606992483835__callbacks
        object.send(name, &blk)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/activesupport-3.2.1/lib/active_support/callbacks.rb in __run_callback
        object.send(name, &blk)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/activesupport-3.2.1/lib/active_support/callbacks.rb in _run_call_callbacks
              self.class.__run_callback(key, :#{symbol}, self, &blk)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/activesupport-3.2.1/lib/active_support/callbacks.rb in run_callbacks
      send("_run_#{kind}_callbacks", *args, &block)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/actionpack-3.2.1/lib/action_dispatch/middleware/callbacks.rb in call
      run_callbacks :call do
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/actionpack-3.2.1/lib/action_dispatch/middleware/reloader.rb in call
      response = @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/actionpack-3.2.1/lib/action_dispatch/middleware/remote_ip.rb in call
      @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/actionpack-3.2.1/lib/action_dispatch/middleware/debug_exceptions.rb in call
        response = @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/actionpack-3.2.1/lib/action_dispatch/middleware/show_exceptions.rb in call
        response  = @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/railties-3.2.1/lib/rails/rack/logger.rb in call_app
        @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/railties-3.2.1/lib/rails/rack/logger.rb in call
          call_app(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/actionpack-3.2.1/lib/action_dispatch/middleware/request_id.rb in call
      status, headers, body = @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/rack-1.4.1/lib/rack/methodoverride.rb in call
      @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/rack-1.4.1/lib/rack/runtime.rb in call
      status, headers, body = @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/activesupport-3.2.1/lib/active_support/cache/strategy/local_cache.rb in call
            @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/rack-1.4.1/lib/rack/lock.rb in call
      response = @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/actionpack-3.2.1/lib/action_dispatch/middleware/static.rb in call
      @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/railties-3.2.1/lib/rails/engine.rb in call
      app.call(env.merge!(env_config))
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/railties-3.2.1/lib/rails/application.rb in call
      super(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/rack-1.4.1/lib/rack/content_length.rb in call
      status, headers, body = @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/railties-3.2.1/lib/rails/rack/log_tailer.rb in call
        response = @app.call(env)
/Users/gabeodess/.rvm/gems/ruby-1.9.2-p318@cityshare/gems/rack-1.4.1/lib/rack/handler/webrick.rb in service
        status, headers, body = @app.call(env)
/Users/gabeodess/.rvm/rubies/ruby-1.9.2-p318/lib/ruby/1.9.1/webrick/httpserver.rb in service
      si.service(req, res)
/Users/gabeodess/.rvm/rubies/ruby-1.9.2-p318/lib/ruby/1.9.1/webrick/httpserver.rb in run
          server.service(req, res)
/Users/gabeodess/.rvm/rubies/ruby-1.9.2-p318/lib/ruby/1.9.1/webrick/server.rb in block in start_thread
          block ? block.call(sock) : run(sock)

The experiment class some pretty large methods in it now, code climate has a couple warnings that I'd like to cleanup: https://codeclimate.com/github/andrew/split/Split::Experiment#method-complexity

Is there the possibility to handle the weight of the alternatives from the dashboard, so as to avoid changing this by changing the code behind?

Thanks in advance.

This may be related to #5, but when I was reading through the code base it seemed that Split allows for more then one experiment at the same time.

If this is correct I would think this would allow for skewed results as experiments may inadvertently affect each other. IMHO, the ab_test method should check to see if an experiment is already in place, and return its control if so.

Just my two cents. Maybe a config option would be a good tool for this.

Cheers!

often the behaviour you have more than one line of code/markup
could be interesting to have split with method name or block
for view: partial
(just an idea)

I'm loading a page with jQuery's load() method. It still hits the rails controller for the html and then passes it back and jquery loads a particular div within the page.

In development. I can pass in the ab_test query and get the proper layout. The test is 'product_cart_form' and my alternative is flipped. The div it's rendering is 'insertion' which contains my ab_test.

load(href + '?product_cart_form=flipped .insertion')

This hits the rails controller and properly displays the right test html. However, the test never shows up unless I place the parameter in and also never finishes correctly. Always going to the default.

Is the test not initialized in the controller? If not, can I manually start it, either via javascript or in the rails controller?

EDIT:

The ab_test was occurring in a partial one more level deep then the div that was being rendered by jquery. When I switched the test to testing two different partials instead of using a ab_test block inside one partial it works flawlessly. Not sure what to make of the prior problem.

i think it would be useful to have an option to disable split testing. this would be helping in continuos integration where it's not necessary to alway run split test and this would save on extra redis calls

The ab_test code block does not work as advertised. In the doc it says that:

<% ab_test("login_button", "/images/button1.jpg", "/images/button2.jpg") do |button_file| %>
And this line will not be shown either...
<%= img_tag(button_file, :alt => "Login!") %>
<% end %>

should output something like:

< img src='/images/button1.jpg '> or < img src='/images/button2.jpg' >

but the only thing it output is either
/images/button1.jpg or /images/button1.jpg

It seems like it ignores the block content and only output the alternative string.

I've just pushed a small read me change and the Travis CI test are now failing on rubinius: http://travis-ci.org/#!/andrew/split/builds/1389933

Not sure whats changed but it looks like an upgrade to rubinius.

I won't be able to look at this for a couple weeks so any help would be appreciated.

The order of keys in a hash in ruby 1.8 is not always the same as when it was created (unlike in ruby 1.9).

This means that when specifying alternatives as a single hash (for example when passing weights).

The two fixes I can see are:

• Using ActiveSupport::OrderedHash in ruby 1.8 to patch the hash ordering (not ideal)
• Deprecating passing a single hash, instead passing multiple hashes for alternatives, or at least two, one as the control and the second+ as other alternatives

I'm running split v0.4.0 with rails 3.2.3 on Heroku, and it seems to count each individual impression as a participant.

Any ideas on how to change this so it counts each session as a participant?

We would like to be able to consider multiple client-side users as the same backend user (say, make it so one version of an experiment is shown for all users associated with a given account).

To make it as useful as possible, passing a Proc in to split may be the best way to accomplish this. Therefore, if the proc is passed in the configuration block, it replaces the #ab_user method's functionality. In our case, we would make a database call to see if user A is part of the same account as user B and return the same split session for both.

I'd like to be able to run certain tests only if the user is using a mobile device. Don't see this documented anywhere so I am assuming it would need to be added. I'd be glad to add Rack Mobile Detect and a convenience method in the right place, but want to run it by you all before I do so to make sure that is the approach you would suggest taking.

Thanks,
Doug

I'm using Split to test wether placing more ads on my rails site results in fewer returning visitors.

To test this I need to save the split session data for a lifetime in a cookie (i.e. 1 year +). The session that is used now is easily erased when a user logs in/logs out, or closes the browser, or after a few days, depending on the parameters used for session expiration.

I created a fork of Split for my own use, which uses the Rails construct cookies.signed, together with a custom expiration date. It was a quick hack without tests or documentation, and I'm pretty sure sinatra doesn't have the same construct, so I'm not creating a pull request for it since I don't deem it gem-worthy, but I think others would benefit from having a similar feature.

Hence, this feature suggestion.

Here's how I'm doing it, note that in an earlier commit I added the ability to set cookie_domain and cookie_expiration in the Split config block: dv@b66348b

If I call ab_test in a view I get the exception mentioned.

Including Split::Helper in my ApplicationHelper will solve this, but not sure of the cause.

I'm presuming it's something to do with load order of Split and the if defined? Rails not getting executed, but can't seem to get to the bottom of it, so no failing test case at the moment.

Have you encountered this before?