lukasvermeer / confidence Goto Github PK

Hello @lukasvermeer, first of all, thank you for your amazing work (I'm a big fan). I want to reuse a small part of this, specifically, the confidence interval SVG, is this MIT licensed, it's not mentioned so I prefer to ask you. Would that be ok?

Omer wrote:

Once you identify a good topic you have such a strong prior that the expected uplift on shipping that topic will be positive that you focus only on exploitation with no exploration and limited concern about downside protection or false positives.

Probability of success higher than 50% likely never happens in real life: if we were to ship every idea around a topic our expected impact would likely be negative, it being easier to hurt than improve conversion and due bad ideas, mistakes, bugs etc. So the mean uplift of every topic being negative could be both more realistic and also make the game more balanced between explore vs exploit and force players to pay more attention to confidence levels and run experiments longer. Protecting against false positives and downside would become more important.

I like this idea. I suspect it means effect sizes for each topic would have to be bimodal (one mass on the plus side, one on the negative side) rather than the simple normal they are now.

There would still be value in figuring out topics with heavy mass on the plus side, but shipping that topic blind would result in net negative impact on average.

Another related idea I had was to add “buggy” experiments. So once in a while for any topic an experiment has very negative impact because of a bug. It would have similar effect, I think.

Good day Lucas.
Thank you very much for the excellent AB test simulator.
https://www.lukasvermeer.nl/confidence/index.html
This is really cool stuff.
Can I ask you to watch it when you have a free minute?
In my opinion it stopped working properly.

Bro, create a gh-pages branch with the exact same content as the master branch (hint: you can do it straight from github) so we can see this thing live at http://lukasvermeer.github.com/confidence

Seems I can no longer rely on Twitter for login. I should remove the integration.

Hello,
This is a notice that your app - So You Think You Can Test? - has been suspended from accessing the Twitter API.
Please visit developer.twitter.com to sign up to our new Free, Basic or Enterprise access tiers.
More information can be found on our developer community forums.
Regards, Twitter Developer Platform

Omer wrote:

Diminishing returns could be a more complex concept to add. As you ship more winning experiments, the effect sizes(at least on the positive side) could start to drop. This is also more like real life and would force players to emphasize exploration & continuous learning by reducing the certainty of their priors.

I actually did consider that, but I didn’t quite know how to implement it, so I left it out. Good to hear Omer had the same idea, that makes me thin I might be on to something there. :-)