mint-metrics / mojito-r-analytics Goto Github PK

We often run experiments that implement a new feature that does not exist on the control group.

We want to see how many users are interacting with a particular feature, however we can't compare it against the control. This leads us to data like:

• Control: 0% conversion
• Treatment: 10% conversion (Infinite increase over the control group)

This makes our summary table hard to read and the metric plots somewhat misleading.

@kingo55 should we consider fleshing out Bayesian analytics again? It would be interesting to develop some functionality to run side-by-side with the Frequentist reports we run, to see how it stacks up.

The main thing to put some thought to is how we calculate priors. We could perhaps calculate it (mean + std deviation) based on the past X months worth of conversion data?

Another question is how to we deal with sizing and presenting the data in our reports.

Refs:

• bayesTest R documentation
• primer on priors
• good primer on bayesian vs frequentist, with nice visual format

I think we can make reports more customisable by changing the way we reference tables in our knits / Mojito.

Currently tables are referenced based on the client ID and subject types defined in wave_params. This makes for a tidy wave_params object:

wave_params <- list(
  client_id="client",
  wave_id="w143",
  start_date="2020-05-25 11:33:00",
  stop_date="2020-06-13 14:42:22",
  time_grain="days",
  subject="usercookie",
  recipes=c("Control","Treatment")
)

This yields tables like so:

• mojito.exposures_usercookie
• mojito.segments_usercookie
• mojito.client_conversions_usercookie

There are some issues with this though:

1. It is quite rigid because users aren't able to deviate their schema naming or table naming conventions to fit their data warehouses.
2. It's also inefficient in Redshift where goals that would normally be defined once-off, inside a report, need to be committed to our datamodelling steps (users can't just define a custom goal table for a goal e.g. (SELECT domain_userid as subject, 'conversion' as goal, 10.00 as revenue, derived_tstamp as conversion_time FROM client.events WHERE event_name = 'custom_schema')).
3. Another inefficiency is requiring users to specify a client ID. Not all users will be multi-tenanted and the additional column uses extra space in the DWH, whilst not strictly needed.

Whilst slightly uglier, I think we can make Mojito easier-to-adopt through customisable table references, like so:

wave_params <- list(
  wave_id="w143",
  start_date="2020-05-25 11:33:00",
  stop_date="2020-06-13 14:42:22",
  time_grain="days",
  tables=list(
    exposure="mojito.exposures_usercookie",
    goal="mojito.client_conversions_usercookie",
    segment="mojito.segments_usercookie",
    failure="mojito.recipe_errors_2"),
  recipes=c("Control","Treatment")
)

Thoughts @dapperdrop ?