Allow dynamic categorical lists about triage HOT 6 CLOSED

@jzanzig what do you think of an interface like this?

        categoricals:
            -
                column: 'color_id'
                choice_query: 'select distinct(color_id) from myschema.colors'
                metrics: ['sum']

Temporal leakage is worth considering, but maybe not as much as I'm thinking. Even if there are choices returned by that query that didn't exist yet at the time of the called as_of_dates, they will have columns but just won't end up with any data. So maybe that's fine?

from triage.

@andreanr also mentioned supporting getting the feature names from lookup tables as well. For instance, using the above interface, you could either

1. Get feature names that have color ids in them instead of the more helpful color names
2. Use a join query in your spacetime aggregation's from_obj, and use the joined name instead of the id. A bunch of joins may make the queries take longer.

Neither of these are optimal, so solving this in a better way could be worth it. @andreanr do you think the work you've done on this problem could be used here?

from triage.

Created dssg/collate#67 for the problem I outlined in the previous comment. It seems more appropriate to solve that problem there than here.

For the other use cases (ie denormalized tables, categoricals which actually are ids, and cases where joining to lookup a value is not too bad), I think we can move ahead here with a simpler interface like outlined above. I'm still unsure if there are any leakage problems with creating a list of choices this way.

from triage.

I'm not sure I completely understand the difference between the way collate is dealing with categoricals and this way that would introduce information leakage. Are you saying that there's a possibility that the select(distinct) query would return some possible values of color_id that wouldn't be present in that specific training or test set, but when the features themselves are generated the feature will be 0 for all observations? If so, that's fine (and it's not leaking information to have a feature that is 0 for all observations, if that's the correct value), and I think the interface (specifying a choice_query) is good

from triage.

Looks like we agree on both points:

1. That the current workarounds for dynamic categoricals work no differently with regards to information leakage than this as proposed, just with more work
2. That this doesn't actually introduce information leakage (I was pretty sure of this but wanted to confirm)

Furthermore, based on Matt's comments above it looks like this feature may make it into collate, so we could get to piggyback on that here.

from triage.

Going to implement v1 of this here instead of waiting for collate.

from triage.