Intuition behind step_tfembed approach? about embed HOT 8 CLOSED

It's meant to be a preprocessor. You are correct about it doesn't fit the whole network and so isn't as supervised as it could be. There are plenty of pre-trained networks that stop at this point and can be plugged into a larger network model. step_embed (name changed) does this but let's you use it as an input for any type of model.

from embed.

Word2vec and similar can only stop there because there's both an enormous amount of data and significant earlier pre-processing (e.g. skip-gram tokenizing, matrix decomp, etc.). Have you had any luck with this technique for any tasks beyond word embeddings, or seen others do so? I notice the vignette stops short of evaluating improvement in the task itself.

I'm worried the description/explanation is a bit misleading, because it seems to suggest this step learns embeddings like in Guo/Berkhahn. That's certainly what I thought (and was so excited about!) before I looked at the code, and I've had to warn a couple others who had the same initial interpretation. I see now that paper is referenced in the README and vignette but not the function doc; perhaps removing it altogether would avoid this confusion?

from embed.

There's a step that allows for other predictors in the network. Let me know if you have any thoughts in this.

from embed.

@topepo step_embed2 is looking better!

if I'm reading this correctly, as in the linked blogpost this makes a separate input/embed layer for each categorical in ..., but forces each of them to have the same output dimension (num_terms)? One number is probably a good default but it would be nice to vary the output dimension on a per-categorical basis (vector input). If you have store_ids (high cardinality) and store_types (much lower cardinality), you probably don't want them to have the same output size.

Now is probably not the right time, but it could also be neat to do a heuristic default for the output dimension; the most up-to-date exposition of deep categorical embeddings I know of is lesson 3 of fast.ai (https://github.com/fastai/fastai/blob/master/courses/dl1/lesson3-rossman.ipynb), which uses min(50, (c + 1)/2) as the output dimension where c is the number of categories present for that input.

from embed.

One number is probably a good default but it would be nice to vary the output dimension on a per-categorical basis

Sure. I'll do that in the next version though (unless you want to put in a PR). I've got to get a working version out soon.

do a heuristic default for the output dimension

I've seen that and it does seen reasonable. I'm all for reasonable defaults but I'd rather add a small number and make a note about that in the docs.

I'll give it a few days for others to take a look at and then replace the current step_embed with this version.

from embed.

@topepo I'd be happy to make a PR for that, but I won't have time to start until mid next-week, so please don't wait around for me!

from embed.

Closed due to inactivity

from embed.

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.

from embed.