zeeraktalat / multitask-abuse Goto Github PK

As discussed, since we don't have much information as to how informative tasks are for each other, we set the weights for all auxiliary tasks equally, more specifically to 1/len(aux_tasks).

That will be the easy thing to try first.

What I forgot to mention are possible ways to quantify task relatedness. These could include measuring the PMI of labels or vocabulary. Could be interesting to look into this.

See our CWI'18 code for reference: https://github.com/bjerva/cwi18

In this project, tasks are different languages.

In src/model.py you'll see that self.inputs is a dictionary mapping a taskID to an input layer. self.outputs should really follow that convention tbh, but for some reason it's a list of layers here that are accessed with integers, don't know why we did it that way. Then in the forward(), you see that the input task ID is provided as an argument.

In src/run.py, during training, a task ID is drawn at random, then a batch of data for that task. That's not best practice I'd say, better to have a predefined schedule, but in any case you wanna know which task you're training on, so that you can tell that to the model.forward().

Make sense?

A bunch of questions have come up today from the train function:

• Can we make one massive vocabulary shared by all tasks or should it be task dependent vocabularies?
• What is clip / does it do?
• What is model.binary?
• Why the CWI epoch loss?
• What does model.train() do?
• Why do you set the zero gradient for the optimizer instead of the model?
• Why do you change the view (logits.view([size, 1]) of the predicted results of your model?