Comments (3)
I think you have two questions:
-
How \nabla_z L^t can be computed in a single backward pass? We assume encoder-decoder structure as we explain in that section. Hence, the path from z to L^t is independent between tasks. Since they are independent, it is just a single backward pass through decoders.
-
Wouldn't this apply to \nabla_theta L^t? No, because \nabla_theta L^t = \nabla_{theta^sh} Z \nabla_Z L^t. You can compute all \nabla_Z L^t values in a single pass. Then, you can compute \nabla_{theta^sh} Z explicitly as well in a single pass but that computation will be much more costly than computing \nabla_{theta^sh} L^t since auto-differentiation works without explicit computation of \nabla_{theta^sh} Z. I think this is hard to explain without getting into details of AD but you can try to go through computation graph explicitly to see the difference. In other words, it is not a single-pass computation in the sense that a single-pass of AD would not be able to compute it.
from multiobjectiveoptimization.
Yes, your understanding is correct. Also, keep in mind that parameters from shared representation to task-losses are disjoint following encoder-decoder assumption. Hence, we compute gradient over any parameter only ones.
from multiobjectiveoptimization.
Thank you for the clarification. After going through the code more carefully and reading your reply I think I understand what you meant. I think the confusion was that by 'single-pass' you meant 'single-pass on the shared parameters', which are typically orders of magnitude more than the task-specific parameters. In your code, you are still calling backward() for each task, but only backproping till Z. So the saving resides in not having to backprop till the \theta^sh, PLUS not having to forward over them T times. Is my understanding correct?
from multiobjectiveoptimization.
Related Issues (20)
- depth loss and metric
- Connection between fmp and task branches HOT 4
- IndexError: list index out of range (when reading base_path in "celeba_loader.py" line 52 )
- size of the segmentation output HOT 1
- run error, Please help me, Thanks HOT 1
- Question about the MinNormSolver
- Unable to obtain the same average error as Table 1 in the paper
- Unable to obtain the same average error as Table 1 in the paper HOT 1
- How to convert .pkl file to .onnx model. I mean there are two parts of the whole model. HOT 4
- Redundancy of optimiser.zero_grad()? HOT 2
- cityscape dataset
- About Cityscapes Instance Segmentation GroundTruth Generation.
- `iter_count` is never incremented in `min_norm_solvers`?
- is the code corresponding to the algorithm in paper ? HOT 1
- If I want to train the 3D image data using MTL by MGDA do I have to change the code min_norm_solver.py? HOT 1
- local variable 'sol' referenced before assignment HOT 1
- Algorithm 2 line 10 HOT 1
- Is there any difference between soft sharing and hard sharing network when using MGDA? HOT 1
- A question about the output of rep HOT 1
- params.json HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from multiobjectiveoptimization.