Comments (1)
The current implementation first does a reverse topological sort on all nodes (No parent comes before its child. For example x and y are parents of the expression (x+y). ) Then it calls _backward() in that order. So when we call the function _backward() for a node, we guarantee that its .grad property has its final (and correct) value and will not change in the future.
Consider the following example:
- a = 2
- b = 3
- c = a + b
- d = a * b
- e = c + d
- f = c * d
- y = e + f
Your method will give an incorrect result for this example. I won't show it step by step as it looks too long and complex in text but you can calculate it with hand to see the problem. The problem will occur in the following way:
- You set y.grad = 1 (correct)
- You set e.grad = 1 (correct)
- Then, because you are using recursion, you will go into the node "c".
- You will set c.grad = 1 (This is not the complete/correct result obviously. This is only the part that comes from e. You will also need to add a +d term when you are calculating the gradient through f. So, as you can verify via calculus dy/dc = 1+d)
- Now, again using recursion you will continue to the "a" node.
- Now you will set a.grad = 1 (This is incorrect, you have calculated it thinking that dy/dc = 1. But the correct value was 1+d.)
I hope I could explain the problem clearly. The problem with your method is basically using the wrong, temporary .grad values of a node to calculate the .grad of its children.
If you are confused by the whole node and graph logic, watching this video would help you understand the structure more clearly as @karpathy imagined it.
from micrograd.
Related Issues (20)
- Issue with zero_grad? HOT 5
- _backward as lambdas? HOT 3
- Sequential MLP implementation
- Zero_grad only zeros the weight and bias nodes, not the nodes for addition and multiplication HOT 1
- A tensor version for micrograd inpired by this work
- Rename engine.py to value.py
- Vectorized implementation with PyTorch flavor
- Vectorized modification with GPU support.
- Another MiniGrad with the RAdam optimizer.
- Ensure backward() is idempotent HOT 4
- Regarding the gradient update of the __sub__ operation HOT 1
- Reseting the grad of weights and biases is not enough HOT 1
- Adjusting parameters by sign and magnitude of gradient HOT 1
- Topological sort - bug HOT 3
- type annotation lacking/ maybe also add docstrings.
- For addition adding incrementing grading makes sense, I can't make sense out of the incrementing it for multiplication too, potential bug? HOT 1
- radd
- Is `y = x * x` an edge case?
- Question/Idea: Automatic Gradient Clearing
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from micrograd.