Comments (4)
Hi, this is my guess. It would definitely work.
But at a certain moment, the depth of the graph can be so large that one risks stack overflow.
It feels like a tradeoff between simplicity and robustness of the example. But I am not sure if that was the real concern.
My concern was that we are creating the order with each call of the back propagation, while it typically remains constant over the whole optimization process. So the order could easily be cached somewhere. But that is just performance problem, I undestand that this was supposed to be explanation of the principle.
And it is a great one! ❤
from micrograd.
Great question, I was wondering the same 👍🏽
from micrograd.
Hi, this is my guess. It would definitely work. But at a certain moment, the depth of the graph can be so large that one risks stack overflow. It feels like a tradeoff between simplicity and robustness of the example. But I am not sure if that was the real concern.
My concern was that we are creating the order with each call of the back propagation, while it typically remains constant over the whole optimization process. So the order could easily be cached somewhere. But that is just performance problem, I undestand that this was supposed to be explanation of the principle.
And it is a great one! ❤
Topological sort in micrograd is implemented using recursion, so the stack overflow concern is same as when calling backward() recursively
from micrograd.
The graph is not necessarily a tree, it can have diamonds, see tinygrad/tinygrad#165 (comment)
Checking visited nodes during topological sort prevents repeat backward calls on same node
from micrograd.
Related Issues (20)
- Issue with zero_grad? HOT 5
- _backward as lambdas? HOT 3
- Sequential MLP implementation
- Zero_grad only zeros the weight and bias nodes, not the nodes for addition and multiplication HOT 1
- A tensor version for micrograd inpired by this work
- Rename engine.py to value.py
- Vectorized implementation with PyTorch flavor
- Vectorized modification with GPU support.
- Another MiniGrad with the RAdam optimizer.
- Ensure backward() is idempotent HOT 4
- Regarding the gradient update of the __sub__ operation HOT 1
- Reseting the grad of weights and biases is not enough HOT 1
- Adjusting parameters by sign and magnitude of gradient HOT 1
- Topological sort - bug HOT 3
- backward member implementation question HOT 1
- type annotation lacking/ maybe also add docstrings.
- For addition adding incrementing grading makes sense, I can't make sense out of the incrementing it for multiplication too, potential bug? HOT 1
- radd
- Is `y = x * x` an edge case?
- Question/Idea: Automatic Gradient Clearing
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from micrograd.