Ginn: Graph Interface to Neural Networks
A minimalistic, header only neural net library
Design goals
Ginn is inspired by dynamic computation graph approaches like PyTorch or DyNet. On top of that it strives for the following:
- Ginn and all its dependencies are header only, which makes it easy to install. Still, headers are compartmentalized so you can include only what you need.
- Minimalism in code. Readability is emphasized by having small code with minimally functioning building blocks. Hopes to be less aggressive towards newcomers to C++ and easy to extend. (Short) inlined definitions means everything can be seen at one place, also makes it stylistically familiar to python users. It is written with the mindset that a newcomer can quickly pick up the whole framework easily and figure out inner workings. Every user should be able to become a developer.
- No centralized flow control. No central data structures to track a computation graph, graphs are only defined through nodes being connected. This makes it quite simple to handle threadsafety or multithreading parallelism.
- No separation between a node in a computation graph and its parameters (or data).
For example, a
WeightNodeobject both implements its node behavior as well as acts as an (owning) container of the values of its parameters. - Small networks or batches are not second class citizens. Many real life systems have latency constraints that make it important to have good performance for small networks or small batch sizes (even completely online).
- Simple autobatching. Automatically convert any computation graph to perform batched operations when it is nontrivial to write batched code, such as TreeLstms.
Example snippets
Mnist multilayer perceptron
auto [X, Y] = mnist_reader(train_file, bs);
auto [Xt, Yt] = mnist_reader(test_file, bs);
auto W = Weight(dev(), {dimh, dimx});
auto b = Weight(dev(), {dimh});
auto U = Weight(dev(), {dimy, dimh});
auto c = Weight(dev(), {dimy});
std::vector<WeightPtr<Real>> weights = {W, b, U, c};
init::Uniform<Real>().init(weights);
update::Adam<Real> updater(lr);
// Single instance (batch)
auto pass = [&](DataPtr<Real> x, Indices& y, auto& acc, bool train) {
auto h = Affine<SigmoidOp>(W, x, b);
auto y_ = Affine(U, h, c);
auto loss = Sum(PickNegLogSoftmax(y_, y));
auto graph = Graph(loss);
graph.forward();
acc.batched_add(argmax(y_->value(), 0).maybe_copy_to(cpu()), y);
if (train) {
graph.reset_grad();
graph.backward(1.);
updater.update(weights);
}
};
std::mt19937 g(seed);
// Single epoch
auto pass_data = [&](auto& X, auto& Y, bool train) {
metric::Accuracy<Int> acc;
auto perm = train ? randperm(X.size(), g) : iota(X.size());
for (size_t j : perm) { pass(X[j], Y[j], acc, train); }
return 100. * (1. - acc.eval());
};
// Main training loop
std::cout << "TrErr%\tTstErr%\tsecs" << std::endl;
for (size_t e = 0; e < epochs; e++) {
using namespace ginn::literals;
timer::tic();
std::cout << ("{:6.3f}\t"_f, pass_data(X, Y, true)) << std::flush;
std::cout << ("{:6.3f}\t"_f, pass_data(Xt, Yt, false)) << timer::toc() / 1e6
<< std::endl;
}Lstm step
template <typename DataPtrPair>
State step(const NodePtr<Scalar>& x, const DataPtrPair& past) {
auto [h_past, c_past] = past;
auto i = Affine<SigmoidOp>(Wix, x, Wih, h_past, Wic, c_past, bi);
auto f = Affine<SigmoidOp>(Wfx, x, Wfh, h_past, Wfc, c_past, bf);
auto g = Affine<TanhOp>(Wcx, x, Wch, h_past, bc);
auto c = CwiseProd(f, c_past) + CwiseProd(i, g);
auto o = Affine<SigmoidOp>(Wox, x, Woh, h_past, Woc, c, bo);
auto h_ = Tanh(c);
auto h = CwiseProd(o, h_);
return {h, c};
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent