Automatic Differentiation Design and Implementation about bnn HOT 2 CLOSED

The problem of implementing AD is a bit difficult and can be handled better by dividing the problem into smaller parts. The following are the components of an AD system,

1. Operations - Pre-defined mathematical operations(+, -, *, /, some transcendental functions).
2. Expressions - A combination of operations on inputs.
3. A framework to generate computation graph from expressions.
4. Final AD on computation graphs.

We should move from 1-4 in a sequential manner.

from bnn.

Operations

Input
The following combinations of inputs are possible,

• 2 Tensors
• 1 Tensor and 1 Operator(base class for all operation classes) pointer
• 2 Operator pointers
• 1 Tensor
• 1 Operator pointer

Output - Class associated with each operator. This will help in retaining an expression un-evaluated.

A simple prototype for add operation,

class Add(Operator):
    ...

Add* add(const Tensor& a, const Tensor& b);
Add* add(const Operator* a, const Operator* b);  

Class Design

Operator
public

• Operator* args;

There will be an identity operator which will act as a wrapper to tensor inputs so that uniformity can be attained in Operator and it's child classes.

from bnn.