principle about batch_fold HOT 2 OPEN

We can use Batch Folding to replace the Convolution(or Dense) followed by Batch Normalization operation.

(Conv)-(BN) --> (Conv with updated w&b)

from batch_fold.

The stability of BN comes from each node would have a "relational ratio" between itself and the rest of the nodes.
So, each time we update the weights, the update is done such that the whole "relation" is optimized.
Analogy: Instead of increase individual's income, we target to increase a whole family's income. (More stable)

from batch_fold.