belltailjp / chainer_computational_cost Goto Github PK

To prevent the error KeyError: 'hook FunctionHook already exists'.
c.f. https://docs.chainer.org/en/stable/reference/generated/chainer.FunctionHook.html#chainer.FunctionHook.name

Nice work, however, it seems that this tool doesn't support calculate the params and Flops of deconvolution and BN layers. Is there any solution?
Thanks a lot.

Need to

• drop Python 2 support
• deal with MaxPooling2D -> MaxPoolingND refactoring
• deal with mode and align_corners arguments in ResizeImages

cf https://github.com/chainer/chainer/releases/tag/v7.0.0b3

As we not have a theoretical computational cost, if we have a theoretical specification of a device, we can estimate theoretical (upper-bound) of inference speed on the device.
It'd be nice if chainer_computational_cost can have a feature to receive a hardware profile and calculate layer-wise and total theoretical speed.

Problems

• Interface design
  • Embed inference time inside layer report?
    • Pros: implementation is the simplest
    • Cons: Need to specify only one hardware profile to Hook instance.
• Layer-specific accelerator
  • NVIDIA Volta has TensorCore, in V100 case convolution operation is 125TFLOPs, although other general arithmetics are 15TFLOPs.
  • If we use only one value as the hardware's theoretical performance, the estimated inference time is totally useless.

Just for CI

In addition to memory transfer amount, estimating the total memory consumption would be nice.

Basic idea

• Simplest way: just sums up size of parameters and outputs of all layers
  • This would be an upper bound estimation
  • From hook, it's generally not easy to know what is parameter and what is input
    • Use LinkHook?
• How to deal with forgetting?
  • An output tensor can be erased if it is no longer used after a certain timing
  • In CNN memory consumption, this is a dominant part
  • So it needs tensor lifetime estimation
    • It is impossible until the entire NN process finishes, so Hook cannot simply solve this.
      • inputs can be obtained through func of a hook method, so the computational graph until right before the last layer can be reconstructed. Once a graph structure is completely known, theoretical memory life-time can be estimated.
        • There's a problem where Variable losts creator information, e.g. cond of `Where
    • Cumulate the total amount of chainer.Variables made during the hook lifetime, and subtract those who are forgotten, would this work?
      • Unable to get peak. Also it depends on how chainer's aggressive buffer release is implemented, which is not good.
• Returned value
  • Lower-bound: tensor-lifetime optimized memory consumption
  • Upper-bound: Just summing up everything

https://github.com/belltailjp/chainer_computational_cost/blob/master/DETAILS.md#sigmoid

says flops of sigmoid is 4*x but

https://github.com/belltailjp/chainer_computational_cost/blob/master/chainer_computational_cost/cost_calculators/activation.py#L87

returns x.size (not 4 * x.size)?