This is a paper list for neural network compression techniques such as quantization, pruning, and distillation. Most of the papers in the list are about quantization of CNNs. Only official codes are crosslinked.

format: ([Nickname]) Paper title, published @, paper link, (official code (if provided))

• [BNN] Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1, arXiv 2016, [paper], [code(Theano)] [code(Torch-7)], [code(Pytorch)], [code(Tensorflow)]

• [XNOR-NET] XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks, ECCV 2016, [paper], [code(Torch-7)]

• [DoReFa] DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients, arXiv 2016, [paper], [code(Tensorflow)]

• [HWGQ] Deep Learning With Low Precision by Half-Wave Gaussian Quantization, CVPR 2017, [paper], [code(caffe)]

• [TWN] Ternary Weight Networks, NIPS workshop 2016, [paper], [code(caffe)]

• [TTQ] Trained Ternary Quantization, ICLR 2017, [paper], [code(Tensorflow)]

• How to train a compact binary neural network with high accuracy?, AAAI 2017, [paper]

• [ABC-Net] Towards Accurate Binary Convolutional Neural Network, NIPS 2017, [paper]

• [WEQ] Weighted-Entropy-Based Quantization for Deep Neural Networks, CVPR 2017, [paper], [code(caffe)]

• [Network Sketching] Network Sketching: Exploiting Binary Structure in Deep CNNs, CVPR 2017, [paper]

• [WRPN] WRPN: Wide Reduced-Precision Networks, ICLR 2018, [paper]

• [PACT] PACT: Parameterized Clipping Activation for Quantized Neural Networks, arXiv 2018, [paper]

• [Bi-Real-Net] Bi-Real Net: Enhancing the Performance of 1-bit CNNs with Improved Representational Capability and Advanced Training Algorithm, ECCV 2018, [paper], [code(caffe)]

• [SYQ] SYQ: Learning Symmetric Quantization For Efficient Deep Neural Networks, CVPR 2018, [paper], [code(Tensorflow)]

• Towards Effective Low-Bitwidth Convolutional Neural Networks, CVPR 2018, [paper], [code(Pytorch)]

• [TSQ] Two-Step Quantization for Low-bit Neural Networks, CVPR 2018, [paper]

• [LQ-Net] LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks, ECCV 2018, [paper], [code(Tensorflow)]

• Alternating Multi-bit Quantization for Recurrent Neural Networks, ICLR 2018, [paper]

• [NICE] NICE: Noise Injection and Clamping Estimation for Neural Network Quantization, arXiv 2018, [paper]

• [Continuous Binarization] True Gradient-Based Training of Deep Binary Activated Neural Networks Via Continuous Binarization, ICASSP 2018, [paper]

• [MCDQ] Model compression via distillation and quantization, ICLR 2018, [paper], [code(Pytorch)]

• [Apprentice] Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy, ICLR 2018, [paper]

• [Integer] Training and Inference with Integers in Deep Neural Networks, ICLR 2018, [paper], [code(Tensorflow)]

• Heterogeneous Bitwidth Binarization in Convolutional Neural Networks, NIPS 2018, [paper]

• An empirical study of Binary Neural Networks' Optimisation, ICLR 2019, [paper]

• [PACT-SAWB] Accurate and Efficient 2-bit Quantized Neural Networks, SysML 2019, [paper]

• [QIL] Learning to Quantize Deep Networks by Optimizing Quantization Intervals With Task Loss, CVPR 2019, [paper]

• [Group-Net] Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation, CVPR 2019, [paper]

• [ProxQuant] ProxQuant: Quantized Neural Networks via Proximal Operators, ICLR 2019, [paper], [code(Pytorch)]

• [CBCN] Circulant Binary Convolutional Networks: Enhancing the Performance of 1-Bit DCNNs With Circulant Back Propagation, CVPR 2019, [paper]

• [CI-BCNN] Learning Channel-Wise Interactions for Binary Convolutional Neural Networks, CVPR 2019, [paper]

• [QN] Quantization Networks, CVPR 2019, [paper]

• [BNN+] BNN+: Improved Binary Network Training, arXiv 2019, [paper]

• [DistributionLoss] Regularizing Activation Distribution for Training Binarized Deep Networks, CVPR 2019, [paper], [code(Pytorch)]

• A Main/Subsidiary Network Framework for Simplifying Binary Neural Networks, CVPR 2019, [paper]

• Matrix and tensor decompositions for training binary neural networks, arXiv 2019, [paper]

• [BENN] Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?, CVPR 2019, [paper], [code(Pytorch)]

• Back to Simplicity: How to Train Accurate BNNs from Scratch?, arXiv 2019, [paper], [code(MXNet)]

• And the Bit Goes Down: Revisiting the Quantization of Neural Networks, arxiv 2019, [paper], [code(Pytorch)]

• [BinaryDuo] BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activations, ICLR 2020, [paper], [code(Torch-7)]

• [RtB] Training Binary Neural Networks with Real-to-Binary Convolutions, ICLR 2020, [paper]

• [LLSQ] Linear Symmetric Quantization of Neural Networks for Low-precision Integer Hardware, ICLR 2020, [paper], [code(Pytorch]

• [AutoQ] AutoQ: Automated Kernel-wise Neural Network Quantization, ICLR 2020, [paper]

• [APOT] Additive Powers-of-two Quantization: An Efficient Non-uniform Discretization for Neural Networks, ICLR 2020, [paper], [code(Pytorch)]

• [LSQ] Learned Step Quantization, ICLR 2020, [paper]

• [MetaQuant] MetaQuant: Learning to Quantize by Learning to Penetrate Non-differentiable Quantization, NeurIPS 2019, [paper], [code(Pytorch)]

• [BGNN] Binary Graph Neural Networks, CVPR 2021, [paper], [code(Pytorch)]

To be updated.

To be updated.

• [MobileNet] MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications, arXiv 2017, [paper]
• [MobileNet v2] MobileNetV2: Inverted Residuals and Linear Bottlenecks, CVPR 2018, [paper]
• [MobileNet v3] Searching for MobileNetV3, arXiv 2019, [paper]
• [Xception] Xception: Deep Learning With Depthwise Separable Convolutions, CVPR 2017, [paper]
• [ShuffleNet] ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices, CVPR 2018, [paper]
• [ShuffleNet v2] ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design, ECCV 2018, [paper]
• [SqueezeNet] SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size, arXiv 2016, [paper]
• [StrassenNets] StrassenNets: Deep Learning with a Multiplication Budget, ICML 2018, [paper]
• [SlimmableNet] Slimmable Neural Networks, ICLR 2019, [paper], [code(Pytorch)]
• [ChannelNets] ChannelNets: Compact and Efficient Convolutional Neural Networks via Channel-Wise Convolutions, NIPS 2018, [paper], [code(Tensorflow)]
• [Shift] Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions, CVPR 2018, [paper], [code(Pytorch)]
• [FE-Net] All You Need is a Few Shifts: Designing Efficient Convolutional Neural Networks for Image Classification, CVPR 2019, [paper]
• [EfficientNet] EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks, ICML 2019, [paper]

By Hyungjun Kim ([email protected]) from Pohang University of Science and Technology (POSTECH).