To keep track of the large number of recent papers that look at the intersection of Transformers and Neural Architecture Search (NAS), we have created this awesome list of curated papers and resources, inspired by awesome-autodl, awesome-architecture-search, and awesome-computer-vision. Papers are divided into the following categories:
- General Transformer search
- Domain Specific, applied Transformer search (divided into NLP, Vision, ASR)
- Transformers Knowledge: Insights / Searchable parameters / Attention
- Transformer Surveys
This repository is maintained by the AutoML Group Freiburg. Please feel free to pull requests or open an issue to add papers.
| Title | Venue | Group |
|---|---|---|
| Searching the Search Space of Vision Transformer | NeurIPS'21 | MSRA, Stony Brook University |
| UniNet: Unified Architecture Search with Convolutions, Transformer and MLP | arxiv [Oct'21] | SenseTime |
| Analyzing and Mitigating Interference in Neural Architecture Search | arxiv [Aug'21] | Tsinghua, MSR |
| BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search | ICCV'21 | Sun Yat-sen University |
| Memory-Efficient Differentiable Transformer Architecture Search | ACL-IJCNLP'21 | MSR, Peking University |
| Finding Fast Transformers: One-Shot Neural Architecture Search by Component Composition | arxiv [Aug'20] | Google Research |
| AutoTrans: Automating Transformer Design via Reinforced Architecture Search | arxiv [Sep'20] | Fudan University |
| NAT: Neural Architecture Transformer for Accurate and Compact Architectures | NeurIPS'19 | Tencent AI |
| The Evolved Transformer | ICML'19 | Google Brain |
| Title | Venue | Group |
|---|---|---|
| NASViT: Neural Architecture Search for Efficient Vision Transformers with Gradient Conflict aware Supernet Training | ICLR'22 | Meta Reality Labs |
| AutoFormer: Searching Transformers for Visual Recognition | ICCV'21 | MSR |
| GLiT: Neural Architecture Search for Global and Local Image Transformer | ICCV'21 | University of Sydney |
| Searching for Efficient Multi-Stage Vision Transformers | ICCV'21 workshop | MIT |
| HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers | CVPR'21 | Bytedance Inc. |
| ViTAS: Vision Transformer Architecture Search | arxiv [June'21] | SenseTime, Tsingua University |
| Title | Venue | Group |
|---|---|---|
| AutoBERT-Zero: Evolving the BERT backbone from scratch | AAAI'22 | Huawei Noah’s Ark Lab |
| Primer: Searching for Efficient Transformers for Language Modeling | NeurIPS'21 | |
| AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models | ACL'21 | MIT |
| NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search | KDD'21 | MSR, Tsinghua University |
| HAT: Hardware-Aware Transformers for Efficient Natural Language Processing | ACL'20 | MIT |
| Title | Venue | Group |
|---|---|---|
| LightSpeech: Lightweight and Fast Text to Speech with Neural Architecture Search | ICASSP'21 | MSR |
| Darts-Conformer: Towards Efficient Gradient-Based Neural Architecture Search For End-to-End ASR | arxiv [Aug'21] | NPU, Xi'an |
| Improved Conformer-based End-to-End Speech Recognition Using Neural Architecture Search | arxiv [April'21] | Chinese Academy of Sciences |
| Evolved Speech-Transformer: Applying Neural Architecture Search to End-to-End Automatic Speech Recognition | INTERSPEECH'20 | VUNO Inc. |
| Title | Venue | Group |
|---|---|---|
| Transformers in Vision: A Survey | arxiv [Oct'21] | MBZ University of AI |
| A Survey of Visual Transformers | arxiv [Nov'21] | CAS |
| Efficient Transformers: A Survey | arxiv [Sept'21] | Google Research |
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent