This is the implementation of Universal Representation Learning from Multiple Domains for Few-shot Classification (ICCV'21) and Cross-domain Few-shot Learning with Task-specific Adapters (CVPR'22) introduced by Wei-Hong Li, Xialei Liu, and Hakan Bilen.

• March'22, Code for Cross-domain Few-shot Learning with Task-specific Adapters (CVPR'22) is now available! See TSA.
• Oct'21, Code and models for Universal Representation Learning from Multiple Domains for Few-shot Classification (ICCV'21) are now available!

This code requires the following:

• Python 3.6 or greater
• PyTorch 1.0 or greater
• TensorFlow 1.14 or greater

• Clone or download this repository.
• Configure Meta-Dataset:
  • Follow the "User instructions" in the Meta-Dataset repository for "Installation" and "Downloading and converting datasets".
  • Edit ./meta-dataset/data/reader.py in the meta-dataset repository to change dataset = dataset.batch(batch_size, drop_remainder=False) to dataset = dataset.batch(batch_size, drop_remainder=True). (The code can run with drop_remainder=False, but in our work, we drop the remainder such that we will not use very small batch for some domains)
  • To test unseen domain (out-of-domain) performance on additional datasets, i.e. MNIST, CIFAR-10 and CIFAR-100, follow the installation instruction in the CNAPs repository to get these datasets.

1. Before doing anything, first run the following commands.

   ulimit -n 50000
   export META_DATASET_ROOT=<root directory of the cloned or downloaded Meta-Dataset repository>
   export RECORDS=<the directory where tf-records of MetaDataset are stored>
   

   Note the above commands need to be run every time you open a new command shell.

2. Enter the root directory of this project, i.e. the directory where this project was cloned or downloaded.

Figure 1. URL - Universal Representation Learning.

1. The easiest way is to download our pre-trained URL model and evaluate its feature using our Pre-classifier Alignment (PA). To download the pretrained URL model, one can use gdown (installed by pip install gdown) and execute the following command in the root directory of this project:

   gdown https://drive.google.com/uc?id=1Dv8TX6iQ-BE2NMpfd0sQmH2q4mShmo1A && md5sum url.zip && unzip url.zip -d ./saved_results/ && rm url.zip
   
   

   This will donwnload the URL model and place it in the ./saved_results directory. One can evaluate this model by our PA (see the Meta-Testing step)

2. Alternatively, one can train the model from scratch: 1) train 8 single domain learning networks; 2) train the universal feature extractor as follow.

1. The easiest way is to download our pre-trained models and use them to obtain a universal set of features directly. To download single domain learning networks, execute the following command in the root directory of this project:

   gdown https://drive.google.com/uc?id=1MvUcvQ8OQtoOk1MIiJmK6_G8p4h8cbY9 && md5sum sdl.zip && unzip sdl.zip -d ./saved_results/ && rm sdl.zip
   

   This will download all single domain learning models and place them in the ./saved_results directory of this project.

2. Alternatively, instead of using the pretrained models, one can train the models from scratch. To train 8 single domain learning networks, run:

   ./scripts/train_resnet18_sdl.sh
   

To learn the universal feature extractor by distilling the knowledge from pre-trained single domain learning networks, run:

./scripts/train_resnet18_url.sh

Figure 2. PA - Pre-classifier Alignment for Adapting Features in Meta-test.

This step would run our Pre-classifier Alignment (PA) procedure per task to adapt the features to a discriminate space and build a Nearest Centroid Classifier (NCC) on the support set to classify query samples, run:

./scripts/test_resnet18_pa.sh

Figure 3. Cross-domain Few-shot Learning with Task-specific Adapters (TSA).

We provide code for attaching task-specific adapters (TSA) to a single universal network learned from meta-train and learn the task-specific adapters on the support set. One can download our pre-trained URL model and evaluate its feature adapted by residual adapters in matrix form and pre-classifier alignment, run:

./scripts/test_resnet18_tsa.sh

Below are the results extracted from our papers. The results will vary from run to run by a percent or two up or down due to the fact that the Meta-Dataset reader generates different tasks each run, randomnes in training the networks and in TSA and PA optimization. Note, the results are updated with the up-to-date evaluation from Meta-Dataset. Make sure that you use the up-to-date code from the Meta-Dataset repository to convert the dataset and set shuffle_buffer_size=1000 as mentioned in google-research/meta-dataset#54.

Models trained on all datasets

To train a vanilla multi-domain learning network (MDL) on Meta-Dataset, run:

./scripts/train_resnet18_mdl.sh

One can use other classifiers for meta-testing, e.g. use --test.loss-opt to select nearest centroid classifier (ncc, default), support vector machine (svm), logistic regression (lr), Mahalanobis distance from Simple CNAPS (scm), or k-nearest neighbor (knn); use --test.feature-norm to normalize feature (l2) or not for svm and lr; use --test.distance to specify the feature similarity function (l2 or cos) for NCC.

To evaluate the feature extractor with NCC and cosine similarity, run:

python test_extractor.py --test.loss-opt ncc --test.feature-norm none --test.distance cos --model.name=url --model.dir <directory of url> 

One can evaluate the feature extractor in meta-testing for five-shot or five-way-one-shot setting by setting --test.type as '5shot' or '1shot', respectively.

To test the feature extractor for varying-way-five-shot on the test splits of all datasets, run:

python test_extractor.py --test.type 5shot --test.loss-opt ncc --test.feature-norm none --test.distance cos --model.name=url --model.dir <directory of url>

We thank authors of Meta-Dataset, SUR, Residual Adapter for their source code.

If you use this code, please cite our papers:

@article{li2022Universal,
    author    = {Li, Wei-Hong and Liu, Xialei and Bilen, Hakan},
    title     = {Universal Representations: A Unified Look at Multiple Task and Domain Learning},
    journal   = {arXiv preprint arXiv:2204.02744},
    year      = {2022}
}

@inproceedings{li2022TaskSpecificAdapter,
    author    = {Li, Wei-Hong and Liu, Xialei and Bilen, Hakan},
    title     = {Cross-domain Few-shot Learning with Task-specific Adapters},
    booktitle = {IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2022}
}

@inproceedings{li2021Universal,
    author    = {Li, Wei-Hong and Liu, Xialei and Bilen, Hakan},
    title     = {Universal Representation Learning From Multiple Domains for Few-Shot Classification},
    booktitle = {IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2021},
    pages     = {9526-9535}
}

@inproceedings{li2020knowledge,
    author    = {Li, Wei-Hong and Bilen, Hakan},
    title     = {Knowledge distillation for multi-task learning},
    booktitle = {European Conference on Computer Vision (ECCV) Workshop},
    year      = {2020},
    xcode     = {https://github.com/VICO-UoE/KD4MTL}
}