This repository contains the assignments and final project for the NTU Deep Learning for Computer Vision 2023 course. There are a total of four assignments and one final project. Detailed information is provided below.

• Validation Accuracy:

  • Model A (ResNet-18): 74.96%
  • Model B (EfficientNetV2-S): 90.20%

• PCA Visualization: Limited class separation, with only some classes clearly distinguishable.

• t-SNE Visualization: Improved class separations compared to PCA, with better clustering.

• mIoU (Mean Intersection over Union):
  • VGG16-FCN32s: 0.7253
  • DeepLabV3-ResNet50: 0.7597

• Inference Time:

  • Initial: 20 minutes for 1000 images
  • After Disabling CFG: 10 minutes

• Accuracy:

  • With CFG: 99.5%
  • Without CFG: 96%

• Eta Values & Image Diversity:

  • Eta = 0: Denoised images identical to originals
  • Eta ≥ 0.5: Increased diversity in denoised images

• Interpolation Observations:

  • Spherical Linear Interpolation (Slerp): Smooth transitions between facial features
  • Simple Linear Interpolation: Blurred intermediate images with less detail

• Validation Accuracy:
  • “This is a photo of {object}”: 67.48%
  • “This is not a photo of {object}”: 69.64%
  • “No {object}, no score.” 45.24%
  • CIFAR100 prompt templates: 82.48%

• Example Images:
  • Correctly identified objects in two out of three example images.
  • Uncertainty in distinguishing between a tree and a bicycle in the third image.

This project builds upon the Flipped-VQA architecture to enhance video-text representation and question answering capabilities. The project introduces significant improvements by replacing key components in the model, such as the visual encoder and the underlying language model. The enhancements are designed to improve the accuracy and overall performance of the model in predicting answers (A), questions (Q), and video frames (V) given pairs of VQ, VA, and QA.

• Visual Encoder Replacement: The visual encoder is replaced with ViCLIP, a video CLIP model specifically designed for transferrable video-text representation.

• Language Model Upgrade: The project upgrades from LLAMA1-7B to LLAMA2-7B, a more powerful large language model (LLM).

• Int_Acc (Interaction Accuracy): 65.12
• Seq_Acc (Sequence Accuracy): 68.38
• Pre_Acc (Prediction Accuracy): 58.10
• Fea_Acc (Feasibility Accuracy): 50.78
• Mean: 60.60