Continual Learning in Open-vocabulary Classification with Complementary Memory Systems

Zhen Zhu, Weijie Lyu, Yao Xiao, Derek Hoiem

Transactions on Machine Learning Research (TMLR).

[Abstract] [Paper] [Code]

We introduce a method for flexible and efficient continual learning in open-vocabulary image classification, drawing inspiration from the complementary learning systems observed in human cognition. Specifically, we propose to combine predictions from a CLIP zero-shot model and the exemplar-based model, using the zero-shot estimated probability that a sample’s class is within the exemplar classes. We also propose a “tree probe” method, an adaption of lazy learning principles, which enables fast learning from new examples with competitive accuracy to batch-trained linear models. We test in data incremental, class incremental, and task incremental settings, as well as ability to perform flexible inference on varying subsets of zero-shot and learned categories. Our proposed method achieves a good balance of learning speed, target task effectiveness, and zero-shot effectiveness.

Region-Based Representations Revisited

Michal Shlapentokh-Rothman*, Ansel Blume*, Yao Xiao, Yuqun Wu, Sethuraman T V, Heyi Tao, Jae Yong Lee, Wilfredo Torres, Yu-Xiong Wang, Derek Hoiem.

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

[Abstract] [Paper] [Code] [Website] [Video]

We investigate whether region-based representations are effective for recognition. Regions were once a mainstay in recognition approaches, but pixel and patch-based fea- tures are now used almost exclusively. We show that re- cent class-agnostic segmenters like SAM can be effectively combined with strong self-supervised representations, like those from DINOv2, and used for a wide variety of tasks, including semantic segmentation, object-based image re- trieval, and multi-image analysis. Once the masks and fea- tures are extracted, these representations, even with linear decoders, enable competitive performance, making them well suited to applications that require custom queries. The representations’ compactness also makes them well-suited to video analysis and other problems requiring inference across many images.

Encouraging Disentangled and Convex Representation with Controllable Interpolation Regularization

Yunhao Ge, Zhi Xu, Yao Xiao, Gan Xin, Yunkui Pang, Laurent Itti.

IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023.

[Abstract] [Paper]

We propose Controllable Interpolation Regularization (CIR), a general module compatible with different algorithms that have auto-encoder structure, to encourage more convex and robust disentangled representation learning. Qualitative and quantitative experiments show improvement in disentangled representation learning and latent convexity by CIR, which helps achieve state-of-the-art attribute controllable image synthesis. Additional experiments demonstrate that CIR can also improve other downstream tasks, such as new attribute value mining, data augmentation, and eliminating dataset bias for fairness.

A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts

Yunhao Ge, Yao Xiao, Zhi Xu, Meng Zheng, Srikrishna Karanam, Terrence Chen, Laurent Itti and Ziyan Wu.

IEEE/ CVF International Conference on Computer Vision and Pattern Recognition (CVPR), 2021.

[Abstract] [Paper] [Code] [Website] [Video] [Poster]

We propose a framework (VRX) to interpret classification NNs with intuitive structural visual concepts. Given a trained classification model, the proposed VRX extracts relevant class-specific visual concepts and organizes them using structural concept graphs (SCG) based on pairwise concept relationships. By means of knowledge distillation, we show VRX can take a step towards mimicking the reasoning process of NNs and provide logical, concept-level explanations for final model decisions.

Contrastive Learning for Domain Transfer in Cross-Corpus Emotion Recognition

Yufeng Yin, Liupei Lu, Yao Xiao, Zhi Xu, Kaijie Cai, Haonan Jiang, Jonathan Gratch, Mohammad Soleymani.

The 9th International Conference on Affective Computing & Intelligent Interaction (ACII) 2021. Oral, acceptance rate 23.6%.

[Abstract] [Paper] [Code] [Video]

Domain adaptation cannot guarantee to preserve local features necessary for emotion recognition while reducing domain discrepancies in global features. In this paper, we propose Face wArping emoTion rEcognition (FATE) to address this problem. We employ first-order facial animation warping to generate a synthetic dataset and utilize contrastive learning to pre-train the encoder. Then, we fine-tune the encoder and the classifier with the source data. After fine-tuning, the model achieves superior emotion recognition performance by preserving the subtle facial features.