Continual Learning in Open-vocabulary Classification with Complementary Memory Systems

Zhen Zhu, Weijie Lyu, Yao Xiao, Derek Hoiem

Transactions on Machine Learning Research (TMLR).

[Abstract] [Paper] [Code]

We introduce a method for flexible and efficient continual learning in open-vocabulary image classification, drawing inspiration from the complementary learning systems observed in human cognition. Specifically, we propose to combine predictions from a CLIP zero-shot model and the exemplar-based model, using the zero-shot estimated probability that a sample’s class is within the exemplar classes. We also propose a “tree probe” method, an adaption of lazy learning principles, which enables fast learning from new examples with competitive accuracy to batch-trained linear models. We test in data incremental, class incremental, and task incremental settings, as well as ability to perform flexible inference on varying subsets of zero-shot and learned categories. Our proposed method achieves a good balance of learning speed, target task effectiveness, and zero-shot effectiveness.

Encouraging Disentangled and Convex Representation with Controllable Interpolation Regularization

Yunhao Ge, Zhi Xu, Yao Xiao, Gan Xin, Yunkui Pang, Laurent Itti.

IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023.

[Abstract] [Paper]

We propose Controllable Interpolation Regularization (CIR), a general module compatible with different algorithms that have auto-encoder structure, to encourage more convex and robust disentangled representation learning. Qualitative and quantitative experiments show improvement in disentangled representation learning and latent convexity by CIR, which helps achieve state-of-the-art attribute controllable image synthesis. Additional experiments demonstrate that CIR can also improve other downstream tasks, such as new attribute value mining, data augmentation, and eliminating dataset bias for fairness.

Contrastive Learning for Domain Transfer in Cross-Corpus Emotion Recognition

Yufeng Yin, Liupei Lu, Yao Xiao, Zhi Xu, Kaijie Cai, Haonan Jiang, Jonathan Gratch, Mohammad Soleymani.

The 9th International Conference on Affective Computing & Intelligent Interaction (ACII) 2021. Oral, acceptance rate 23.6%.

[Abstract] [Paper] [Code] [Video]

Domain adaptation cannot guarantee to preserve local features necessary for emotion recognition while reducing domain discrepancies in global features. In this paper, we propose Face wArping emoTion rEcognition (FATE) to address this problem. We employ first-order facial animation warping to generate a synthetic dataset and utilize contrastive learning to pre-train the encoder. Then, we fine-tune the encoder and the classifier with the source data. After fine-tuning, the model achieves superior emotion recognition performance by preserving the subtle facial features.