Region-Based Representations Revisited

Michal Shlapentokh-Rothman*, Ansel Blume*, Yao Xiao, Yuqun Wu, Sethuraman T V, Heyi Tao, Jae Yong Lee, Wilfredo Torres, Yu-Xiong Wang, Derek Hoiem.

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

[Abstract] [Paper] [Code] [Website] [Video]

We investigate whether region-based representations are effective for recognition. Regions were once a mainstay in recognition approaches, but pixel and patch-based fea- tures are now used almost exclusively. We show that re- cent class-agnostic segmenters like SAM can be effectively combined with strong self-supervised representations, like those from DINOv2, and used for a wide variety of tasks, including semantic segmentation, object-based image re- trieval, and multi-image analysis. Once the masks and fea- tures are extracted, these representations, even with linear decoders, enable competitive performance, making them well suited to applications that require custom queries. The representations’ compactness also makes them well-suited to video analysis and other problems requiring inference across many images.