Yao Xiao

CS Ph.D. candidate @ UIUC

Google Scholar
GitHub
X
Linkedin

About Me

I am a final-year Ph.D. candidate in Computer Science at the University of Illinois Urbana-Champaign, advised by Prof. Derek Hoiem.

I aim to build intelligent agents that plan under partial information, remember what matters, and draw on past experience to avoid repeating failures. More specifically:

  • Closed-loop Planning: How can we make decisions under partial information? Each decision must build on past observations while shaping what comes next.

  • Memory Representation: How can a model compress its past into a compact, updatable state that preserves essential details and turns experience into reusable lessons?

  • Experience Retrieval: How can an agent retrieve useful past experiences or distilled reflective lessons?

I'm seeking full-time industry roles starting in summer 2027. Please email me if you know of a good match!

News

  • [Oct 8, 2026] I was recognized as a Top Reviewer at NeurIPS 2026 and awarded complimentary registration.

Selected Work

For the full publication list, please refer to my Google Scholar.

ReToken: Improving Long-Context VLMs with Visual Retrieval Token

Yao Xiao, Reuben Tan, Zhen Zhu, Yuqun Wu, Jianfeng Gao, Derek Hoiem

NeurIPS 2026.

[TL;DR]
Attention tells you where a VLM looks, so shouldn't it tell you which frames matter? It doesn't: the retrieval signal lives in the value space, and a single learned token is enough to read it out.
[Website] [Paper] [Code] [Poster]

TextRegion: Text-Aligned Region Tokens from Frozen Image-Text Models

Yao Xiao, Qiqian Fu, Heyi Tao, Yuqun Wu, Zhen Zhu, Derek Hoiem

NeurIPS 2026 Journal Track & TMLR 2025 J2C certificate.

[TL;DR]
A simple, training-free approach to get region tokens directly comparable to text embeddings, achieving state-of-the-art zero-shot region-level understanding.
[Website] [Paper] [Code] [Poster]

How to Teach Large Multimodal Models New Skills?

Zhen Zhu, Yiming Gong, Yao Xiao, Yaoyao Liu, Derek Hoiem

ECCV 2026πŸ…Spotlight.

[Website] [Paper] [Code]

Blog

A casual blog for sharing thoughts and tracking my journey. No tech talk, just real life.

β†’ Full list

Industry Experience

Microsoft Research

Microsoft Research, Deep Learning Group

Research Intern Β· Long-Context VLMs

Jul 2025 – Nov 2025
Amazon Web Services

Amazon, AWS AI Labs

Applied Scientist Intern Β· Segmentation via VLMs

May 2024 – Aug 2024

Academic Service

Last Updated: 11/22/2026, 10:49:44 AM