OSLOPROMPT: Bridging Low-Supervision Challenges and Open-Set Domain Generalization in CLIP

Mohamad Hassan N C1, Divyam Gupta1, Mainak Singha1, Sai Bhargav Rongali1, Ankit Jha2, Muhammad Haris Khan3, Biplab Banerjee1
1Indian Institute of Technology Bombay 2The LNM Institute of Information Technology (LNMIIT) 3Mohamed Bin Zayed University of Artificial Intelligence

Abstract

We introduce Low-Shot Open-Set Domain Generalization (LSOSDG), a novel paradigm unifying low-shot learning with open-set domain generalization (ODG). While prompt-based methods using models like CLIP have advanced DG, they falter in low-data regimes (e.g., 1-shot) and lack precision in detecting open-set samples with fine-grained semantics related to training classes. To address these challenges, we propose OSLoPrompt, an advanced prompt-learning framework for CLIP with two core innovations. First, to manage limited supervision across source domains and improve DG, we introduce a domain-agnostic prompt-learning mechanism that integrates adaptable domain-specific cues and visually guided semantic attributes through a novel cross-attention module, besides being supported by learnable domain- and class-generic visual prompts to enhance cross-modal adaptability. Second, to improve outlier rejection during inference, we classify unfamiliar samples as unknown and train specialized prompts with systematically synthesized pseudo-open samples that maintain fine-grained relationships to known classes, generated through a targeted query strategy with off-the-shelf foundation models. This strategy enhances feature learning, enabling our model to detect open samples with varied granularity more effectively. Extensive evaluations across five benchmarks demonstrate that OSLoPrompt establishes a new state-of-the-art in LSOSDG, significantly outperforming existing methods.

Method

Paper Preview

To address these challenges, we propose OSLOPROMPT, a novel prompt-learning framework for CLIP featuring:

  • 📌 Domain-agnostic prompt-learning with visually guided semantic attributes from domain specific prompts.
  • 🖼️ Fine-grained pseudo-open sample synthesis using GPT-4o & Stable Diffusion.
  • 📊 State-of-the-art results across five benchmark datasets, significantly outperforming existing DG/ODG methods.

Pseudo Open samples

Paper Preview