📝 Abstract:
Foundation vision–language models such as CLIP achieve strong zero-shot performance, but adapting them to specialised few-shot tasks still remains difficult. Two failure modes recur: models tend to overfit to the few samples given druing training if the process is not carefully designed, and also model selection becomes unreliable because the few-shot validation set is too small to serve as a faithful proxy for test performance. Existing methods address these problems by different techniques such as parameter efficient fine-tuning which preserves most of the model’s weights frozen, prompt tuning which aims to learn only text input tokens that best translate to the FSL problem, or with prototypical networks re-structuring the objective as learning to match with a known class prototypes. Proposed architecture ConWise-FT fine-tunes only the final visual projection layer of a frozen image encoder with three extra ingredients: a Supervised Contrastive (SupCon) auxiliary loss that explicitly shapes a well-clustered embedding geometry alongside the cross-entropy objective; tail Stochastic Weight Averaging over late-epoch projecv tors to reduce variance; and a WiSE-FT-style post-training blend between the learned and the initial projector that acts as a regulariser. We further use the Regret metric to quantify the cost of unreliable model selection. On the eleven-dataset BiomedCoOp medical benchmark, ConWise-FT surpasses established prompt-learning baselines and is competitive with the state of the art; on the eleven-dataset natural-image benchmark it ranks second overall while requiring orders of magnitude less compute. The post-training block consistently lowers Regret and partly recovers the novel-class accuracy lost during base-to-novel adaptation on the medical benchmark, though novel-class generalisation remains a limitation of the single global blend.