See & Sniff

Abstract

While modern multimodal models integrate vision with language, audio, or touch, olfaction remains largely unexplored due to the lack of paired visuo-olfactory data. We introduce SmellNet-V, a scalable visuo-olfactory dataset built on the insight that odor identity is largely invariant to visual transformations within a semantic category. This allows us to synthetically pair smell-only samples with semantically aligned in-the-wild web images, converting a unimodal olfactory dataset into a cross-modal benchmark without costly co-collection. Building on this dataset, we propose See & Sniff, a self-supervised framework that learns joint visuo–olfactory representations via dense local alignment and naturally produces smell saliency maps for spatial grounding of odor sources. We further introduce pixel-level smell localization task and a benchmark for evaluation. Our method surpasses smell-only baselines by 7% in smell classification from smell alone and generalizes to cross-modal retrieval and smell localization, establishing visuo-olfactory learning as a new direction in multimodal perception.

Overview

We show that See & Sniff, the framework that learns joint visuo-olfactory representations, can handle both unimodal and cross-modal tasks. (Left) Smell localization identifies the locations of smell sources within a visual scene based on input olfactory signals. (Right Top) Cross-modal retrieval demonstrates semantic alignment through bidirectional retrieval. (Right Bottom) Smell classification predicts ingredients from olfactory inputs.

Framework

See & Sniff is a self-supervised framework that learns joint visuo-olfactory representations from SmellNet-V. A frozen vision backbone and an olfactory transformer backbone, each with a lightweight aligner, project both modalities into one shared embedding space. Instead of aligning only global features, See & Sniff uses dense local alignment, measuring similarity between a sniff and every region of the image. This local similarity map serves as a smell saliency map that grounds odor sources, and the learned representation transfers across classification, retrieval, and localization.

SmellNet-V Dataset

16K images 50 categories Pixel-level annotations

We introduce SmellNet-V, a visuo-olfactory extension of SmellNet[1], a dataset originally built for smell classification from smell alone. SmellNet-V is constructed by pairing each smell sample in SmellNet with semantically aligned web images, leveraging the insight that odor identity is largely invariant to visual transformations within a semantic category. It serves as both a training source covering 16K images across 50 ingredient categories, and an evaluation benchmark with pixel-level annotations for smell localization.

Results

Smell Classification

Classify which of the 50 ingredients an olfactory signal corresponds to, using a linear probe on the frozen smell encoder. With no image at inference, See & Sniff still beats the smell-only baseline by 7%, showing that vision is effective supervision for olfaction.

Model	p = 25, W = 50		p = 25, W = 100
Model	Acc.	F1	Acc.	F1
SmellNet (LSTM)	50.6	48.8	57.9	56.0
SmellNet (Transformer)	50.6	49.5	56.1	55.5
Global-CLIP	53.19	52.22	61.55	60.01
Ours-Global	53.74	52.64	59.36	57.96
Ours-Local	54.94	53.78	62.75	62.31
See & Sniff	57.71	56.68	63.75	62.66

Cross-Modal Retrieval

Retrieve images of the same ingredient given a smell, and vice versa. See & Sniff gives the best R@1 in both directions, confirming semantic alignment between vision and smell.

Model	Smell → Vision			Vision → Smell
Model	R@1	R@5	R@10	R@1	R@5	R@10
Global-CLIP	53.28	55.96	58.82	56.90	87.40	91.40
Ours-Global	50.97	62.60	67.50	52.80	78.30	85.30
Ours-Local	53.65	58.45	61.50	55.40	82.90	88.90
See & Sniff	56.14	60.94	63.90	63.20	87.40	91.90

Cross-Modal Retrieval Qualitative Results

Figure 1. Qualitative Cross-Modal Retrieval Results. Blue borders indicate correct matches, and red borders indicate mismatches. The failure cases are visually similar to the queried ingredients, indicating fine-grained visual ambiguity rather than random mismatch.

Smell → Smell Retrieval

Retrieve other smell samples of the same ingredient. The large gain over SmellNet shows that visual supervision also strengthens the smell space itself, since inference uses smell features only.

Method	R@1	R@5	R@10
SmellNet	52.54	76.18	85.50
See & Sniff	69.44	85.04	89.47

Smell Localization

Localize the image region expected to emit a given smell, a new task we introduce with a pixel-level benchmark. Emerging from dense local alignment, See & Sniff outperforms other baselines.

Model	mAP	mIoU
Visual Heatmap
DINOv3 Att. Map	0.6596	0.5157
Global Alignment
Global-CLIP	0.1736	0.2073
Ours-Global	0.6676	0.5016
Local Alignment
Ours-Local	0.7970	0.6099
See & Sniff	0.8362	0.6456

Figure 2. Qualitative Smell Localization Results.

Interactive Localization

With several smells for one image, localize each smell's region. The predicted region shifts with the input smell rather than the most salient object, showing that localization is driven by smell semantics.

Model	IIoU
Visual Heatmap
DINOv3 Att. Map	0.01
Global Alignment
Global-CLIP	0.00
Ours-Global	0.22
Local Alignment
Ours-Local	0.32
See & Sniff	0.35

Interactive Localization Qualitative Results

Figure 3. Qualitative Results on Interactive Localization.

Additional Results

Additional qualitative examples.

Figure 4. Additional Qualitative Smell Localization Results.

BibTeX

@inproceedings{kim2026seeandsniff, author = {Seongyu Kim and Seungwoo Lee and Hyeonggon Ryu and Joon Son Chung and Arda Senocak}, title = {See \& Sniff: Learning Visuo-Olfactory Representations}, booktitle = {European Conference on Computer Vision}, year = {2026}, }

References

Dewei Feng et al., "SmellNet: A Large-Scale Dataset for Real-World Smell Recognition," ICLR, 2026.

See & Sniff: Learning Visuo-Olfactory Representations