Overview
Tactile Localization
We introduce tactile localization, a task where a model is given a tactile input and must identify all regions in an image that share the same material properties. Unlike prior visuo-tactile methods that determine whether an image and a touch correspond to the same material, our model localizes where in a visual scene a given tactile sensation exists — producing dense saliency maps that highlight material regions matching the touch input. This cross-modal task requires fine-grained spatial reasoning beyond the coarse semantic alignment addressed by existing approaches.
Framework
Seeing Through Touch (STT) encodes paired tactile and visual inputs into a shared feature space and learns fine-grained local cross-modal alignment via contrastive learning. Rather than comparing global pooled representations, STT computes dense similarity maps between aggregated tactile features and local visual features, enabling spatially precise material localization.
Web-Material Dataset
Existing visuo-tactile datasets consist mostly of close-up, single-material images that lack scene-level diversity. We introduce Web-Material, a new dataset comprising web-crawled scene-level images and curated samples from prior material recognition datasets, together covering 32K images across 18 categories. Each image captures materials in diverse real-world contexts, and is paired with tactile signals from Touch-and-Go[1] via material diversity-based pairing. Web-Material serves as both a training source and an evaluation benchmark with pixel-level material segmentation annotations.
