TokenMatch: 3D Mesh Correspondence Transformer with Curvature-Guided Tokenisation

While data-driven 3D shape correspondence estimation has recently seen substantial progress, robust matching under partial observations and strong non-isometric deformations remains challenging. Existing learning-based approaches often rely on hand-crafted descriptors or template-based representations, whereas recent generative models over functional maps suffer from high inference cost, limited interpretability, and poor generalisation to partial shapes. In response to these limitations, this paper introduces TokenMatch, a new transformer-based unified model for estimating 3D shape correspondences. Our feed-forward approach trained exclusively on BeCoS, a challenging non-isometric partial-to-partial shape-matching dataset, can generalise to matching full shapes without retraining or fine-tuning.

TokenMatch uses self- and cross-attention mechanisms to efficiently learn patch-level and point-level relations as well as dense correspondences between shape pairs. Our core insight is that meshes can be adaptively tokenised into patches using shape curvature guidance, enabling effective learning of shape-specific geometric descriptors for correspondence estimation. We evaluate TokenMatch on standard benchmarks for partial and full shape matching, including CP2P, PSMAL, BeCoS, FAUST, SCAPE, and SHREC’19. Our method achieves consistently high performance, in most cases outperforming existing methods for partial and full shape matching in the mean geodesic error and intersection-over-union metrics, while also running faster at sub-second inference speeds.

Our framework diverges from standard spatial sampling by prioritising geometric distinctness. First, we compute the local curvature for every vertex. We then construct shape tokens using a geodesic farthest-point sampling weighted by curvature magnitude, ensuring that critical features like fingertips or joint creases are actively tokenised.

Figure 1: Overview of our 3D mesh correspondence estimation framework. Given a pair of input meshes, we extract geometry-aware tokens and process them with a transformer encoder to obtain shape features. Cross-attention enables inter-shape interaction, and the resulting representations are used to estimate functional maps. An additional overlap prediction module identifies shared regions between shapes. The model is trained using self-supervised pretraining, functional map supervision, and geometric consistency objectives.

This figure visualises the Masked Autoencoding Pretraining on 3D human pose models. It starts with a solid 3D human pose provided as the input shape, which is then decomposed into colorful overlapping patches, or shape tokens. After extracting their centers, a large portion of these tokens is masked, and the resulting partially visible shape is passed through an encoder-decoder ($\mathcal{E}|\mathcal{D}$) network architecture. The network generates a partial reconstruction of the shape from the unmasked patches. Finally, to train the network, a Feature Loss ($\mathcal{L}_{feat}$) and a Chamfer Loss ($\mathcal{L}_{CD}$) are computed by comparing the partial reconstruction against the original full shape and its features.

Tokenisation design exploration: We compare alternative mesh tokenisation strategies to highlight the trade-offs between global structure, locality, and spatial continuity, ultimately motivating our curvature-guided approach. Click any figure below to expand it.

This project extensively evaluates robustness using the BeCoS dataset. For more details on the dataset, its construction, and to access the data, please refer to the official website:

Explore the BeCoS Dataset