AmbiGest: A Dataset of Social Gestures with Inter-Class Similarity and Intra-Class Variability

We propose AmbiGest, a dataset of dyadic social gestures designed for action recognition, interaction understanding, and reaction generation. AmbiGest contains 117 short clips organized into four communicative categories: greetings, calling, refusal, and thanks.

The dataset is designed around two central challenges: intra-class variability, where the same communicative intent can be performed in different styles, and inter-class similarity, where different intents may share visually similar motion patterns.

To preserve participant anonymity, raw RGB videos are not released. Instead, AmbiGest provides extracted 3D SMPL-X representations together with hierarchical annotations.

AmbiGest focuses on subtle everyday social gestures rather than large-scale action categories. It is designed to evaluate whether models can reason about communicative intent when different gestures look similar or when the same intention is expressed in different ways.

• 117 dyadic gesture sequences.
• 4 communicative categories: greetings, calling, refusal, and thanks.
• 3 actor pairs for studying interpersonal variability.
• Hierarchical annotations: class, subclass, semantic group, actor-pair ID, take index, and natural language description.

AmbiGest deliberately combines two properties that make social gesture understanding challenging:

• Intra-class variability: the same intent can be performed in different styles, amplitudes, and timings across actor pairs.
• Inter-class similarity: different intents can share similar motion patterns, such as a wave used for greeting and a wave used to call someone.

This design encourages models to move beyond coarse motion cues and to reason about social meaning.

The examples below illustrate the two main challenges targeted by AmbiGest: intra-class variability, where the same intention appears in different styles, and inter-class similarity, where different intentions may look visually close.

AmbiGest provides two complementary 3D reconstruction outputs:

• PromptHMR: preserves the global placement and relative spatial positioning of the two subjects, making it useful for interaction-level spatial reasoning.
• SMPLest-X: provides more expressive whole-body motion and finer hand articulation, but with less reliable relative placement between the two subjects.

Releasing both representations allows researchers to select the representation that best matches their task or to compare performance across reconstruction pipelines.

We evaluate gesture recognition using a leave-one-couple-out protocol, where two actor pairs are used for training and the remaining pair is used for testing. Two temporal models are compared: a TCN and a Transformer encoder.

The results show that interaction context is important for resolving ambiguity. With PromptHMR features, actor-only input is limited, while adding the reactor and translation improves performance, reaching 91.3% accuracy with the Transformer. With SMPLest-X features, the best Transformer configuration reaches 89.0%.

This project is supported by the France 2030 program, the FR CNRS 2052 Sciences et Cultures du Visuel, and the ANR Investissements d’Avenir program Equipex+ Continuum.