← 返回 2026-03-31

HandX：可扩展的双手机动与交互生成 HandX: Scaling Bimanual Motion and Interaction Generation

Zimu Zhang, Yucheng Zhang, Xiyan Xu, Ziyin Wang, Sirui Xu, Kai Zhou, Bing Zhou, Chuan Guo, Jian Wang, Yu-Xiong Wang, Liang-Yan Gui 📅 2026-03-30 👍 12 2026-07-13 08:36
LLM标注 人体动画 动作捕捉 双手机动生成 扩散模型 自回归模型
arXiv HuggingFace PDF

大规模双手机动数据集与生成框架，支持精细手指协同与接触交互

前置知识

扩散模型

扩散模型通过前向扩散过程逐步向数据添加高斯噪声，再通过学习反向去噪过程来从噪声中恢复数据。在连续时间t的前向过程中，数据x_0通过x_t = sqrt(bar-alpha_t)x_0 + sqrt(1-bar-alpha_t)epsilon转换为噪声版本，其中bar-alpha_t = prod_{t'=1}^t(1-beta_{t'})，beta_{t'}是噪声方差。训练目标是学习神经网络G(x_t, t, T)预测干净信号tilde{x}，推理时通过迭代去噪逐步从随机噪声生成样本。

本文采用扩散模型作为主要生成范式之一，理解其去噪原理对于掌握HandX如何生成精细双手机动至关重要。

有限标量量化(FSQ)

FSQ(Finite Scalar Quantization)是一种向量量化方法，将连续潜在表示映射到离散的整数空间。给定潜在特征y，通过y-hat = Q(y) = round(sigma(y) times (L-1))将其量化到L个均匀间隔的整数级别，其中sigma是sigmoid函数。优化目标为L = ||x - D(y-hat)||_2^2，D是解码器。相比传统VQ-VAE，FSQ能提供更好的码本利用率和重建质量。

自回归模型使用FSQ将连续运动序列离散化，理解FSQ的工作原理有助于理解本文AR模型的运动表示方式和token化策略。

动作捕捉与骨架重建

动作捕捉通过在皮肤表面贴反光标记物来跟踪人体运动。HandX使用36摄像头OptiTrack系统，每只手25个标记点覆盖手腕、掌背和每个手指的MCP、PIP、DIP关节及指尖。从表面标记重建骨架需要先计算法线方向n和深度d，通过J_p = M_p + n times d计算关节位置。手腕位置需要迭代优化以保持MCP到腕关节的骨长常数。

理解动作捕捉和骨架重建技术对于理解HandX数据集的高保真特性以及数据质量控制的重要性。

研究动机

现有的人体运动生成方法在处理手部运动时存在严重不足。大多数全身模型如SMPL将手部简化为刚体末端执行器，完全忽略手指关节的精细运动。即使包含手部模型的数据集如Motion-X(144.2小时)和InterAct(30.7小时)，也主要提供全身运动的粗粒度文本描述，而非手部细节。手部中心数据集如GigaHands(2.58小时)、ARCTIC(1.06小时)和H2O(0.47小时)虽然包含手部，但规模小、物体中心、标注粗糙，缺乏双手接触和协调的多样性。现有评估指标如FID、Diversity和R-Precision无法有效衡量手部接触质量，使得难以诊断生成失败和衡量进展。

本文的目标是本文目标是构建一个统一的数据基础来解决双手机动生成的数据瓶颈。具体包括：(1)通过整合现有高质量开源数据集和采集新的高保真双手机动数据，构建大规模数据集；(2)开发可扩展的自动标注框架，生成细粒度的多级文本描述；(3)建立扩散模型和自回归模型的基准，研究模型规模和数据规模的scaling行为；(4)提出接触专注的评估指标，如Contact Precision、Recall和F1分数，以量化双手接触的保真度。

与已有工作不同的是，本文的独特切入角度是同时解决数据和模型两个层面的问题。与CLUTCH[53]从野外视频重建手部运动不同，HandX使用专业动作捕捉系统获得高质量数据。与BOBSL3DT[6]专注于手语数据不同，HandX覆盖更广泛的双手机动类型。本文最核心的创新是解耦的标注策略：先提取结构化运动学特征(接触事件、手指屈曲、手掌间距等)，再利用大语言模型推理生成语义丰富的细粒度描述。这种两阶段策略使大规模高质量标注成为可能，这是以往工作未曾尝试的。

核心方法

HandX框架分为三个主要组件：数据集构建、自动标注和生成模型。数据集构建阶段通过整合7个现有开源数据集(Motion-X、InterAct、GigaHands、HOT3D、ARCTIC、H2O、HoloAssist)并统一骨骼表示和坐标系，获得初始数据。然后使用36摄像头OptiTrack系统采集新的双手机动数据，演员佩戴每只手25个反光标记点，覆盖手腕、掌背和手指关节。自动标注采用两阶段策略：首先计算6种运动学特征(手指屈曲、手指间距、指间距离、手掌关系、指掌距离、手腕轨迹)并分割为事件，然后将这些结构化特征输入T5编码器，通过设计的prompt让LLM生成5个不同粒度的文本描述。生成模型支持扩散和自回归两种范式，通过masked partial denoising实现多种条件生成任务。

核心创新点包括：(1)解耦的标注框架，将高维连续运动转化为结构化特征，再让LLM基于特征生成语义描述，避免了让LLM直接处理原始运动数据的困难；(2)运动表示中的旋转标量，为每个关节添加1-DoF旋转标量s_i，通过将关节角度alpha投影到垂直于食指MCP到小指MCP向量的平面计算s = |proj_perp(alpha)|，使表示为x_i = [p_i; s_i] in R^(2J times 4)；(3)分离的文本嵌入处理，扩散模型分别编码左手、右手和双手交互三个prompt，使用可学习的CLS token区分类型，通过tilde{z} = z'_t + sum_{k in {L,R,I}} CrossAttention(z'_t, T_k)融合；(4)versatile generation通过部分去噪实现spatiotemporal control，在去噪时混合预测和目标x_0(t, j) = (1-gamma_{t,j})x^{pred}_0(t, j) + gamma_{t,j}x^{gt}_0(t, j)。

方法步骤详情

数据构建步骤：(1)从7个开源数据集提取双手机动序列，统一到21关节骨骼拓扑和右手坐标系；(2)使用OptiTrack采集新数据，25标记/手，36摄像头，重建骨架时通过优化腕关节位置保持MCP-腕骨长常数；(3)将长序列分割为60帧(2秒)clip，检测并移除缺陷帧；(4)计算clip强度bar-omega_h(W) = (sum_{t in W}sum_{j in J_h} lambda_j omega^{(j)}_t)/(sum_{t in W}sum_{j in J_h} lambda_j)，仅保留bar-omega_left >= 25, bar-omega_right >= 25, bar-omega >= 30的clip。标注步骤：(1)计算6种运动学特征并分割为事件，存为JSON；(2)设计prompt要求描述左右手及其关系、报告接触分离事件、包含时间上下文；(3)让LLM生成5个粒度的描述。扩散模型训练：(1)前向过程x_t = sqrt(bar-alpha_t)x_0 + sqrt(1-bar-alpha_t)epsilon；(2)编码x_t -> z_t = F(x_t)，timestep编码t，拼接z'_t = [t; z_t]；(3)T5编码三个prompt为T_L, T_R, T_I；(4)训练G(x_t, t, T)预测tilde{x}。AR模型训练：(1)FSQ量化y -> y-hat；(2)T5编码文本前缀T；(3)自回归预测下一个token y-hat_k | y_{<k}, T。

技术新颖性

技术新颖性体现在多个方面。在数据层面，HandX是首个大规模(54.2小时，590万帧，49万描述)双手机动数据集，其contact ratio(48.5)和contact duration(25-55秒)远超现有数据集，提供真正丰富的双手交互。在标注层面，解耦的两阶段策略与PoseScript[12]和MotionScript[68]相关但针对双手机动专门设计，提取了双手关系的独特特征。在模型层面，旋转标量设计简单有效，避免了复杂的四元数或旋转矩阵表示。versatile generation通过部分去噪实现多种控制模式，与InterDiff[60]的物理引导不同，HandX专注于时空控制。在评估层面，提出的contact metrics填补了双手交互质量评估的空白。最重要的是scaling law研究，观察到R_prec = 0.4391 times log_10(FLOPS) - 3.8707的对数线性关系，为该领域的scaling提供了实证依据。

(a) We introduce HandX, a large-scale dataset of bimanual and dexterous motions paired with fine-grained textual descriptions. The examples highlight the high-fidelity captures produced by our motion capture system (Figure A), and demonstrate instantiation on a real-world humanoid with dexterous hands. (b) We benchmark two generative paradigms: diffusion-based and autoregressive (AR) models. (c) Our models support flexible conditioning and synthesize highly dynamic, expressive hand motions. (d) We observe clear scaling trends: increasing dataset size and model capacity yields substantial performance gains.
Figure 1: (a) We introduce HandX, a large-scale dataset of bimanual and dexterous motions paired with fine-grained textual descriptions. The examples highlight the high-fidelity captures produced by our motion capture system (Figure A), and demonstrate instantiation on a real-world humanoid with dexterous hands. (b) We benchmark two generative paradigms: diffusion-based and autoregressive (AR) models. (c) Our models support flexible conditioning and synthesize highly dynamic, expressive hand motions. (d) We observe clear scaling trends: increasing dataset size and model capacity yields substantial performance gains.
Two benchmark models. (a) Diffusion model. Text embeddings for the left hand, right hand, and bimanual interaction are separately cross-attended with noisy motion embeddings, and then fused through residual connections to predict denoised motion embeddings. (b) Autoregressive model, consisting of Finite Scalar Quantization (FSQ) and a text-prefix autoregressive model. Unlike the diffusion model, it concatenates the left-hand, right-hand, and bimanual text descriptions with separator tokens to form a text prefix, and formulates bimanual motion generation as a token prediction task over motion tokenized by FSQ.
Figure 2: Two benchmark models. (a) Diffusion model. Text embeddings for the left hand, right hand, and bimanual interaction are separately cross-attended with noisy motion embeddings, and then fused through residual connections to predict denoised motion embeddings. (b) Autoregressive model, consisting of Finite Scalar Quantization (FSQ) and a text-prefix autoregressive model. Unlike the diffusion model, it concatenates the left-hand, right-hand, and bimanual text descriptions with separator tokens to form a text prefix, and formulates bimanual motion generation as a token prediction task over motion tokenized by FSQ.

实验结果

实验揭示了清晰的scaling趋势。对于扩散模型，在5%数据预算下运行更密集实验，Top-3 R-Precision与FLOPS呈现对数线性关系，相关系数0.96。完整数据集上，12层模型在contact metrics上表现最佳(Cprec=0.722, Crec=0.549, CF1=0.624)，而16层超大规模模型(参数量6.7倍)性能下降，表明存在饱和点。扩散模型scaling显示，同时增加模型深度和数据量持续改进主要指标，12层全数据达到Top-3 R-Precision 0.631，显著优于4层5%数据的0.296。对于自回归模型，仅增加FSQ codebook大小不能可靠提升性能，需要联合增加codebook和模型大小。最佳配置是92.27M模型size搭配2,048 codebook，达到FID 2.949，但R-Precision在模型与codebook匹配时最高。定性分析显示，全数据模型生成更富表现力的运动，文本对齐更好；更大模型产生更准确的双手接触。模拟到真实迁移显示，学到的灵巧技能可以迁移到配备灵巧机械手的人形机器人平台。

Comparison of major hand motion datasets. Left: Dataset scale. Values are reported as HQ (Raw) where HQ denotes high-quality filtered data and Raw (when available) indicates the original data. Coarse denotes short descriptions without articulation detail, while action denotes only categorical labels. HandX provides fine-grained, multi-level language descriptions. Right: Statistics of bimanual motion quality. Metrics are defined in Sec. A.5. HandX provides contact-rich bimanual motions.
Table 1: Comparison of major hand motion datasets. Left: Dataset scale. Values are reported as HQ (Raw) where HQ denotes high-quality filtered data and Raw (when available) indicates the original data. Coarse denotes short descriptions without articulation detail, while action denotes only categorical labels. HandX provides fine-grained, multi-level language descriptions. Right: Statistics of bimanual motion quality. Metrics are defined in Sec. A.5. HandX provides contact-rich bimanual motions.
Ablation study on model size and data size. For R-precision, we adopt a batch size of 32. We observe clear scaling trends for our primary metrics, e.g., R-Precision improves consistently as we scale both data and model sizes, while Intra-hand CF1 shows a strong positive trend, culminating in the best performance with 12 layers of decoder and all training data.
Table 2: Ablation study on model size and data size. For R-precision, we adopt a batch size of 32. We observe clear scaling trends for our primary metrics, e.g., R-Precision improves consistently as we scale both data and model sizes, while Intra-hand CF1 shows a strong positive trend, culminating in the best performance with 12 layers of decoder and all training data.
Ablation study on the codebook size of FSQ and the model size of autoregressive models. For R-precision, we adopt a batch size of 32. Both the FSQ and autoregressive models are trained on the full training dataset. The primary metrics, e.g., FID, achieve the best performance when both model capacity and codebook size are scaled up. In contrast, scaling only one while keeping the other fixed can degrade performance. For example, R-precision is highest when the autoregressive model size is comparable to the codebook size.
Table 3: Ablation study on the codebook size of FSQ and the model size of autoregressive models. For R-precision, we adopt a batch size of 32. Both the FSQ and autoregressive models are trained on the full training dataset. The primary metrics, e.g., FID, achieve the best performance when both model capacity and codebook size are scaled up. In contrast, scaling only one while keeping the other fixed can degrade performance. For example, R-precision is highest when the autoregressive model size is comparable to the codebook size.
Qualitative results of our unified framework, showing (a) high-fidelity text-to-motion generation with fine-grained articulation and contact, and (b) bimanual motion synthesis given versatile spatiotemporal conditions. Gray hands denote the input condition, green hands denote the generation, and orange hands denote the extended long-horizon generation.
Figure 3: Qualitative results of our unified framework, showing (a) high-fidelity text-to-motion generation with fine-grained articulation and contact, and (b) bimanual motion synthesis given versatile spatiotemporal conditions. Gray hands denote the input condition, green hands denote the generation, and orange hands denote the extended long-horizon generation.
Scaling trend of computational scale. We observe a clear log-linear relationship between R-precision and FLOPS, with a high correlation coefficient of 0.96. R-Precision is evaluated with a batch size of 16.
Figure 4: Scaling trend of computational scale. We observe a clear log-linear relationship between R-precision and FLOPS, with a high correlation coefficient of 0.96. R-Precision is evaluated with a batch size of 16.
Qualitative comparison of diffusion models trained with different data scales. The model trained on the full dataset generates more expressive motion with better text alignment.
Figure 5: Qualitative comparison of diffusion models trained with different data scales. The model trained on the full dataset generates more expressive motion with better text alignment.
查看结构化数据
任务指标本文基线提升
文本到双手运动生成 Top-3 R-Precision 0.631 Ground Truth: 0.948 与4层5%数据基线0.296相比提升113%
双手接触准确性 Contact F1 (CF1) 0.624 Ground Truth: 0.984 与4层全数据基线0.517相比提升21%
生成真实性 FID (↓) 1.349 Ground Truth: 0.000 与8层20%数据基线3.053相比改善56%

局限与改进

作者承认的局限性包括：(1)采集数据集中于日常活动，可能缺乏极端手部动作的多样性；(2)方法仅限于手部，未考虑全身运动；(3)运动捕捉系统有固有约束，标记物可能限制细微动作表现。独立观察的局限性：(1)固定2秒clip长度可能限制更长交互序列的学习；(2)intensity-aware filter可能移除一些有意义的低强度但重要的交互；(3)contact metrics使用2cm阈值是启发式的，不同动作类型可能需要不同阈值；(4)评估主要依赖自动指标，缺乏人类主观评估的系统研究；(5)模型仅在文本条件上评估，未研究跨模态生成如audio-driven；(6)real-world transfer仅展示定性结果，缺乏定量评估机器人部署性能。

独立分析的弱点

独立分析的第一个弱点是数据集缺乏action diversity，虽然contact rich但主要是日常活动，缺乏工业操作、医疗手术、音乐演奏等专业领域的双手机动。改进方向是与专业领域合作采集特定数据。第二个弱点是clip length固定为2秒，限制了学习更长时序依赖的能力，可以引入hierarchical模型或memory机制支持可变长度生成。第三个弱点是contact metrics仅评估接触事件的发生时机，未评估接触的正确性(如正确的指尖对正确的表面)，可以引入surface-aware contact evaluation。第四个弱点是模型在ultra-large配置下性能下降，表明存在过拟合或优化困难，需要研究stabilization技术或alternative architectures。第五个弱点是缺乏失败案例分析，无法理解模型在哪些特定运动模式上失效，可以引入failure mode analysis guide改进。

未来方向

作者提出的未来工作包括扩展到更长序列的生成、探索更复杂的多对象交互场景、结合音频或其他模态条件、以及将学到的灵巧技能迁移到不同机器人平台。基于成果可延伸的方向：(1)开发实时推理系统，使双手运动生成能够应用于VR/AR交互；(2)研究conditional generation的更细粒度控制，如单独控制每个手指的轨迹；(3)探索半监督和自监督学习，利用未标注的双手机动视频扩展数据规模；(4)开发physics-aware生成模型，确保生成的运动满足物理约束；(5)研究cross-domain adaptation，将学到的motion prior迁移到不同骨骼拓扑或运动风格；(6)开发evaluation的human-in-the-loop系统，结合主观评估和自动指标；(7)探索generative editing能力，允许用户编辑生成运动的特定属性；(8)研究multi-person双手交互，如握手、传递物体等社交场景。

复现评估

复现性评估显示该项目开源友好。作者承诺发布数据集、代码和预训练模型，提供了详细的实验配置和超参数。数据采集使用36摄像头OptiTrack系统，计算需求在大型服务器集群上运行(NCSA Delta、DeltaAI、PTI Jetstream2、TACC Frontera)，使用AWS和OpenAI API通过NAIRR Pilot项目获得资源。训练配置在Table 2和3中详细列出，包括模型层、hidden size、FFN size、FLOPs等。评估指标定义在Sec. E supplementary material中，包含contact threshold(2cm)、intensity threshold(tau_hand=25, tau_avg=30)等细节。唯一限制是motion capture系统需要专业设备，但公开数据部分可以立即使用。总体而言，该研究复现性良好，为社区提供了solid baseline和数据基础。

论文图表

Figure 6: Qualitative comparison of diffusion models with different model scales. Larger models produce motion that is better aligned with the text and exhibits improved bimanual contact.

该图定性对比不同模型规模的扩散模型。展示两个例子，每个例子对比4层、8层和12层解码器的模型生成的结果。使用彩色框标注关键视觉差异，对应的文本线索用相同颜色的粗体标记。第一个例子中，12层模型正确实现了The thumb and middle finger stay extended, the ring and pinky fingers perform a slow, continuous bending motion，而4层模型未能保持拇指伸展。第二个例子中，12层模型的双手接触更准确。每个例子的文本描述包含手指姿态和运动速度的细节。

这张图对理解模型scale的影响至关重要，它直观展示了模型容量如何影响生成的保真度和交互质量，与Figure 4的定量分析相互印证。