NVIDIA Releases Cosmos 3 Edge: A 4B-Parameter Open World Model That Reasons and Generates Robot Actions On-Device 英伟达发布Cosmos 3 Edge：一款能在端侧推理并生成机器人动作的40亿参数开放世界模型

TL;DR

• NVIDIA released Cosmos 3 Edge, a 4-billion-parameter open world model designed for on-device execution, featuring a 2B dense reasoner and a diffusion generator.
• The architecture utilizes a Mixture-of-Transformers design with shared multimodal attention, enabling simultaneous reasoning about scenes and predicting visual/action outcomes.
• It supports a unified action representation across diverse embodiments, including cameras, vehicles, and various robot arms, facilitating cross-platform policy development.
• The model achieves real-time control at 15 Hz on NVIDIA Jetson Thor hardware, targeting memory-constrained edge environments in industrial and robotic applications.
• Cosmos 3 Edge is available under the Linux Foundation OpenMDW-1.1 license, allowing developers to post-train models for specific tasks within approximately one day.

Why It Matters

This release bridges the critical gap between high-performance AI reasoning and the latency/memory constraints of edge devices, enabling robots and autonomous systems to process complex multimodal data locally without relying on cloud infrastructure. By providing an open-source world model that integrates perception, reasoning, and action generation, NVIDIA lowers the barrier for developing embodied AI agents capable of real-time decision-making in dynamic physical environments.

Technical Details

• Architecture: Employs a Mixture-of-Transformers structure with two distinct towers: an autoregressive tower for vision/text understanding and reasoning, and a diffusion tower for generating video, audio, and action tokens. These towers share multimodal attention layers but maintain separate normalization and MLP components.
• Action Representation: Maps disparate physical controls into a common geometric vector space capturing translation, rotation, and manipulation states. Supported dimensions include 9D for cameras/vehicles, 10D for single-arm robots, up to 57D for egocentric motion.
• Performance Specs: Operates at 640x360 resolution for control tasks. On NVIDIA Jetson Thor, it generates 32 actions per inference with a 15 Hz real-time control loop. Generation capabilities support 256p and 480p resolutions at 12-30 fps.
• Deployment & Training: Designed for NVIDIA edge hardware (Jetson, RTX GPUs). Includes post-trained variants like Cosmos 3 Edge Policy (DROID) for pick-and-place tasks. Developers can fine-tune the model on local clusters (e.g., H100, DGX Station) or even consumer-grade GPUs (RTX 3070+) for prototyping.

Industry Insight

The standardization of action representations across different robotic embodiments suggests a future where universal policy models can be adapted to various hardware platforms with minimal retraining, significantly accelerating robotics development cycles. Furthermore, the emphasis on on-device inference highlights a strategic shift toward privacy-preserving, low-latency AI solutions, making edge-deployable world models essential for next-generation autonomous systems in manufacturing and logistics.

TL;DR

• NVIDIA发布Cosmos 3 Edge，一款40亿参数的开源世界模型，专为在内存受限的边缘设备上运行而设计。
• 采用混合Transformer架构，结合自回归推理塔和扩散生成塔，通过共享多模态注意力层实现理解与生成的协同。
• 支持多种具身智能形态（如机器人臂、自动驾驶车辆），将不同物理系统的动作映射为统一的几何向量表示。
• 在NVIDIA Jetson Thor上实现15Hz实时控制，支持从原型开发到部署的快速微调，显著降低边缘AI部署门槛。
• 在VANTAGE-Bench基准测试中声称达到4B参数级别的第一名，并遵循Linux Foundation OpenMDW-1.1标准。

为什么值得看

这篇文章揭示了边缘AI向具身智能演进的关键技术路径，展示了如何在资源受限的硬件上实现接近数据中心的性能。对于从事机器人、自动驾驶及边缘计算开发的从业者而言，Cosmos 3 Edge提供了一套完整的、可本地部署的世界模型解决方案，极大地简化了从感知到决策的闭环开发流程。

技术解析

• 双塔混合架构：模型由两个塔组成，自回归塔处理视觉和文本令牌用于理解和推理，扩散塔处理视觉、音频和行动令牌用于预测和生成。两者共享多模态注意力层以对齐信息，但拥有独立的归一化和MLP层，分别采用因果注意力和广泛上下文注意力。
• 统一动作表示：Cosmos 3 Edge将不同具身系统（如相机运动、自动驾驶、单/双机械臂、人形机器人）的动作映射为紧凑的几何向量（平移、旋转、操纵状态），实现了控制与视觉结构的直接关联，支持动作的双向预测（预测效果或推断动作）。
• 边缘部署性能：作为后训练的世界动作模型（WAM），它在640x360分辨率下运行，在Jetson Thor上每推理生成32个动作，达到15Hz的实时控制频率。支持256p和480p的视频生成，帧率为12-30fps，长度50-150帧。
• 开发与微调生态：开发者可使用GeForce RTX 3070及以上显卡进行本地原型开发，并在约一天内针对特定具身形态和传感器集进行后训练。提供了Cosmos 3 Edge Policy (DROID)版本，专门针对抓取放置任务进行了后训练。

行业启示

• 边缘具身智能的标准化：随着世界模型向边缘下沉，统一的动作表示标准和开源框架（如OpenMDW-1.1）将成为连接不同硬件平台和算法的关键基础设施，加速具身智能的规模化应用。
• 软硬协同优化的必要性：NVIDIA强调其在自家边缘芯片（Jetson系列）上的性能表现，这表明未来高性能AI模型的落地将越来越依赖于特定的硬件加速和软件栈优化，开发者需关注软硬件匹配度。
• 快速迭代与低代码部署：能够在消费级GPU上进行快速微调并在一周内完成部署的特性，降低了具身智能应用的开发门槛，使得中小团队也能参与到机器人和自动化解决方案的创新中。