Muse Spark Open: Meta’s 816B multimodal MoE model specs

Hugging Face Blog USA

Overview

Meta has released Muse Spark Open, an open-weight multimodal mixture-of-experts (MoE) model with 816 billion parameters and a 1-million-token context window. Optimized for agentic coding and multimodal inputs, it utilizes 42 billion active parameters per token to balance high-capacity reasoning with inference speed.

In Depth

Key Findings

Muse Spark Open is a frontier multimodal model from Meta designed for agentic coding tasks, including reading codebases, debugging, and tool calling. It supports text, image, audio, and video inputs, allowing the model to process screenshots, voice notes, and screen recordings. The model is specifically trained on real developer workflows, such as long debugging sessions and multi-file refactors, enabling it to plan, act, observe, and self-correct.

Technical Details

The model architecture features a Mixture-of-Experts (MoE) design with 816 billion total parameters, though only 42 billion are active per token. Each MoE block contains 256 experts, with a router selecting 8 experts for each token. The language model consists of 62 layers with a hidden dimension of 8192, utilizing grouped-query attention (128 query heads and 16 key-value heads) and QK-norm.

• Context Window: 1,048,576 tokens, including multimodal tokens.
• Memory and Precision: The BF16 checkpoint is approximately 1.5 TB. An NVFP4 4-bit precision checkpoint is available for NVIDIA Blackwell GPUs.
• Hardware Requirements: BF16 runs across four nodes of 8 H200 GPUs (TP=32), while NVFP4 runs on a single B200 node.
• Multimodal Encoders: A 1.8B parameter perception encoder handles images and video (14×14 patches). Audio is processed via a convolutional pyramid encoder, emitting one token every 80 ms (12.5 tokens per second).

Background and Context

Muse Spark Open is the full-size version of the Muse series. It follows Muse Glimmer, a smaller dense model released in August. Both models share the same tokenizer and vocabulary, though Muse Spark Open expands the chat template to accommodate new modalities. The model’s input embedding and output layers are separate matrices of approximately 1.66B parameters each.

Significance and Outlook

By combining a massive parameter count with MoE efficiency, Muse Spark Open provides the capabilities of an 816B model with the execution speed of a 42B model. Its integration of audio, video, and image perception into a coding-optimized agent makes it the most capable US open-weights model to date.

Get our weekly technology intelligence — free

Receive an infographic that lets you judge at a glance whether each field’s analysis report is worth reading.

Subscribe Free — Weekly Tech Intelligence

By subscribing, you’ll receive Troy-Technical’s weekly technology intelligence newsletter.

• Your email and selected fields are used only to deliver the newsletter.
• We never share your information with third parties.
• You can unsubscribe anytime via the link in each email.

See our Privacy Policy for details.

Agree & Continue ▶

Takes about a minute · Unsubscribe anytime