Key Findings
Beam is a sparse MoE model with 501B total parameters, of which 23B are active per token. It is designed specifically for long-context reasoning and agentic workflows, supporting a context window of 1M tokens. The model is released under the Apache 2.0 license and is compatible with transformers, vLLM, and SGLang, while also shipping with a dedicated Rust-based harness for optimized performance.
Technical Details
The architecture consists of 52 layers with a hidden size of 5,120. It employs a hybrid attention mechanism: 43 layers use sliding-window attention (512 tokens) with RoPE, while 9 global layers use scalable-softmax query scaling without positional encoding. Attention is implemented via grouped-query attention with 48 query heads and 8 key-value heads.
The MoE structure includes 256 experts per layer (except the first three layers, which use dense SwiGLU MLP). Following a group-limited routing scheme similar to DeepSeek-V3, the model splits experts into 8 groups of 32, selecting the top 4 groups and then the top 8 experts within them. To maintain stability, Beam uses a unique residual connection where the stream is scaled by sqrt((k-1)/k) and sublayer outputs are added with a weight of 1/sqrt(k).
Checkpoint formats are available in BF16 (approximately 1 TB), FP8, and NVFP4 for NVIDIA Blackwell GPUs. The model supports five reasoning levels: “low”, “medium”, “high”, “xhigh”, and “max”.
Background & Context
Beam is the first model from the US-based lab Reflection. It is intended for teams requiring a frontier-scale model that can be customized via SFT (supervised fine tuning) and RL (reinforcement learning) on their own infrastructure. To encourage adoption, Reflection is offering $10,000 in Hub credits for top projects involving Beam, such as quantization, synthetic dataset generation, or benchmarking.
Significance & Outlook
By utilizing a sparse MoE architecture, Beam provides the capacity of a 501B model while operating at the speed of a 23B model. The integration of a 1M token window and specialized reasoning controls makes it a scalable option for complex, long-context agentic applications.
Source: https://huggingface.co/blog/reflection-beam
Get our weekly technology intelligence — free
Receive an infographic that lets you judge at a glance whether each field’s analysis report is worth reading.
Subscribe Free — Weekly Tech Intelligence
By subscribing, you’ll receive Troy-Technical’s weekly technology intelligence newsletter.
- Your email and selected fields are used only to deliver the newsletter.
- We never share your information with third parties.
- You can unsubscribe anytime via the link in each email.
See our Privacy Policy for details.
Takes about a minute · Unsubscribe anytime
