Key Findings
The latest version of MLX allows a cluster of four Mac Studios to generate over 36 tokens per second using Muse Spark Open, a mixture of experts (MoE) model with 816B parameters. On a single Mac Studio with 512GB of unified memory, the M5 Ultra demonstrates significant performance gains over the M3 Ultra, specifically achieving up to 4.6× faster time to first token (TTFT) due to the GPU Neural Accelerator’s specialized matrix multiplication support.
Technical Details
Performance varies by model architecture and precision:
- Muse Spark Open (816B): Evaluated in NVFP4 precision. Due to its size, it requires parallelization across multiple machines as it exceeds the 512GB capacity of a single Mac Studio.
- Mistral Medium 3.5 (128B): A dense architecture evaluated in FP8 precision; fits on a single Mac Studio, utilizing 158GB of RAM at a 65,536 token context size.
- Qwen 3.8 (27B): A dense architecture evaluated in BF16 precision; utilizes 68GB of RAM at a 65,536 token context size.
The M5 Ultra provides 1.2 TB/s of unified memory bandwidth, compared to 819 GB/s on the M3 Ultra, resulting in 1.6× faster generation speeds. Power consumption remains low, with peak usage at 300W for the M5 Ultra and 200W for the M3 Ultra.
Background & Context
To handle models that exceed single-machine memory, MLX implements pipeline parallelization. This is supported by a low-latency RDMA driver over Thunderbolt 5 introduced in macOS 26.2. MLX (v0.30.1 and later) utilizes this driver via new communication collectives called “jaccl” to facilitate fast inter-machine communication.
Significance & Outlook
The integration of the GPU Neural Accelerator and RDMA over Thunderbolt allows for the local deployment of trillion-parameter models in private environments. This setup is particularly relevant for agentic coding applications that require large context windows (up to 65,536 tokens) while maintaining energy efficiency suitable for home or office settings.
Source: https://machinelearning.apple.com/research/mlx-distributed-inference
Get our weekly technology intelligence — free
Receive an infographic that lets you judge at a glance whether each field’s analysis report is worth reading.
Subscribe Free — Weekly Tech Intelligence
By subscribing, you’ll receive Troy-Technical’s weekly technology intelligence newsletter.
- Your email and selected fields are used only to deliver the newsletter.
- We never share your information with third parties.
- You can unsubscribe anytime via the link in each email.
See our Privacy Policy for details.
Takes about a minute · Unsubscribe anytime
