Key Findings
IBM has established the Spyre dataflow accelerator as a first-class PyTorch backend following software updates in the second half of 2026. This integration, developed with the PyTorch Foundation, Meta, Hugging Face, and Red Hat, enables native-device integration and hardware-agnostic model support in vLLM. In demonstrations using Granite 3.3 8B (batch size 1, sequence length 1024), removing graph construction and compute dispatch from the path resulted in prefill speeds 1.7x faster and decode speeds 2.4x faster.
Technical Details
Spyre is a 75-watt, PCIe-attached accelerator featuring 32 small cores connected via a ring. Each core possesses a 2 MB scratchpad and the device supports up to 128 GB of LPDDR5. Unlike GPUs that execute instruction streams, Spyre triggers computation as data arrives in 128-byte “sticks.”
The integration centers on torch-spyre, which transitioned its torch.compile backend from a custom stack called sendnn to backend=inductor. This shift reduced the codebase from approximately 273,000 lines of C++ and Python to roughly 38,600 lines of Python. Performance gains include:
- Matmul kernel time reduced by 29–69%.
- Softmax median kernel time dropped from 32.51 microseconds to 23.67 microseconds.
- Core utilization increased from roughly two-thirds to about 80% via a fused MLP-style kernel, cutting kernel time by around 17%.
- On-chip relayout reduced a device-memory round trip from 80 microseconds to a 1.1-microsecond shuffle between cores.
Background & Context
IBM outlined this plan in October 2025, adopting vLLM as its inference runtime and torch.compile as the frontend compiler. Spyre is registered as a native PyTorch device via PrivateUse1, allowing tensor.to(“spyre”) to function with native allocator storage and stream ordering. To improve portability, IBM, Meta, and Hugging Face introduced hardware-agnostic layers in vLLM, which can be enabled using USE_HW_AGNOSTIC=1.
Significance & Outlook
The integration allows Spyre to maintain its dataflow execution model while providing users with familiar PyTorch tensor placement and compilation. Additionally, the development of torch-spyre contributed to improvements in out-of-tree Inductor backend support merged into PyTorch 2.12.
Source: https://www.unite.ai/ibm-makes-spyre-accelerator-a-first-class-pytorch-backend/
Get our weekly technology intelligence — free
Receive an infographic that lets you judge at a glance whether each field’s analysis report is worth reading.
Subscribe Free — Weekly Tech Intelligence
By subscribing, you’ll receive Troy-Technical’s weekly technology intelligence newsletter.
- Your email and selected fields are used only to deliver the newsletter.
- We never share your information with third parties.
- You can unsubscribe anytime via the link in each email.
See our Privacy Policy for details.
Takes about a minute · Unsubscribe anytime
