Key Findings
Groq, the AI chip company that licensed its inference technology to NVIDIA, is exploring the adoption of 3D-stacked DRAM (3DS) for its next-generation chips. This potential move away from a primary reliance on SRAM is driven by the need for greater memory capacity as AI workloads expand beyond the limits of traditional SRAM architectures.
Technical Details
The proposed 3DS memory utilizes through-silicon vias (TSVs) to connect DRAM dies vertically. While both 3DS and high-bandwidth memory (HBM) utilize TSVs, 3DS specifically prioritizes increasing memory capacity while retaining conventional DRAM interfaces, whereas HBM focuses on data transfer speeds.
Current specifications for NVIDIA’s Groq 3 LPX, designed for the Vera Rubin platform, include:
- Per LPU: 500MB of SRAM with 150TB/s on-chip memory bandwidth.
- Per LPX rack (256 LPUs): 128GB of SRAM and 40PB/s aggregate SRAM bandwidth.
- Additional capacity: Each rack provides 12TB of DDR5 memory alongside the SRAM.
Background & Context
The shift is necessitated by a disparity in scaling: SRAM capacity increases by approximately 10–20% every two years, while AI model sizes can double every six to nine months. This makes it increasingly difficult for SRAM alone to meet demand. Other industry players are pursuing similar paths; Cerebras is developing the CS-6 system, which will integrate stacked DRAM with wafer-scale SRAM and compute.
Significance & Outlook
Manufacturing competition for these next-generation components is intensifying. Samsung Foundry currently produces the Groq 3 LPU using a 4nm process at its Pyeongtaek facility, and Samsung executives have discussed further foundry cooperation with NVIDIA. Simultaneously, TSMC has stated it is working with customers on the development of next-generation LPUs.
Get our weekly technology intelligence — free
Receive an infographic that lets you judge at a glance whether each field’s analysis report is worth reading.
Subscribe Free — Weekly Tech Intelligence
By subscribing, you’ll receive Troy-Technical’s weekly technology intelligence newsletter.
- Your email and selected fields are used only to deliver the newsletter.
- We never share your information with third parties.
- You can unsubscribe anytime via the link in each email.
See our Privacy Policy for details.
Takes about a minute · Unsubscribe anytime
