LLM Test-Time Scaling: Candidate Generation Strategy Dramatically Increases Energy Consumption by 4.86x and Latency by 6.12x on A100 GPUs for Phi-3-mini and Qwen2.5-1.5B

September 20, 2026

Mobina Kashaniyan (referencing IEEE/ACM SC26 Workshop) Unknown

Overview

New research reveals that for LLM test-time scaling, the candidate generation strategy, not just the candidate count, critically impacts energy and performance. Evaluating Phi-3-mini and Qwen2.5-1.5B on A100 GPUs, 8 sequential calls consumed 4.64–4.86 times more energy and exhibited 5.77–6.12 times higher P95 latency compared to a single batch call for 8 candidates. This highlights the urgent need to optimize inference strategies for sustainable and efficient LLM deployment.

In Depth

Key Findings

A seminal study presented at the IEEE/ACM SC26 Workshop demonstrates that the energy consumption and performance of Large Language Model (LLM) test-time scaling are dramatically shaped by the candidate generation strategy, not merely the number of candidates. The evaluation of Phi-3-mini and Qwen2.5-1.5B on NVIDIA A100 GPUs revealed that performing eight sequential API calls resulted in a 4.64 to 4.86 times increase in GPU energy consumption and a 5.77 to 6.12 times higher P95 latency when compared to processing eight candidates in a single batch call.

Technical Details

The research posits that simply considering the candidate count ‘N’ is insufficient to characterize the cost of multi-candidate inference. Instead, the specific strategy—sequential versus batched calls for multiple candidates—exerts a profound influence on latency, throughput, GPU time, utilization, and GPU device energy. The models chosen for evaluation, Phi-3-mini and Qwen2.5-1.5B, represent typical LLM inference workloads. Experiments were conducted on state-of-the-art NVIDIA A100 GPUs, providing realistic performance and power metrics for datacenter environments.

Background & Context

As LLM adoption accelerates, inference costs, particularly energy consumption and latency, have become critical challenges for service providers and developers. While previous optimization efforts often focused on model quantization or architectural improvements, this study uniquely highlights the significant impact of inference-time API call patterns and candidate generation strategies on operational efficiency. In many LLM applications, complex decision-making or exploration requires generating multiple candidates, where the efficiency of this process directly dictates overall system performance.

Strategic Significance & Outlook

The findings from this research offer a crucial new vector for optimizing LLM service deployment. Developers and cloud providers can now achieve substantial cost reductions and performance gains by meticulously designing and selecting candidate generation strategies tailored to application requirements. Future work is expected to include validation across a wider range of LLM models and GPU architectures, as well as the development of dynamic mechanisms to switch between strategies based on real-time load. This insight is poised to directly influence the design and operation of large-scale AI infrastructure, fostering a more sustainable and high-performing LLM ecosystem.

Source: https://mobinakashaniyan.github.io/publication/sample-count-is-not-enough-candidate-generation-strategy-shapes-the-energy-and-performance-of-llm-test-time-scaling/

Get our weekly technology intelligence — free

Receive an infographic that lets you judge at a glance whether each field’s analysis report is worth reading.

Subscribe Free — Weekly Tech Intelligence

By subscribing, you’ll receive Troy-Technical’s weekly technology intelligence newsletter.