GPT-6-Astra: Research-Level Physics Test Performance and Costs

Human Progress (citing arXiv) USA

Overview

GPT-6-Astra, harnessed in Codex, achieved 98% accuracy on a suite of 50 unpublished research-level theoretical physics problems. The agent performed these tasks autonomously at an average cost of under USD 3 per problem. This demonstrates the ability of state-of-the-art agents to execute concrete calculations required in theoretical physics research.

In Depth

Key Findings

GPT-6-Astra, when harnessed in Codex, achieved a near-perfect accuracy of 98% on a test consisting of 50 unpublished research-level theoretical physics problems. The agent operated autonomously without human hints, incurring an average cost of under USD 3 per problem.

Technical Details

The evaluation provided the AI agent with tools reflecting a realistic research setting, including the ability to write scripts for symbolic and numerical work and the capacity to search existing literature. Notably, the researchers observed that accuracy levels remained largely unchanged even when the agent was denied access to web searches and background research papers, suggesting a significant level of memorization of existing literature.

Background and Context

The test suite comprised problems with verifiable answers that would typically take a world expert approximately one day or less to solve, while requiring considerably more time from experts in adjacent specialties. During the assessment, some initial errors attributed to Astra were later found to be subtle errors in the original solutions. Upon consultation with experts and subsequent correction of those original solutions, Astra’s results were confirmed as correct.

Significance and Outlook

These results indicate that current state-of-the-art agents can accurately perform the concrete calculations necessary for theoretical physics research. The ability to solve complex, unpublished problems with high precision and low computational cost suggests a shift in how numerical and symbolic work in the physical sciences may be conducted.

Get our weekly technology intelligence — free

Receive an infographic that lets you judge at a glance whether each field’s analysis report is worth reading.

Subscribe Free — Weekly Tech Intelligence

By subscribing, you’ll receive Troy-Technical’s weekly technology intelligence newsletter.

• Your email and selected fields are used only to deliver the newsletter.
• We never share your information with third parties.
• You can unsubscribe anytime via the link in each email.

See our Privacy Policy for details.

Agree & Continue ▶

Takes about a minute · Unsubscribe anytime