Deep Learning Reproducibility: Riddled Basin Geometry Explained

Nature Machine Intelligence

Overview

Researchers propose riddled basin geometry as the unifying mechanism behind the intrinsic unpredictability of deep learning training outcomes. The study finds that training destinations form a “fat fractal” where increasing initialization precision yields only marginal gains in predictability. This suggests a fundamental limit to reproducibility that persists even when all extrinsic randomness is controlled.

In Depth

Key Findings

The study identifies riddled basin geometry as a candidate organizing principle for irreproducibility in deep learning. The core finding is that basins of attraction in parameter space are intertwined such that any initialization leading to one solution lies arbitrarily close to initializations leading to different solutions. This “fat fractal” structure results in near-zero uncertainty exponents, meaning that even sharp increases in the precision of initial conditions do not materially improve the predictability of the training outcome.

Technical Details

To quantify this phenomenon, the researchers utilized the following methods:

• Minimal Neural Network Model: Used to analytically establish the route to riddling by linking symmetries and convergence toward symmetry-induced invariant subspaces.
• Uncertainty Exponents: Adapted from dynamical systems theory to measure fine-scale fractal structure in parameter space.
• Functional Uncertainty Exponents: Introduced to extend diagnostics from parameter space to the learned functions themselves, using standard measures of functional similarity.

These signatures of riddling were found to be robust across various tested architectures, optimizers, regularization methods, and activation functions.

Background & Context

While unpredictability is often attributed to extrinsic stochasticity—such as random initialization, mini-batch ordering, data augmentation, and GPU numerical errors—empirical evidence shows that changing a single bit in a parameter can produce variability comparable to multiple noise sources combined. This differs from “skinny” fractals of zero Lebesgue measure, where predictability can be recovered by decreasing uncertainty. Riddling represents a qualitative, outcome-level unpredictability that persists under arbitrarily small uncertainty, distinguishing it from trajectory-level chaos.

Significance & Outlook

The discovery of riddled basin geometry provides a common dynamical account for disparate empirical observations regarding AI irreproducibility. By establishing that certain intrinsic limits constrain the predictability of training, these results have practical implications for the optimization and reliable deployment of artificial intelligence systems.

Get our weekly technology intelligence — free

Receive an infographic that lets you judge at a glance whether each field’s analysis report is worth reading.

Subscribe Free — Weekly Tech Intelligence

By subscribing, you’ll receive Troy-Technical’s weekly technology intelligence newsletter.

• Your email and selected fields are used only to deliver the newsletter.
• We never share your information with third parties.
• You can unsubscribe anytime via the link in each email.

See our Privacy Policy for details.

Agree & Continue ▶

Takes about a minute · Unsubscribe anytime