Smartphones, MRI scanners, and self-driving cars all process raw sensor data that humans cannot interpret directly. The quality of these systems depends on how much useful information the measurements contain, not how they look.
Engineers rarely evaluate this information content directly. Traditional metrics like resolution or signal-to-noise ratio assess individual aspects separately. Training neural networks to reconstruct images conflates the quality of the hardware with the quality of the algorithm.
Researchers at the Berkeley Artificial Intelligence Research (BAIR) Institute have developed a framework to evaluate and optimise imaging systems based on their information content. Their work, published in a NeurIPS 2025 paper, shows that this metric predicts system performance across four imaging domains. Optimising for it produces designs that match state-of-the-art end-to-end methods while requiring less memory, less compute, and no task-specific decoder.
Why mutual information?
Mutual information quantifies how much a measurement reduces uncertainty about the object that produced it. Two systems with the same mutual information are equivalent in their ability to distinguish objects, even if their measurements look completely different.
This single number captures the combined effect of resolution, noise, sampling, and all other factors that affect measurement quality. A blurry, noisy image that preserves the features needed to distinguish objects can contain more information than a sharp, clean image that loses those features.
Previous attempts to apply information theory to imaging faced two problems. The first approach treated imaging systems as unconstrained communication channels, ignoring the physical limitations of lenses and sensors. This produced wildly inaccurate estimates. The second approach required explicit models of the objects being imaged, limiting generality.
The new method avoids both problems by estimating information directly from measurements.
Estimating information from measurements
Estimating mutual information between high-dimensional variables is notoriously difficult. Sample requirements grow exponentially with dimensionality, and estimates suffer from high bias and variance.
However, imaging systems have properties that enable decomposing this hard problem into simpler subproblems. Mutual information can be written as:
\[I(X; Y) = H(Y) – H(Y \mid X)\]
The first term, $H(Y)$, measures total variation in measurements from both object differences and noise. The second term, $H(Y \mid X)$, measures variation from noise alone.
Imaging systems have well-characterized noise. Photon shot noise follows a Poisson distribution. Electronic readout noise is Gaussian. This known noise physics means we can compute $H(Y \mid X)$ directly, leaving only $H(Y)$ to be learned from data.
For $H(Y)$, the team fits a probabilistic model to a dataset of measurements. The model learns the distribution of all possible measurements. They tested three models spanning efficiency-accuracy tradeoffs: a stationary Gaussian process (fastest), a full Gaussian (intermediate), and an autoregressive PixelCNN (most accurate). The approach provides an upper bound on true information; any modeling error can only overestimate, never underestimate.
Validation across four imaging domains
Information estimates should predict decoder performance if they capture what limits real systems. The team tested this relationship across four imaging applications.
Color photography. Digital cameras encode color using filter arrays that restrict each pixel to detect only certain wavelengths. The researchers compared three filter designs: the traditional Bayer pattern, a random arrangement, and a learned arrangement. Information estimates correctly ranked which designs would produce better color reconstructions, matching the rankings from neural network demosaicing without requiring any reconstruction algorithm.
Radio astronomy. Telescope arrays achieve high angular resolution by combining signals from sites across the globe. Selecting optimal telescope locations is computationally intractable because each site’s value depends on all others. Information estimates predicted reconstruction quality across telescope configurations, enabling site selection without expensive image reconstruction.
Lensless imaging. Lensless cameras replace traditional optics with light-modulating masks. Their measurements bear no visual resemblance to scenes. Information estimates predicted reconstruction accuracy across a lens, microlens array, and diffuser design at various noise levels.
Microscopy. LED array microscopes use programmable illumination to generate different contrast modes. Information estimates correlated with neural network accuracy at predicting protein expression from cell images, enabling evaluation without expensive protein labeling experiments.
In all cases, higher information meant better downstream performance.
Designing systems with IDEAL
Information estimates can do more than evaluate existing systems. The Information-Driven Encoder Analysis Learning (IDEAL) method uses gradient ascent on information estimates to optimise imaging system parameters.
The standard approach to computational imaging design, end-to-end optimisation, jointly trains the imaging hardware and a neural network decoder. This requires backpropagating through the entire decoder, creating memory constraints and potential optimisation difficulties.
IDEAL avoids these problems by optimising the encoder alone. The team tested it on color filter design. Starting from a random filter arrangement, IDEAL progressively improved the design. The final result matched end-to-end optimisation in both information content and reconstruction quality.
What it means
Information-based evaluation creates new possibilities for rigorous assessment of imaging systems in real-world conditions. Current approaches require either subjective visual assessment, ground truth data that is unavailable in deployment, or isolated metrics that miss overall capability. This method provides an objective, unified metric from measurements alone.
The computational efficiency of IDEAL suggests possibilities for designing imaging systems that were previously intractable. By avoiding decoder backpropagation, the approach reduces memory requirements and training complexity. The framework may extend beyond imaging to other sensing domains. Any system that can be modeled as deterministic encoding with known noise characteristics could benefit from information-based evaluation and design, including electronic, biological, and chemical sensors.
This post is based on the NeurIPS 2025 paper “Information-driven design of imaging systems”. Code is available on GitHub. A video summary is available on the project website.
