Sakana AI researchers have released a training method called PC-ALM that trains residual multilayer perceptrons with up to 1000 layers to within roughly 2 percentage points of standard backpropagation on the MNIST dataset.
The technique, described in a preprint on arXiv, offers a local learning alternative to backpropagation. It operates without the global backward pass required by standard methods. The team provides an MIT-licensed JAX reference implementation that runs on CPU and reproduces the paper’s experimental grid.
Why standard predictive coding fails in depth
Standard predictive coding treats every hidden activation as an optimization variable. It penalizes the mismatch between a layer’s actual output and the prediction arriving from the layer below. Learning occurs via a Hebbian-like weight step. The problem is that supervision enters only at the output and must diffuse through a chain of local compromises.
In deep, narrow networks, this credit signal fades long before it reaches the input. Previous work by Innocenti et al. characterised this gap as a function of width and depth. The issue is worst when the network width is smaller than its depth.
How PC-ALM handles the constraints
The new method treats training as a constrained optimisation problem. It minimises the supervised loss while keeping the network’s internal activation rules intact. The constraint requires that each layer’s output equals the activation function applied to its input and the previous layer’s output.
Standard predictive coding relaxes this constraint by adding a quadratic penalty. PC-ALM uses an augmented Lagrangian approach instead. This attaches a Lagrange multiplier to each layer’s constraint while keeping the penalty term. Setting the multiplier to zero recovers the standard predictive coding method exactly.
Inference alternates between two local steps. The first is a primal gradient step on the activations. The second updates the multiplier by adding a term proportional to the layer’s prediction error. This accumulation of error shifts the prediction target used in the primal step.
After multiple steps, the weight update acts on a composite signal combining the multiplier and the prediction error. The researchers interpret this structure as a proportional-integral controller per layer. The prediction error serves as the proportional term, while the multiplier acts as the integral term.
Convergence to backpropagation gradients
Yann LeCun observed in 1988 that the Lagrange multipliers of a constrained network equal the backpropagation adjoints at a specific KKT point. The team proves that in linear PC networks, under a spectral-radius stability condition, PC-ALM converges to that point.
During convergence, activations return to their forward-pass values. Each multiplier integrates to the exact backpropagation adjoint. The stability bound for this process is stricter than that of standard predictive coding. Unlike the monotone gradient flow of standard PC, PC-ALM’s iteration matrix has complex eigenvalues that produce damped oscillations.
The parameter alpha sets the frequency of these oscillations but does not control their decay rate.
Performance results
The team tested residual MLPs with widths and depths ranging from 8 to 128 on Fashion-MNIST and MNIST. They trained for one epoch under the mean-field parameterization used by Innocenti et al. The inference budget was set to two times the number of layers.
PC-ALM matched backpropagation across every width, depth, and activation function tested. Standard predictive coding dropped sharply in deep, narrow cells. A reference cell with a width of 32, depth of 32, and ReLU activation reported a test accuracy of 78.66% for backpropagation, 68.13% for predictive coding, and 77.75% for PC-ALM. The cosine similarity between the gradients of PC-ALM and backpropagation rose from 0.604 to 0.909.
The method also trained 1000-layer residual MLPs on MNIST with a width of 32 using ReLU activations over five epochs. These models stayed within roughly 2 percentage points of backpropagation. PC-ALM improved upon predictive coding on every benchmark tried, including ResNet-18 on CIFAR-10 and Tiny ImageNet.
What it means
For developers building large models, this work suggests a viable path away from global backpropagation without sacrificing performance. The method is currently a research tool rather than a production standard. It has only been tested on small image benchmarks. The code is available for those wishing to experiment with layer-local training on their own hardware.
Paper | Blog | GitHub Repo
