Sakana AI has released a method that trains neural networks to 96.7% accuracy on MNIST and 61.7% on CIFAR-10 without using backpropagation. The approach relies on a local learning rule called Error Diffusion, which sidesteps the weight transport problem by updating weights based only on presynaptic activity, postsynaptic activation derivatives, and a global error sign.
The Dual-Stream Architecture
To satisfy Dale’s principle, the researchers split each layer into two separate streams. One stream handles excitatory signals, while the other handles inhibitory ones. The forward pass calculates preactivations by subtracting the inhibitory stream from the excitatory stream.
All four weight matrices within this structure must remain non-negative element-wise. Biases are the exception, as they do not require this constraint. The negation signs applied to cross-stream connections are structural rather than learned. This design necessitates four weight sub-matrices per layer, resulting in a parameter count roughly four times larger than a standard single-stream network. A standard dual-stream architecture uses approximately 32 million parameters, compared to 8 million for a simpler DFA configuration.
Modulo Error Routing
The core extension to Error Diffusion is modulo error routing. For a hidden unit at index i, the team defines the routing function as r(i) = i mod C, where C represents the output dimension. Each hidden unit then learns from the error component assigned to its specific output channel. Unlike Direct Feedback Alignment, which relies on random feedback matrices, this method uses a structured correspondence between units and channels.
Three Classification Innovations
Building on this routing mechanism, the research team implemented three specific fixes to handle multi-class classification effectively:
- Layer-specific sigmoid widths employ the function
φi(z) = 1/(1 + e−2z/αi). Because the sigmoid derivative directly gates the error signal, attenuation is severe. Post-hoc analysis shows a 25-fold decay from the output layer to the first hidden layer. Wider sigmoids maintain larger derivatives to prevent premature saturation. The team setα = 3.0for CIFAR-10 convolutional layers andα = 6.0for fully connected layers. - Batch-centered class error subtracts the per-class mini-batch mean. This ensures the one-vs-all error is zero-mean across the batch for every class, reducing persistent suppression caused by the inherent 9:1 target imbalance.
- Asymmetric initialization scales excitatory weights by 1.5 times and inhibitory weights by 0.5 times. This creates an expected excitatory-to-inhibitory scale ratio of 3:1, while the output layer remains symmetric.
Performance
Applying all three innovations allows Error Diffusion to reach 96.7% on MNIST and 61.7% on CIFAR-10. Without these fixes, the baseline seed ED collapses to 50.4% and 11.6% respectively. Direct Feedback Alignment scores higher on both tasks but violates Dale’s principle by using approximately 2.84 million negative weights. Notably, this marks the first time Error Diffusion has successfully trained convolutional networks. Previously, Fujita (2026) achieved roughly 55.2% on CIFAR-10 using a flattened MLP. Even so, 61.7% remains far from the results of standard gradient-based methods.
Method | MNIST | CIFAR-10 | Dale-compliant | Notes
Proposed ED | 96.7% | 61.7% | Yes | All weights non-negative; first ED on CNNs
Seed ED | 50.4% | 11.6% | Yes | No innovations; α = 1.0, raw error, symmetric init
DFA | 97.6% | 69.1% | No | Random feedback; ~2.84M negative weights
The Ablation Reversal
The importance of these innovations flips depending on the task. On MNIST, removing layer-specific widths is catastrophic, dropping accuracy by 71.4 percentage points and collapsing performance toward chance. Batch-centering has a negligible impact here, with a drop of only 0.3 percentage points. On CIFAR-10, the order reverses. Removing batch-centered error becomes the largest drop of 47.9 percentage points, causing four of five seeds to fail. This reversal highlights task-dependent credit-assignment bottlenecks that single-benchmark evaluations often miss.
Error Diffusion in Reinforcement Learning
Beyond classification, the team integrated Error Diffusion with Proximal Policy Optimization (PPO). They named the result ED-PPO and tested it on Brax locomotion and Craftax. Here, policy-output error is routed to hidden units by output channel. For the scalar value network, the error is broadcast to all units. Importantly, ED-PPO drops the three classification innovations entirely. Across five seeds, ED-PPO beats BP-PPO on HalfCheetah (5494 vs 3520; p < 0.001) and matches DFA-PPO. On Ant, it stays on par with both PPO variants. On Craftax, DFA-PPO is the weakest method (19.8 vs BP-PPO 27.0). Thus random feedback that suffices for classification can fail on open-ended RL.
Use Cases and Examples
Three settings make this concrete:
- Neuromorphic and photonic hardware often encode non-negative synaptic magnitudes physically. ED’s fixed-sign routing maps cleanly onto such substrates, complementing prior photonic DFA work.
- The non-negative floor drives 37.3% of weights to the floor (10⁻⁴) after training. Inhibitory cross-stream fully connected connections are pruned most, up to 68.8%. This implicit sparsity hints at model compression “for free.”
- The dedicated inhibitory stream may help continual and open-ended learning. It provides a structural mechanism for dampening large gradient excursions.
Comparison
How Dale-Compliant Error Diffusion Compares
Proposed approach vs. other backpropagation-free and biologically motivated learning rules. “Dale-compliant” means separate excitatory/inhibitory populations with non-negative weights. Method names link to primary sources.
| Method | Backprop-free (no weight transport) | How error reaches hidden layers | Dale-compliant (E/I, non-negative) | Shown on RL | Demonstrated reach / notes |
|---|---|---|---|---|---|
| Error Diffusion — ED / ED-PPO (proposed) | Yes | Global error sign routed directly to hidden units via modulo routing r(i) = i mod C | Yes — dual-stream E/I, non-negative weights | Yes (Brax, Craftax) | 96.7% MNIST, 61.7% CIFAR-10; RL returns on par with DFA-PPO |
| Backpropagation | No — needs transposed forward weights | Exact gradient, layer by layer | No — arbitrary-sign weights | Yes (BP-PPO) | Reference baseline; state of the art across tasks |
| Feedback Alignment (FA) | Yes | Fixed random backward weights, layer by layer | No — arbitrary-sign feedback | Not shown | Learns deep and convolutional nets; limited on harder benchmarks |
| Direct Feedback Alignment (DFA) | Yes | Output error to each hidden layer via fixed random matrices | No — random signed feedback (~2.84M negative weights) | Yes (DFA-PPO) | Scales to convnets and transformers; 97.6% MNIST, 69.1% CIFAR-10; weakest on Craftax |
| Dale’s ANNs (DANNs) | No — trained with backprop | Backpropagation | Yes — separate E/I populations | Not shown | Matches standard ANNs on supervised tasks |
| Predictive coding | Yes | Local prediction-error units (Hebbian) | No — not enforced |
