Four Routes Out of the Backward Pass and Only One of Them Is New

The weight transport problem has been the standing objection to backpropagation as a model of cortical learning for close to forty years. To send an error signal backward, each layer needs a reverse path through the transpose of its forward weights. Biology has no obvious way to build that path, and neither does an analog or neuromorphic substrate, where a transposed-weight readout is an expensive and often impossible operation.

The standard response has been substitution. Feedback alignment replaces the transpose with a fixed random matrix. Forward-Forward replaces the global objective with a local goodness function. Target propagation replaces the gradient with a layerwise inverse. Each substitution buys plausibility and pays for it in accuracy, stability, or both.

Five papers over the past ten days approach the problem from different directions, and one of them declines the trade entirely.

Reconstruction, not substitution

Shuhei Ikemoto’s paper on noise-modulated neural networks makes the strongest claim of the group. An NNN learns and infers only in the presence of noise, treating noise as a computational resource rather than a disturbance — the noise is what allows the network to be trained by gradient descent while transmitting spike-like signals. But the gradient still requires the reverse path, so the plausibility gained on the signalling side is lost again on the learning side.

The result is that the reverse path is not needed, because the information it carries is already present in the forward statistics. A weight mirror estimates each weight matrix from the covariance between a previous-layer unit’s output and the next-layer unit’s input. Combined with local differential estimation inside the units, this propagates the output error recursively along the computational graph with no transposed-weight readout and no backward data path at all.

The distinction from prior forward-only work is precise and worth stating plainly. Feedback alignment substitutes a different matrix for the transpose and relies on the forward weights aligning to it over training. This reconstructs the transpose. The resulting gradient is reported as empirically near-unbiased, and with local per-weight Adam updates it matches the final accuracy of backpropagation on simple regression tasks.

Two caveats belong here and the paper states both. The tasks are simple regression problems, and near-unbiased is an empirical characterisation rather than a proof. What survives those caveats is the mechanism: under uniformly distributed noise the local operations reduce to polynomials and comparators, which makes the entire system — learning rule included — implementable in digital circuits without a training-specific data path.

Predictive coding without the error neurons

Andrew Smith, Linxing Preston Jiang, Jason Eshraghian, Matthew Bull and Stefano Recanatesi come at the same territory from the neuroscience side. Hierarchical predictive coding has been the dominant computational story about cortex for two decades, and its persistent weakness has been implementation: most models require dedicated error-coding neurons or a generative model whose biological realisation is unclear.

Their construction is a recurrent variant of the Forward-Forward algorithm with an inverted objective that increases activity for negative data. The framework rests on local contrastive learning and simple activity cancellation — no error units, no explicit generative model. What comes out are predictive representations across layers that reproduce hallmark features of cortical computation, specifically top-down modulation and surprise signalling.

This is an existence argument rather than a benchmark result, and it is the more valuable for it. It says the functional goals of predictive coding do not require the machinery predictive-coding models have been importing to achieve them. The machinery was an implementation choice that the field had begun mistaking for the theory.

The formal version

Moumita Das, Dipanjan Ray and Sourabh Bhattacharya take the opposite approach: rather than removing structure, they ask what exact structure predictive coding corresponds to. Their answer connects it to Recursive Gaussian Processes, which use a single Gaussian process indexed by both layer and input value, avoiding the representational collapse that afflicts standard deep Gaussian processes while permitting learnable cross-layer dependence.

RGPs intrinsically implement hierarchical Bayesian inference, uncertainty propagation and precision-weighted prediction error — the three components predictive-coding models normally build by hand. The mapping onto the canonical cortical microcircuit is explicit, with the shared GP, spike-and-slab variable selection and MCMC dynamics assigned to circuit components, and RGP inference is shown to minimise variational free energy.

The paper’s value to a practitioner is not the architecture but the predictions: laminar-specific dynamics and spectral asymmetries between feedforward and feedback processing. Those are measurable, which means the formalism can be wrong in a way that a free-energy argument alone cannot.

Pruning by the same principle

Sanjith Senthil and Rishidev Chaudhuri apply the noise-statistics idea to structure rather than gradient. Noise-prune is an unsupervised local rule that uses noisy fluctuations to rank connection importance; it had previously only been tested on random networks with no computational function, which is a weak test of a rule meant to preserve computation.

On task-trained recurrent networks it holds up. Noise-prune greatly outperforms magnitude-based pruning and performs on par with or better than a non-local strategy using second-order information — a strong comparator, and a non-local one, which is the point. The mechanism has two parts that both turn out to matter: connections are sampled for retention in proportion to importance rather than thresholded deterministically, and retained connections are rescaled to preserve average synaptic strength. Ablating either degrades performance. One detail is honest in a way that is easy to skip past: the empirically optimal degree of rescaling is lower than the original theoretical argument predicted.

The same substrate — local fluctuation statistics — supports gradient reconstruction in one paper and structural pruning in another. That is a signal about what noise is doing in a recurrent network, not two unrelated tricks.

The counterexample

Zhaotian Gu, Jie Su, Weiwei Wang, Chang Liu, Tianyi Qian and Dahui Wang import biology in the opposite direction, and their paper is included here because the contrast clarifies what the other four are claiming.

Their Recurrent Divisive Normalization Network keeps backpropagation through time entirely. What it changes is the unit: divisive normalisation, a canonical computation observed across cortical circuits, is built into the recurrent dynamics. The payoff is on continuous working memory, where classical continuous attractor networks are fine-tuning-fragile and GRUs and LSTMs shatter the state space into discretised point attractors instead of learning a manifold. The RDNN converges to robust, high-fidelity slow manifolds. The analysis of why is the interesting part: divisive normalisation introduces an activity-dependent local gradient scaling under BPTT that damps updates in highly active regimes, and this coincides with a self-compression of the network’s effective rank that confines dynamics to a low-dimensional subspace without the optimisation pathologies of explicit low-rank factorisation. Ablations show subtractive inhibition can hold a static memory but that division is mathematically necessary to prevent manifold shattering under time-varying input.

So there are two distinct ways to import a biological constraint. Change the learning rule and keep the architecture, or change the architecture and keep the learning rule. Four of these papers do the first; this one does the second, and it is the one with the cleanest functional result.

That asymmetry is worth sitting with. The forward-only literature has been evaluated almost exclusively on whether it recovers backpropagation’s accuracy, which frames it as an optimisation-cost story. The divisive-normalisation result is evaluated on whether it recovers a representational geometry that gradient descent alone does not find, which frames it as an architecture story. The forward-only work has not yet been asked the second question, and the answer would be more informative than another accuracy table.