  Target Propagation (TP) is a biologically more plausible algorithm than the
error backpropagation (BP) to train deep networks, and improving practicality
of TP is an open issue. TP methods require the feedforward and feedback
networks to form layer-wise autoencoders for propagating the target values
generated at the output layer. However, this causes certain drawbacks; e.g.,
careful hyperparameter tuning is required to synchronize the feedforward and
feedback training, and frequent updates of the feedback path are usually
required than that of the feedforward path. Learning of the feedforward and
feedback networks is sufficient to make TP methods capable of training, but is
having these layer-wise autoencoders a necessary condition for TP to work? We
answer this question by presenting Fixed-Weight Difference Target Propagation
(FW-DTP) that keeps the feedback weights constant during training. We confirmed
that this simple method, which naturally resolves the abovementioned problems
of TP, can still deliver informative target values to hidden layers for a given
task; indeed, FW-DTP consistently achieves higher test performance than a
baseline, the Difference Target Propagation (DTP), on four classification
datasets. We also present a novel propagation architecture that explains the
exact form of the feedback function of DTP to analyze FW-DTP.
