  Credit assignment problem of neural networks refers to evaluating the credit
of each network component to the final outputs. For an untrained neural
network, approaches to tackling it have made great contributions to parameter
update and model revolution during the training phase. This problem on trained
neural networks receives rare attention, nevertheless, it plays an increasingly
important role in neural network patch, specification and verification. Based
on Koopman operator theory, this paper presents an alternative perspective of
linear dynamics on dealing with the credit assignment problem for trained
neural networks. Regarding a neural network as the composition of sub-dynamics
series, we utilize step-delay embedding to capture snapshots of each component,
characterizing the established mapping as exactly as possible. To circumvent
the dimension-difference problem encountered during the embedding, a
composition and decomposition of an auxiliary linear layer, termed minimal
linear dimension alignment, is carefully designed with rigorous formal
guarantee. Afterwards, each component is approximated by a Koopman operator and
we derive the Jacobian matrix and its corresponding determinant, similar to
backward propagation. Then, we can define a metric with algebraic
interpretability for the credit assignment of each network component. Moreover,
experiments conducted on typical neural networks demonstrate the effectiveness
of the proposed method.
