  This paper derives a complete set of quadratic constraints (QCs) for the
repeated ReLU. The complete set of QCs is described by a collection of
$2^{n_v}$ matrix copositivity conditions where $n_v$ is the dimension of the
repeated ReLU. We also show that only two functions satisfy all QCs in our
complete set: the repeated ReLU and a repeated "flipped" ReLU. Thus our
complete set of QCs bounds the repeated ReLU as tight as possible up to the
sign invariance inherent in quadratic forms. We derive a similar complete set
of incremental QCs for repeated ReLU, which can potentially lead to less
conservative Lipschitz bounds for ReLU networks than the standard LipSDP
approach. Finally, we illustrate the use of the complete set of QCs to assess
stability and performance for recurrent neural networks with ReLU activation
functions. The stability/performance condition combines Lyapunov/dissipativity
theory with the QCs for repeated ReLU. A numerical implementation is given and
demonstrated via a simple example.
