  We define and establish the conditions for `equivalent neural networks' -
neural networks with different weights, biases, and threshold functions that
result in the same associated function. We prove that given a neural network
$\mathcal{N}$ with piece-wise linear activation, the space of coefficients
describing all equivalent neural networks is given by a semialgebraic set. This
result is obtained by studying different representations of a given piece-wise
linear function using the Tarski-Seidenberg theorem.
