  We demonstrate that 1x1-convolutions in 1D time-channel separable
convolutions may be replaced by constant, sparse random ternary matrices with
weights in $\{-1,0,+1\}$. Such layers do not perform any multiplications and do
not require training. Moreover, the matrices may be generated on the chip
during computation and therefore do not require any memory access. With the
same parameter budget, we can afford deeper and more expressive models,
improving the Pareto frontiers of existing models on several tasks. For command
recognition on Google Speech Commands v1, we improve the state-of-the-art
accuracy from $97.21\%$ to $97.41\%$ at the same network size. Alternatively,
we can lower the cost of existing models. For speech recognition on
Librispeech, we half the number of weights to be trained while only sacrificing
about $1\%$ of the floating-point baseline's word error rate.
