  The study of the expressive power of neural networks has investigated the
fundamental limits of neural networks. Most existing results assume real-valued
inputs and parameters as well as exact operations during the evaluation of
neural networks. However, neural networks are typically executed on computers
that can only represent a tiny subset of the reals and apply inexact
operations, i.e., most existing results do not apply to neural networks used in
practice. In this work, we analyze the expressive power of neural networks
under a more realistic setup: when we use floating-point numbers and operations
as in practice. Our first set of results assumes floating-point operations
where the significand of a float is represented by finite bits but its exponent
can take any integer value. Under this setup, we show that neural networks
using a binary threshold unit or ReLU can memorize any finite input/output
pairs and can approximate any continuous function within an arbitrary error. In
particular, the number of parameters in our constructions for universal
approximation and memorization coincides with that in classical results
assuming exact mathematical operations. We also show similar results on
memorization and universal approximation when floating-point operations use
finite bits for both significand and exponent; these results are applicable to
many popular floating-point formats such as those defined in the IEEE 754
standard (e.g., 32-bit single-precision format) and bfloat16.
