  We propose and study a method for learning interpretable representations for
the task of regression. Features are represented as networks of multi-type
expression trees comprised of activation functions common in neural networks in
addition to other elementary functions. Differentiable features are trained via
gradient descent, and the performance of features in a linear model is used to
weight the rate of change among subcomponents of each representation. The
search process maintains an archive of representations with accuracy-complexity
trade-offs to assist in generalization and interpretation. We compare several
stochastic optimization approaches within this framework. We benchmark these
variants on 100 open-source regression problems in comparison to
state-of-the-art machine learning approaches. Our main finding is that this
approach produces the highest average test scores across problems while
producing representations that are orders of magnitude smaller than the next
best performing method (gradient boosting). We also report a negative result in
which attempts to directly optimize the disentanglement of the representation
result in more highly correlated features.
