  Activation in deep neural networks is fundamental to achieving non-linear
mappings. Traditional studies mainly focus on finding fixed activations for a
particular set of learning tasks or model architectures. The research on
flexible activation is quite limited in both designing philosophy and
application scenarios. In this study, three principles of choosing flexible
activation components are proposed and a general combined form of flexible
activation functions is implemented. Based on this, a novel family of flexible
activation functions that can replace sigmoid or tanh in LSTM cells are
implemented, as well as a new family by combining ReLU and ELUs. Also, two new
regularisation terms based on assumptions as prior knowledge are introduced. It
has been shown that LSTM models with proposed flexible activations P-Sig-Ramp
provide significant improvements in time series forecasting, while the proposed
P-E2-ReLU achieves better and more stable performance on lossy image
compression tasks with convolutional auto-encoders. In addition, the proposed
regularization terms improve the convergence, performance and stability of the
models with flexible activation functions.
