  In the current research of neural networks, the activation function is
manually specified by human and not able to change themselves during training.
This paper focus on how to make the activation function trainable for deep
neural networks. We use series and linear combination of different activation
functions make activation functions continuously variable. Also, we test the
performance of CNNs with Fourier series simulated activation(Fourier-CNN) and
CNNs with linear combined activation function (LC-CNN) on Cifar-10 dataset. The
result shows our trainable activation function reveals better performance than
the most used ReLU activation function. Finally, we improves the performance of
Fourier-CNN with Autoencoder, and test the performance of PSO algorithm in
optimizing the parameters of networks
