The tanh activation function is: 

$$tanh \left( x \right) = 2 \cdot \sigma \left( 2 x \right) - 1$$

Where $\sigma(x)$, the sigmoid function, is defined as:
 $$\sigma(x) = \frac{e^x}{1 + e^x}$$.

Questions:    


Does it really matter between using those two activation functions
(tanh vs. sigma)?
Which function is better in which cases?


Yes it matters for technical reasons. Basically for optimization. It is worth reading Efficient Backprop by LeCun et al.

There are two reasons for that choice (assuming you have normalized your data, and this is very important):


Having stronger gradients: since data is centered around 0, the derivatives are higher. To see this, calculate the derivative of the tanh function and notice that its range (output values) is [0,1].


The range of the tanh function is [-1,1] and that of the sigmoid function is [0,1]


Avoiding bias in the gradients. This is explained very well in the paper, and it is worth reading it to understand these issues.

