Today I came across a new topic called the Mathematical Expectation. The book I am following says, expectation is the arithmetic mean of random variable coming from any probability distribution. But, it defines expectation as the sum of product of some data and the probability of it. How can these two (average and expectation) be same? How can the sum of probability times the data be the average of whole distribution?
Informally, a probability distribution defines the relative frequency of outcomes of a random variable - the expected value can be thought of as a weighted average of those outcomes (weighted by the relative frequency). Similarly, the expected value can be thought of as the arithmetic mean of a set of numbers generated in exact proportion to their probability of occurring (in the case of a continuous random variable this isn't exactly true since specific values have probability $0$). 

The connection between the expected value and the arithmetic mean is most clear with a discrete random variable, where the expected value is

$$ E(X) = \sum_{S} x P(X=x) $$ 

where $S$ is the sample space. As an example, suppose you have a discrete random variable $X$ such that:

$$ X = \begin{cases} 1 & \mbox{with probability  } 1/8 \\ 
2 & \mbox{with probability } 3/8 \\
3 & \mbox{with probability } 1/2
\end{cases} $$

That is, the probability mass function is $P(X=1)=1/8$, $P(X=2)=3/8$, and $P(X=3)=1/2$. Using the formula above, the expected value is 

$$ E(X) = 1\cdot (1/8) + 2 \cdot (3/8) + 3 \cdot (1/2) = 2.375 $$

Now consider numbers generated with frequencies exactly proportional to the probability mass function - for example, the set of numbers $\{1,1,2,2,2,2,2,2,3,3,3,3,3,3,3,3\}$ - two $1$s, six $2$s and eight $3$s. Now take the arithmetic mean of these numbers: 

$$ \frac{1+1+2+2+2+2+2+2+3+3+3+3+3+3+3+3}{16} = 2.375 $$

and you can see it's exactly equal to the expected value. 
