I read the definition of work as 
$$W ~=~ \vec{F} \cdot \vec{d}$$ 
$$\text{ Work  =  (Force) $\cdot$ (Distance)}.$$
If a book is there on the table, no work is done as no distance is covered. If I hold up a book in my hand and my arm is stretched, if no work is being done, where is my energy going?
While you do spend some body energy to keep the book lifted, it's important to differentiate it from physical effort. They are connected but are not the same. Physical effort depends not only on how much energy is spent, but also on how energy is spent.

Holding a book in a stretched arm requires a lot of physical effort, but it doesn't take that much energy. 


In the ideal case, if you manage to hold your arm perfectly steady, and your muscle cells managed to stay contracted without requiring energy input, there wouldn't be any energy spent at all because there wouldn't be any distance moved.
On real scenarios, however, you do spend (chemical) energy stored within your body, but where is it spent? It is spent on a cellular level. Muscles are made with filaments which can slide relative to one another, these filaments are connected by molecules called myosin, which use up energy to move along the filaments but detach at time intervals to let them slide. 
When you keep your arm in position, myosins hold the filaments in position, but when one of them detaches other myosins have to make up for the slight relaxation locally.  Chemical energy stored within your body is released by the cell as both work and heat.* 


Both on the ideal and the real scenarios we are talking about the physical definition of energy. On your consideration, you ignore the movement of muscle cells, so you're considering the ideal case. A careful analysis of the real case leads to the conclusion that work is done and heat is released, even though the arm itself isn't moving.

* Ultimately, the work done by the cells is actually done on other cells, which eventually dissipates into heat due to friction and non-elasticity. So all the energy you spend is invested in keeping the muscle tension and eventually dissipated as heat.
