According to some chemistry textbooks, the maximum number of valence electrons for an atom is 8, but the reason for this is not explained. 

So, can an atom have more than 8 valence electrons? 

If this is not possible, why can't an atom have more than 8 valence electrons?

Yes, it can. We have molecules which contain "superoctet atoms". Examples:

$\ce{PBr5, XeF6, SF6, HClO4, Cl2O7, I3- ,  K4[Fe(CN)6], O=PPh3 }$

Almost all coordination compounds have a superoctet central atom.

Non-metals from Period 3 onwards are prone to this as well. The halogens, sulfur, and phosphorus are repeat offenders, while all noble gas compounds are superoctet. Thus sulfur can have a valency of +6, phosphorus +5, and the halogens +1, +3, +5, and +7. Note that these are still covalent compounds -valency applies to covalent bonds as well.

The reason why this isn't usually seen is as follows. We basically deduce it from the properties of atomic orbitals.

By the aufbau principle, electrons fill up in these orbitals for period $n$:

$n\mathrm{s}, (n-2)\mathrm{f},(n-1)\mathrm{d},n\mathrm{p}$

(theoretically, you'd have $(n-3)\mathrm{g}$ before the $\mathrm{f}$, and so on. But we don't have atoms with those orbitals, yet)

Now, the outermost shell is $n$. In each period, there are only eight slots to fill in this shell by the Aufbau principle - 2 in $n\mathrm{s}$, and 6 in $n\mathrm{p}$. Since our periodic table pretty much follows this principle, we don't see any superoctet atoms usually.

But, the $\mathrm{d,f}$ orbitals for that shell still exist (as empty orbitals) and can be filled if the need arises. By "exist", I mean that they are low enough in energy to be easily filled. The examples above consist of a central atom, that has taken these empty orbitals into its hybridisation, giving rise to a superoctet species(since the covalent bonds add an electron each)

I cooked up a periodic table with the shells marked. I've used the shell letters instead of numbers to avoid confusion. $K,L,M,N$ refer to shell 1,2,3,4 etc. When a slice of the table is marked "M9-M18", this means that the first element of that block "fills" the ninth electron in the M(third) shell, and the last element fills the eighteenth.

Click to enlarge:



(Derivative of this image)

Note that there are a few irregularities, with $\ce{Cu}$, $\ce{Cr}$, $\ce{Ag}$, and a whole bunch of others which I've not specially marked in the table.
