  We analyze the time delay between emission of photoelectrons from the outer
valence $ns$ and $np$ sub-shells in noble gas atoms following absorption of an
attosecond XUV pulse. By solving the time dependent Schr\"odinger equation and
carefully examining the time evolution of the photoelectron wave packet, we
establish the apparent "time zero" when the photoelectron leaves the atom.
Various processes such as elastic scattering of the photoelectron on the parent
ion and many-electron correlation affect the quantum phase of the dipole
transition matrix element, the energy dependence of which defines the emission
timing. This qualitatively explains the time delay between photoemission from
the $2s$ and $2p$ sub-shells of Ne as determined experimentally by attosecond
streaking [{\em Science} {\bf 328}, 1658 (2010)]. However, with our extensive
numerical modeling, we were only able to account for less than a half of the
measured time delay of $21\pm5$ as. We argue that the XUV pulse alone cannot
produce such a large time delay and it is the streaking IR field that is most
likely responsible for this effect.
