Almost every time somebody talks about atoms, at some point they mention something like this:


  If we remove the spaces between the atoms and atomic components, we can fit the solar system in a thimble.


Or


  If we remove the spaces between the electrons and the nucleus, we can fit the universe in a baseball.


I know that atoms are mostly empty, but I've always thought that those statements are exaggerating.

Can we really fit the solar system in a thimble (if we remove all those spaces)?

Neither of those statements are true.  It's an easy approximation to make:  a neutron star has all of that 'space' removed from between nucleons --- so we just need to know how big a neutron star of mass equal to the solar system would be.  Well, the only significant mass is the sun (jupiter is about 1% the mass of the sun---negligible).  If the sun were compressed into a neutron star, it would have a radius of about 10km (up to 50% or so accuracy).  See this nice talk about neutron star radii.

Solar System:
So if you removed all of the 'space' between all of the atoms in the solar systems, it would form an object about the size of a large town, or small city.

Universe:
Obviously collecting all of this mass would yield a black-hole.  But conceptually, using some very order of magnitude estimates for the universe as a whole, if we assume there are roughly $10^{20}$ - $10^{22}$ stars (I think this estimate is quite high), then the radius would be something like a 1-100 Mpc or roughly 10 million to 1 billion light-years.



Edit (To address the question itself):
The concept of 'size' for atoms and nuclei has some grey area, but you can define the size of a hydrogen atom, or the size of a proton/neutron to an order of magnitude.  A statement like 'remove all of the empty space' is much more nebulous, and ends up being largely a question of semantics.  A more accurate way of phrasing the underlying concept being addressed might be something like:
 'Roughly how much volume do the dominant mass-constituents of matter take up?'
 The idea is that nucleons (protons and/or neutrons) are 2000 times more massive than electrons, and thus the important component of mass.  At the same time, the electrons are the dominant volume-fillers (by a factor of about $10^{15}$).
