According to Steven Krantz's Mathematical Apocrypha (pg. 186):


  As was custom, Weil often attended tea
  at [Princeton] University . Graduate
  student Steven Weintrab one day went
  about the room asking various famous
  mathematicians who was the greatest
  mathematician of the twentieth
  century. When he asked Weil, the
  answer (without hesitation) was "Carl
  Ludwig Siegel (1896-1981)."


As the title of Krantz's book suggests, the anecdote may be apocryphal. However, there are other better grounded accounts of great mathematicians expressing the highest admiration for Siegel:

(A) In The Map of My Life Shimura wrote:


  I always thought that few people
  really understood my work. I knew that
  Chevalley, Eichler, Siegel, and Weil
  understood my work, and that was
  enough for me [...] Of course [Siegel]
  established himself as one of the
  giants in the history of mathematics
  long ago [...] Among his
  contemporaries, [Weil] thought highly
  of Siegel [...]


(B) In an published interview (pg. 30) Selberg said


  [Siegel] was in some ways, perhaps,
  the most impressive mathematician I
  have met. I would say, in a way,
  devestatingly so. The things that
  Siegel tended to do were usually
  things that seemed impossible. Also
  after they were done, they seemed
  still almost impossible.


Why might Weil, Shimura and Selberg have been so impressed by Siegel? I should emphasize that I'm not trying to precipitate a debate about the relative standing of historical mathematicians - rather - I'm hoping to learn about aspects of Siegel's work that I might otherwise overlook. I'm also not looking for, e.g. quotations from the Wikipedia article on him, but rather, less familiar material.

No one with any familiarity with his work can doubt that Siegel was one of the greatest mathematicians of the 20th century.  Weil was a decisive, opinionated man -- just the type of person who would have an answer to this question ready at hand.  And "Carl Ludwig Siegel" is a totally unsurprising answer from anyone.  (Also "Andre Weil" would be a totally unsurprising answer from anyone: it might be my answer!)

But it is especially unsurprising coming from Weil.  The list of contemporary mathematicians of the Siegel-Weil caliber is short enough, and among mathematicians on that list -- e.g. Wiener, von Neumann, Kolmogorov, Godel -- 
the research interests of Siegel and Weil were especially close: for instance, there is a Siegel-Weil formula.  Both brought their prodigious knowledge and technique to bear on number theory, but with distinct, and distinctive, styles.  To be very brief and crude, Weil had a fundamentally algebraic approach, whereas Siegel had a fundamentally analytic approach.  My own approach to mathematics is rather close to Weil's (although in magnitude, microscopic compared to his): I very much appreciate that finding the right bit of "structure" can make the solution of your problems self-evident.  A lot -- by no means all -- of Weil's work is like that: the finished product is so tidy and efficacious that you too easily forget to ask how he thought of any of it in the first place.  To someone with this "algebraic" style, Siegel's work looks like a sequence of miracles.  So it is unsurprising to me that someone like Weil would select someone like Siegel to give his top regards.

I think you can also gain some insight into why Weil named Siegel by considering their ages: Siegel (born in 1896) was ten years older than Weil (born in 1906).  Ten years is long enough for Siegel always to have been ahead of Weil in his career and stature, but short enough for them to be true contemporaries and competitors.  Most other great mathematicians that spring to mind when I think of Weil are actually quite a bit younger, e.g. Serre (born 1926), Tate (born 1925), Shimura (born 1930); it makes sense that Weil is not going to name any of these as the greatest mathematician of the 20th century.  Indeed all three are alive well into the 21st century.  

[Added: I just remembered that Chevalley (born 1909) was a contemporary of Weil of a similar stature.  But Chevalley was very close to Weil, both personally and in mathematical styles and tastes.  It is psychologically natural to esteem (and fear) most that which is most different from ourselves, not that which is most similar.  Anyway, for Weil to name Chevalley would have sounded arrogant, as if not being able to name himself he picked the person standing right next to him.]  

By the way, I think that Shimura and Siegel are quite similar in style as well as stature.  I read Shimura's autobiography, and I think he is right to be profoundly disappointed that Siegel did not take more of an interest in his work.  Shimura's work is closer to being a continuation of Siegel's (including a continuation of the brilliance, creativity and orginality!) than any other mathematician I can think of, so it is natural that Shimura holds Siegel in high regard.  

There is also something "organic" in the work of both Siegel and Shimura which naturally bristles a bit at the "Bourbakistic" influence of the French school: it seems clear enough, for instance, that the modern theory of "Shimura varieties" is both an addition and a subtraction from what Shimura himself intended.  I know several of Shimura's students, and though they work in what the rest of the mathematical world thinks of as parts of algebraic number theory and arithmetic geometry, in the way they actually think about mathematics they take a more analytic approach...like Siegel.  I have even fewer credentials to speak for Selberg than I do for any of these others, but I imagine that he may have felt a similar kinship to Siegel, i.e., the use of an "analytic" approach to studying problems that others regard as being more algebraic.  
