  In the chapter "Magic with a Matrix" in \emph{Hexaflexagons and Other
Mathematical Diversions} (1988), Martin Gardner describes a delightful "party
trick" to fill the squares of a $d$-by-$d$ chessboard with nonnegative integers
such that the sum of the numbers covered by any placement of $d$ nonthreatening
rooks is a given number $N$. We consider such chessboards from a geometric
perspective which gives rise to a family of lattice polytopes. The polyhedral
structure of these Gardner polytopes explains the underlying trick and enables
us to count such chessboards for given $N$ in three different ways. We also
observe a curious duality that relates Gardner polytopes to Birkhoff polytopes.
