My son is one year old, so it is perhaps a bit too early to worry about his mathematical education, but I do. I would like to hear from mathematicians that have older children: What do you wish you'd have known early? What do you think you did particularly well? What do you think would be particularly bad? Is there a book (for children or parents) that you recommend?

(This a community wiki, so please give one advice per answer, as usual.)

Background

I ask here because I believe that the challenges a mathematician faces in educating a child are special. For example, at least some websites and books address the parents' fear of not knowing how to solve homework, which keeps them from becoming involved. On the contrary, I fear I might get too involved and either bore my son or make him think he likes math when in fact his skills are elsewhere.

Christos Papadimitriou said in an interview that, even though his father was teaching math in high-school, they never discussed math. I wonder if that means his father didn't teach him how to count and I wonder if it's a good strategy. (It certainly turned out well in one case.)

Timothy Gowers (in Mathematics, a very short introduction) says that it was inappropriate to explain to his son, who was six, the concept 'zero' using the group axioms. (Or something to this effect, I don't have the book near to check.) That was surprising to me, because I wouldn't have thought that I need to restrain myself from mentioning abstract concepts. (Update. Here's the quote: "[The non-abstract] way of thinking makes it hard to answer questions such as the one asked by my son John (when six): how can nought times nought be nought, since nought times nought means that you have no noughts? A good answer, though not one that was suitable at the time, is that it can be deduced from the [field axioms] as follows. [...]")

There is a somewhat related Mathoverflow question. This one is different, because I'm looking for advice (rather than statistics/anecdotes) and because my goal is to give my son a good math education (rather than to make him a mathematician). I also found an online book that seems to give particularly good generic advice. Here I'm looking more for advice geared towards parents that are mathematicians.

In short, I'm looking for specific advice on how a mathematician should approach his/her child's math education, especially for the 1 to 10 age range.

The story I heard from a senior colleague when I was at this stage was:
"Twenty years ago I had no children and five theories on how to bring up children.
Now I have four grown-up children and no theories."