Male and female brains are wired differently according to this article:


  Maps of neural circuitry showed that on average women's brains were highly connected across the left and right hemispheres, in contrast to men's brains, where the connections were typically stronger between the front and back regions.


But since learning in the brain is associated with changes of connection
strengths between neurons, this could be or not the result of learning. 
What about physical differences from birth? Are there differences in size, regions, chemical composition, etc. from birth?  

Short answer
Yes, men and women's brains are different before birth.  

Background
First off, learning effects versus genetic differences is the familiar nature versus nurture issue. Several genes on the Y-chromosome, unique to males, are expressed in the pre-natal brain. In fact, about a third of the genes on the Y-chromosome are expressed in the male prenatal brain (Reinius & Jazin, 2009). Hence, there are substantial genetic differences between male and female brains.  

Importantly, the male testes start producing testosterone in the developing fetus. The female hormones have opposing effects on the brain as testosterone. In neural regions with appropriate receptors, testosterone influences patterns of cell death and survival, neural connectivity and neurochemical composition. In turn, while recognizing post-natal behavior is subject to parenting influences and others, prenatal testosterone may affect play behaviors between males and females, whereas influences on sexual orientation appear to be less dramatic (Hines, 2006). 

The question is quite broad and I would start with the cited review articles below, or if need be, the wikipedia page on the Neuroscience of sex differences.

References
- Hines, Eur J Endocrinol (2006); 155: S115-21
- Reinius & Jazin, Molecular Psychiatry (2009); 14: 988–9
