Lots of the research that I've seen in the 80's and 90's supports your argument that women are definitively worse than men in the visual-spatial aspects of mathematics. However, pretty well everything I've seen published in the last 15 years doesn't really agree with this, in the sense that yes, by the end of high school males are performing better but if you look at children the trend is absent or less apparent. These papers I found seem to suggest that it is our constant parroting of the idea that girls suck at math that makes them think they shouldn't pursue math and are bad at it. Additionally, the women who do pursue math feel the need to emphasize their masculine traits and still feel a lack of inclusion, such that only those that are the most self-sufficient tend to stick with it.
From the paper : Where Are All the Women? Gender Differences in Participation in Physical Science and Engineering.
Our analyses suggest that the main source of gender differences in entry into physical science and engineering occupations is not gender differences in either math aptitude or a sense of personal efficacy to succeed at these occupations, rather it is a gender difference in the value placed on different types of occupations. Furthermore, our results suggest that these differences begin influencing educational decisions quite early in life.
http://psycnet.apa.org/index.cfm?fa=buy.optionToBuy&uid=2006-22337-016
From the review : Sex Differences in Intrinsic Aptitude for Mathematics and Science?
This article considers 3 claims that cognitive sex differences account for the differential representation of men and women in high-level careers in mathematics and science:
(a) males are more focused on objects from the beginning of life and therefore are predisposed to better learning about mechanical systems;
(b) males have a profile of spatial and numerical abilities producing greater aptitude for mathematics; and
(c) males are more variable in their cognitive abilities and therefore predominate at the upper reaches of mathematical talent.
Research on cognitive development in human infants, preschool children, and students at all levels fails to support these claims. Instead, it provides evidence that mathematical and scientific reasoning develop from a set of biologically based cognitive capacities that males and females share. These capacities lead men and women to develop equal talent for mathematics and science.
http://rhig.physics.yale.edu/~nattrass/WomenInPhysicsArticles/ReviewAmericanPsychologistSpelkeDec2005.pdf
From the paper: Implicit Stereotypes, Gender Identification, and Math-Related Outcomes: A Prospective Study of Female College Students
The present findings suggest that implicit gender stereotypes about math aptitude, in conjunction with gender identification, reduces women's math performance and their desire to performance pursue
math-intensive careers. We examined math-related outcomes among women enrolled in rigorous introductory college calculus courses. The less women were gender identified and the less they
possessed implicit gender stereotypes about math ability, the better they performed on the final exam and the more likely they were to express interest in a math-related career.
Women's math performance is debilitated by stereotypes when the stereotypes are automatically activated or when women perceive them to be self-relevant. This debilitation may explain why math-identified women distance themselves from stereotypically feminine characteristics (Pronin et al 2004)
From the paper: Identity bifurcation in response to stereotype threat: Women and mathematics
Three studies explored women's bifurcation of feminine identity as a response to threatening stereotypes in the domain of mathematics. Study 1 demonstrated that women in a math class who previously had completed a large number of math courses disavowed ‘‘feminine characteristics’’ strongly associated with stereotypes about women's potential for math success (e.g., flirtatiousness, planning to have children) but not characteristics perceived to be weakly associated with these stereotypes (e.g., empathy, nurturance), more than women who had completed fewer math courses. Studies 2 and 3 directly manipulated stereotype threat by presenting a scientific article reporting stereotype-consistent sex differences in math aptitude. As predicted, women strongly iden- tified with mathematics responded to this threatening article by disavowing feminine characteristics strongly associated, but not those weakly associated, with the relevant negative stereotypes, while women not strongly identified with mathematics showed no such differentiation in response. Theoretical and practical implications of these results are discussed.
From the paper: Women Engineering Students and Self-Efficacy: A Multi-Year, Multi-Institution Study of Women Engineering Student Self-Efficacy.
Results indicate that while women students show positive progress on some self-efficacy and related subscales, they show a significant decrease on feelings of inclusion from the first to second measurement period and further suggest a relationship between ethnicity and feelings of inclusion. Additionally, correlations show that self-efficacy is related to women students' plans to persist in this predominantly male discipline.
In my experiences from living in residence, the guys who were in engineering weren't doing any better in classes that they shared with us girls and guys in science. I remember a group of them telling me to drop linear algebra because they almost failed so I wouldn't stand a chance - I cruised through with a B or B+ pretty easily. Because guess what, I may be a girl in biology but I'm smart and I've never struggled to grasp mathematical concepts.
What did seem to distinguish the guys from the girls was the guys tended to be more practical with their choice of degrees - they were pursuing something that would make them money so they were almost all in engineering or business. The girls on the other hand tended to study what they were interested in and many of them had the idealistic 'I want to help people/animals/the planet' view - they were spread between biology, anthropology, geology, nursing, business, IR, economics, architecture, as well as a couple in engineering. I'm not sure if there's any biological difference at work here - I'm sure evo-anth and evo-psych people would argue that this is because females evolved to fulfil the role of carer and males evolved to fulfil the role of breadwinner. I could also see it be a result of parents having different expectations for their sons, so they are raised expecting to fulfil the breadwinner role.
Additionally, engineering was actually something I considered pursuing at one point because it emphasizes skills that I do well with (for example these spatial skills that are repeatedly mentioned here - when I was tested on that section of the IQ test I fell off the top end of the bell curve). However, I couldn't stand the attitude of these engineering students. Why would I want to spend 4 years studying with followed by a life-time of working exclusively with these guys who would never shut up about their superiority (to anyone), and felt the need to tell me that I would fail because I was a woman and currently in biology.