I’ve already said that Sam Harris is an idiot. This is only further confirmation of that fact: Sam Harris doesn’t understand how variations between genders work.
If the differences you are looking for are small, you will not be able to see them unless you have a proportionally large population in your study. If psychological differences exists at all, it seems to be quite small since most studies find nothing or find only tiny effects. Even if these effects are real, you could not see them out in the world because the sample of men and women you come across as you go about your daily life is just too small to be able to distinguish slight differences. This is why whenever a new evo-psych study comes out that says something like “women are slightly more cooperative than men” and the response from a couple of your annoying facebook friends that you haven’t spoken to since High School is “see, I told you women were just [insert sexist stereotype here]” they are wrong even if the science is right. Small differences in populations just cannot be seen in the small sample of people you meet in your life. Plus, you can’t even measure psychological traits very accurately so you are not going to be able to get accurate enough measurements to see those differences even if they were big enough to see otherwise. Your own judgments and biases are going to be larger than any actual differences between the groups if those differences exist at all.