The excellent Feminism 101 entry about sexism includes discussions of reverse sexism, benevolent sexism and unintentional sexism.
A running theme in a lot of feminist theory is that of institutional power: men as a class have it, women as a class don’t. Obviously the power dynamics do shift around depending on the culture and the time period (not to mention the individual, the other privileges that the person does/does not have, etc etc), but ultimately the scales remain tipped in favor of men in general (if you disagree with that statement, please go read the Why do we still need feminism? FAQ entry first before proceeding).
What this imbalance of power translates to on an individual level is a difference in the impact of a man being prejudiced towards a woman and a woman being prejudiced towards a man. While both parties are human, and therefore have the same capacity to be hurt by the prejudice, whether they like it or not, the men have a whole system of history, traditions, assumptions, and in some cases legal systems and “scientific” evidence giving their words a weight that the women don’t have access to.