Sorry, but it goes both ways. Many men might otherwise be fine with a taller woman, but women have always expressed a strong desire for taller men. If you doubt that, just enroll in an online dating site and check out the preferences on womens' profiles.

There are always exceptions, as you point out, but they are exceptions to the rule. It could be that shorter men are just more desirable and lovable and talented. But that is just me guessing for fun.