At the time, as I recall, a lot of men were envious and a lot of women sighed and shook their heads. And of course, the assumption by many was that, in the way of women everywhere and throughout history, Margaret had essentially prostituted herself in return for money and status.
You can't ignore the fact Trudeau was hugely admired and for some reason I've never understood, thought of as extremely sexy and debonair by women of the time.
Brown... not so much.