Ok. Sounds like you think everything is ok.
It depends what you mean by OK. It depends on the difference between illegal and stupid. It's legal for a University to ban speakers based on their speech, but pretty stupid, on a sliding scale. While I would advocate allowing anything, banning David Duke would not be as big a disservice to one's students as banning Ayaan Hirsi Ali.