Waiting on friends while not making any purchases IS loitering.
And people do it all the time.
I think you miss the obvious racism here that this manager thought black men chilling outside a Starbucks was somehow threatening. That's something I've never experienced even if I was technically loitering.
Starbucks is the place people go to, to loiter. Buy ONE coffee and use their Wifi for 3 hours.