I am completely on the fence on this one.
The thing about it is... the arguments have bleeded from practical ones into theoretical, which is messy.
Trans people are a tiny segment of the population, however they are abused and targeted at a very high rate. The idea of protecting their rights, to allow them to live safety in society is laudable, but there are always limits to rights as well.