Because that isn't how Human Rights Codes work, Magoo. Or most law, for that matter.
It isn't a case of limits so much as when recognizing rights becomes a potentially greater rights violation for someone else. In this case, women. There is no way you are going to anticipate every possible future conflict in order to put it down on paper, even if you tried. It all comes down to practical application.
But while I think women have some legitimate concerns (even if I don't think this amendment is going to have an impact on that), there are plenty of critics looking on this thinking about the threat to their own privilege, just like other rights struggles, and wondering about how to recognize some rights so long as they don't get too uppity.
That is why I think the notion of limits is a questionable one when we are talking about universal rights. If we are evaluating that in an honest way the only limit is when it impacts on others' universal rights, and that is a hard thing to weigh.
(edit)
I mentioned upthread a number of examples of laws which, if taken by the letter, would cause serious problems. That generally doesn't happen, and when it does happen, they often wind up being struck down. That is unlikely to happen in the case of recognizing a fundamental right like gender identity and expression, because it actually is very solid grounds for protection.