Giving someone else equal right does not infringe or take away rights from you.
It just makes it illegal to enforce your prejudice and hate. It’s that simple.