Actually I would prefer the concept of “tolerance” to “right”. The problem with “right” is its essentially positive and affirmative connotations, that if one has the right to do something then that thing is in some sense a right thing to do and to be affirmed. This is why it is a short path from having the right to do something to universal compulsory affirmation of that something otherwise one is a bigot or hater, etc.
On the other hand, the connotations of tolerance is precisely pertaining to things which you dont approve of, or may even violently object to. You do not need to tolerate opinions or behaviours which you like or approve. However, prudence prescribes that we tolerate, we suffer, all kinds of unpleasant and disagreeable people and behaviours for the cost of coercive censures may outweigh the benefits for particular cases. Thus, tolerance, or longsuffering, is a virtue whereby we learn to suffer all kinds of fools and morons we disagree with or dislike.
Let us not forget that even John Locke’s Letter was styled “A Letter Concerning Toleration” and not rights. I can sharply and violently dismiss Buddhism as nothing more than a cosmic suicide pact to exterminate one’s personal existence in nirvana to escape suffering while tolerating their foolish and idolatrous practice in society, etc.
The moment when America diverged from the original English tradition of toleration was when “tolerance” was displaced by “rights” as a key concept of civic polity. If there is an intelligible concept of rights in the civic space, then it can only be the right to do good and not evil, and there are definite certain limits to the amount of evils a society can tolerate.