Quite often, actually.
Not often, but I sometimes get invited to a social event without realizing that the group has a common interest that I don't wish to share (like rabid politics). I try to slip away as discreetly as possible. There are also places where I personally wouldn't feel comfortable, so I stay away.
I know for a fact that i fit in with a FB group called Pizza Lovers All Around The World United. Sadly, those people have no clue that i love pizza way more than they ever could.
Not often, but when I feel that way I move on.
Everybody goes through with that frame of mind, now and then.