When the moment comes to make it clear,

 

just say: "sorry, it's not my cup of tea."

There's no need to be indignant.  People see what they want to see, and believe what they want to believe.  People do things because who they are, not because who you are.

If you are overly concerned about what others see in you, that means you have little of your self.  You need to work on your own stuff and not to be distracted by your own fear.