"Just say COVID,

 

then you are not harassing him, and he'll not harass you. Otherwise, you are making my job difficult."

If he insists...

"This is not an issue of your right of speech.  This is an issue of propriety.  This is a work place, not a bar.  We are your co-workers, not your drinking buddies.  Something you can do does not mean something you should do.  If you insist on creating an inhospitable environment for your co-worker, I'd like to hear your reason.  You have choices.  Choose wisely."