Don't play with fire, you will be burned.

What he did was harassment. If you say yes  once, there will be 2nd time and 3rd time....
Don't let him get started.  Be smart, be firm, say no.  He will respect you if you respect yourself.

Unless you want to take advantage of this as gold digger, then be it, be a pushover.

I agree with one say 苍蝇不盯无缝的蛋!.  I am sure there is a reason why he chose you as a target of harassment.  If some physical relationship happens between you and him, you are partially responsible, I am sure.