Help! help!

My personal trainer asked me out for coffee next week. My feeling is complicated. It is the combination of a little bit sadness, a little bit feeling sorry for him, a little bit worried, and and everything but no happiness.
I have been thinking after he asked if/what I have done or said wrong to give him a wrong impression.

I don't enjoy it. My problem is I am the kind of person who is difficult to say no.

Thanks for all of the kindly posts JJMM gave to me last time.