if he doesn't even bother to answer

why would you bother to keep going out with him? Hey, I am a guy, if I care about someone's feeling, I wouldn't answer it that way, and no one would. Stop fooling yourself, and looking for excuses for him.