Since you don't get the usage of annoy, I take your words as a compliment :) Does he sound a little annoyed, doesn't mean he is annoying, it means he is worrying about something.