Our VP used to come to our office to talk to my friend every day

Very often, the conversation last more than an hour. I was very annoyed. At one point, I even wanted to change office. But I didn't because I did not want to hurt my friend's feelings. Our direct boss, who is below the VP in rank was annoyed too. But he didn't do much to stop that either. We had to deal with him for almost 8 years. He is a good person otherwise though. He is gone now.

If you have other opportunity, that'll be great. But you never know if you'll get into the same situation or not. If you like the current job, you should not have to quit. If he wants to talk, talk with him but remind him you have work to do. It's very inconsiderate of that kind of people to use their power to make other people feel uncomfortable. I don't have much respect for them.