Is your boss single too? I would suggest keeping everything prof

At work. If he's single and wants to hang out with you after work. I think it should be fine. Don't let that coworker's insecurity bothers you. That's her problem. I don't think you need to be worried about what they said over the phone.