Absoulately right! The real problem is your wife.

Her ex can surely sense the tone of your wife. If your wife is firm and simply ask him to fu@k off, I don't think he is going to call again.
It feels like your wife is playing a very dangerous game to you. it might be fun to her, but certainly not to you.