If it is investment property as long as the deal make sense, I would say don't be too emotional. If you will live in there, then the question is do you want to live there or not. Don't tie it to the problem with your agent.
If you don't like the service of the buyer agent you can complaint and see if she will settle with you or risk getting something logged with the board.