I don't know which state you are in

But in New York, there is no buy's agent unless you sign an agreement with him/her. In essence, any such agent is a seller's agent and his/her commission will come out of the sale price after the closing. He/she should have made it clear to you in the outset. I believe you can change them any time you want. Just be aware that they may not represent you even if they help you to locate the property