You don't have legal right here based on your contract.

Your agent is correct: if you don't want to sell, you need to consult an attorney to see whether you have other legal options. The other option is try to get the buyer to agree to not enforce the contract, e.g. by giving him money.