Not sure which state you are at, in my state, the

earnest money definitely goes to seller under this situation.

(I am not a lawyer nor a real estate agent. Not responsible for errors in my statements.)

I think your situation is,
1) Your agent is not a good one and he is not represent your interest.
2) The buyer wants to back off but does not want to lose the earnest money.

I assume, it passes closing date already.

If I were you, I would,
1) Read contract carefully and find local/state rules regarding real estate transaction. You can ask city hall, local library people to help.
2) Once verify that you are entitled to the earnest money (and potential financial loss, interests, insurance and etc., as buyer refusing to close the deal), sit down with your realtor. Either you can request his manager's presence or tell him in front that you are going to record your conversation. Tell him that you think he is not doing his job. He is lying to you. He should talk to the seller, either close the deal or sign the document to back off the deal. So you can keep the earnest money and put your house back to market again. As you want to close the deal, you can forgive the late closing loss if the buyer close the deal. If the buyer decides to back off, you will ask for financial loss for the period between closing date and buyer sign-off the deal date. (You can go to small claim court for financial loss part.)
3) If the buyer still refuses to do either (close or sign-off the deal), you should consult a lawyer. Ask the lawyer to write a letter to seller. This won't cost too much (a couple of hundred dollars). Lots of people comply after receiving lawyer's letter.

Good luck and keep us updated.