Please give me some advice,thx

I locked in a 30 year's rate with a bank last week, but the next day, I found a lot better rate with a mortgage broker, so I switched to this broker. Now the bank is telling me I need to pay for 300$ apprisal fee since they have locked in the rate for me. But first: They have not done the apprisal yet since I bought a new construction and me house is not ready yet; plus, I told them that I changed my mind the very next day after they lock the rate, so they have not start anything yet. Second, before they lock the rate for me, they did not tell me that I need to pay for this 300$ upfront, and I did not sign any agreement or disclosure package. In the good faith estimate, this is included in the closing cost. Again, since they have not done the apprisal, I see no reason to pay for this 300$. Can someone give me some suggestion. Thank you very much in advance!