Don't worry

OK. Here are the things about closing cost and others:
As per contract, you need to pay all closing cost. This comes from Fannie Mae and written in contract. You didn't catch it before signing it, your fault or agents (depending on which one you wanted to blame). Have nothing to do with closing agent.
Will closing agent charge you more? Maybe a few hundred $, but not much. Also you can always ask for a bit reduction. I did 2 closings once and the settlement change be double settlement fee. I tried to fight it, to no avail. Remember that and tell other people, but cann't kill yourself for that.
What LZ was saying is that he ended up paying all closing cost for 3-4K, which sounds a lot, but not that much considering the contract issue. All tax, escrow, title insurance, some settlement fee are legit. What might be a bit high is the settlement fee, which is only a few hundred. I know it's a few hundred that you'll never see, but it's cost of doing business.

hope this helps.