If a large sum of deposit has been paid, you want to only keep the portion that reflects the "damage" - namely the "potential" rent loss due to taking the property off market.
Prepare a written agreement, explain to tenant and ask him/her to sign a mutual release and agreement on retaining the EMD.
Be firm but flexible. Don't push too hard as it may lead to a lawsuit/complaint.
Also consider tenant situation and the reason for the change. If they are really less privileged people, make it easier for them.
Remember renting is a business, sometimes things just don't work out and you loose some money
