you may have no choice but return all deposit

most contract specifies that the security deposit is for damages to the property, not for breach of contract. Unless you have such provisions in your contract, you have no legal basis to keep the deposit.