if you directly signed with the tenant,-- that's easy, he has to pay what he had agreed at. End of the contract term, you don't renew the contract with him, he has to pack up and go.
if you signed a contract with the agent, and the agent signed a contract with the tenant, then tell the tenant that the agent has dissolved the contract with you, so there is not contract standing. If the tenant does not agree, ask him for evidence. If he still argues, just tell him you received no rent from the agent, so you have to right to take back your property. If the tenant wants to carry on with the tenancy, he has to sign another contract with you. About the tenant's deposit, that's a business between him and the agent, it has nothing to do with you.
If it comes to the worst, you have to hire a solicitor represent you and send him a letter; if it does not stop here, you have to get the solicitor to apply a court order to evict him. A good solicitor will charge you 163;130 per hour, and 163;50 per letter, but it is worth it, trust me.
If the tenant is an English yob, you will have some trouble to get rid of him;
if he is a foreign guy, he will be deterred by the letter; tell him if he does not comply, you will report to the immigration office. It is not because the Immigration office will interfere with this civil matter, just to scare a foreign git.