I had hoped that you might worked out but from what you said here it doesn't look likely.
It looks that you have married the wrong man in the beginning. He has not cared much about you anyway. He doesn't care much about your children either.
Think carefully if this is the man that you want to live with for another 20, 30, ... years. That you have been with him for x number of years does not mean you should not have a better man.
In my judgement, what he has done in the past makes him an unqualified hu*****and to a good wife. You have done so much for him but he shows no appreciation of that. That must have hurt you a lot in the past and I am afraid this is going to be a factor that will doom this marriage eventually.
You can ask him to come back for some time (1-2 month minimum) to help you out as well as repairing the relationship. (Evidently you can not go back given your situation; who is going to pay the mortgage? Him?) If he is not willing to take that, then just tell him to come back or the game is over.
Tell me where you are and I will help you to find a good lawyer. I think you will need one.