What should I do?

Fell in love with a 31 yo extremly beautiful lady. She had a daughter 2 yo and I have two kids 2 yo and 4 yo respectively. We separted with our own spouses and moved out and are living together now. We often miss our own kids and want to go back. That thought made us feeling bad. We cannot live in harmony recently only for that reason. What should we do? Any suggestions. So many thanks.