You can talk with him

Don't make things get too complicated. Just tell him that you don't think his behavior was right, and the other lady has a family too.

Some people act like that, men and women both. He should be aware of after your conversation. Move on, forget about it. It is more important to keep your family together.