Stop to thinking him anymore! He is other woman's husband.

Try to have more connection with other colleagues and less with him. You obviously behaved weirdly too him. He just wants to treat you as one of the normal colleagues.