Have you talked with other parents?

If they also have the same issue, you guys shall talk to the principal.

I agree with HappyChild, "trust your instinct". If you are the only one not comfortable with him, switch your daughter. You can still invite all her classmates to her party.

My son has a couple of male teachers in his daycare. We really like them as we can tell that they enjoy working with kids.