Not Really
What your hu*****and said was BS and not the law. Child custody is really in the discretion of the judge. Most jurisdictions adopt "no fault" divorce these days and what you did before is not to be considered for the child custody, even you committed an adultry. The basic idea is that what is the best for the child. If the judge thinks that it is better for the child to see both parents then you get a joint custody and to my knowledge most family court judges like to do that. Unless he could raise some evidence such as child endangerment or child abuse, he can not deprave you of the child. No job and no income is not the thing a judge would consider. Even if you don't have any income, your hu*****and need provide child support and spousal support and that is law.
Don't be fooled by him. Read the law frist and try to settle with him. If you really need go to court for the child custody, then hiring a good lawyer is the key.