You didn't even say which state you are adopting, impossible to answer.
Ultimately, the court look at the "best interest of the child" when dispute arises. So if you had a good family life and the child was happy, it would be difficult for the birth mother to take the child back.