Do not worry

Here is america, chaging a job is very normal. I got a same situation with you. Finally we three postdoc all left that lab due to the same reason. So, go ahead and find a new position. It is simple.