1. Did you have the police report when you got involved in the accident? That helped you determining whether that person got injuried or not. Usually they do their best to get something for themselves since they don't lose anything at all.
2. How can you know that person over the phone is from your car insurrance compony? Usually when you reported the accident to insurrance company, they will take care of it and they may inspect your car damage. You don't need to talk with others any more.
Therefore, take it easy and sleep well. You have nothing to lose except the increase of your car insurance premium in the future.