If you want police's help, yes, you do need to follow up, and they may not be able to do what you want them to do. They have to go by book.
If those two men are targeting you specifically, problem will drag for long time, be prepared. Hopefully, it’s just an accident.
First, get a fence is a good idea.
Then, get a copy of incident repot from the police and document future incidents.
Have a camera ready, pictures don't lie.
If you are concerned about children's safety, go to your local police station, talk to somebody in charge, and get his advice how to deal with the situation. He probably tells you to call them if the problem happens again.
Whenever incident happens, get in the house,specially, with the kids, call police (if it's not an emergency, call local police might be faster.) emphasize kids are involved, but don't exaggerate. Never, ever, take the law into your own hands.