need to take it seriouly

To some states - if people cross your property all the time, they have the right to claim your property after certain years.
If I were you, I would -
1) get a property survey
2) set up a sign
3) If they persist, get a lawyer to send them a legal letter