and you wanted to run your own life as you should. You need to live your own life, not to live as an extension of someone else's life. Should not feel quilty about that.
The way we were brought up, it is very hard to say to ourselves that "I am fully responsible for myself" without feeling somewhat shaky about such claim, i.e., am I allowed to make such claim?
The ideal picture you painted for yourself and them is a hark back to the days when you did not have to be concerned with all these. The pain you feel replaces the need for such claim. The pain itself may not be a bad thing if it is easier on your mind.
