Move the mortgage to your primary residence is one way to achieve that. By doing this, you don't have this problem.