So, the best is, refi your primary to a lower rate, then buy a bigger house to get primary rate again(If you want to switch to a bigger house originally)