后续，contact with the broker for refinance found that

because the tri-level house value in our area go down, so we are ask to buy mortgage insurance for the refinance, it will not worth to do that.

We paid too much for the house even in a buyers market.