Really bad idea.

You do not have much equity in your house (<20%).   Therefore, you always have an option to walk away if anything bad happens.  I assume you have no other asset.  (otherwise you willnot borrow from 401k.) If you loss your job, you have to pay back your 401k loan immediately.  How do you do that?  If the hosuing market goes down again in your area and you choose to walk away, you just loss you retirement in this house.

Never mixed your retirement money with other investment (non-retirement asset).  Those money is protected from most creditors (even in bankruptcy).  It is your last line of defense in your financial future.  Use it wisely.