Don't forget that the amount (say $50K) you borrow from your account are before tax money you use as if it is "after tax" dollars. So what's the double taxing coming from?
The biggest risk of borrowing from the 401K is that your money left the market and could miss a big run-up.
But it can go both ways. If the market is heading down big time, you could actually avoid a meltdown