Yes, you’re right and it's true that any interest you pay on the loan goes back to your account. However, please don’t forget:
A) The interest paid on your 401k loan is not tax deductible.
B) You’re paying back the loan from your tax-deferred account with funds that have also been taxed.
Think about this, the money you are borrowing from your 401k is dollars that were put there before they were taxed, and they will therefore be taxed when you retire and/or they become distributions. However, the dollars you are repaying with have been taxed already, but will be taxed AGAIN with the rest of the money. Hence, you will be taxed TWICE on money used to repay a 401k loan.
C) The term of the loans, five years / 60 months, is relatively short when compared to alternatives such as personal loans.
D) As I mentioned before, one of the biggest risks with 401k loans is that if you lose your job, the balance of the loan becomes due and payable. If you can't cough up the lump sum that's due when this happens, you're on the hook for taxes and a 10% penalty, and you can never get that money back in your 401k.