Your concern is valid

But there is a solution: http://skloff.com/iras-avoid-double-taxation-on-withdrawals-080116/

Use IRS form 8606 for good record keeping. When you withdraw principal without earning, it is tax free.

when you covert it to Roth, you also need this contribution base to avoid tax.