You need to file Form 8606 Nondeductible IRAs for the pro-rated

The only way to avoid this situation is to rollback all your other rollover IRAs (from pre-tax 401k) to your current 401k plan, leaving only nondeductible IRAs (100% nondeductible). Otherwise, you will need to do this form every year untill you empty all your traditional IRA. It's actually not that bad if you start doing this and Tax software will keep your updated ratio every year.