The interest shall go against your primary property

The closing cost shall not be able to write off (this is my opinion). But, there is only 1% of the chance the IRS will tell you to do otherwise and you have a good explanation (though may not correct) to cover yourself and worth a try.