Correction. You can qualify. See the answers here

Ownership of a vacation home or rental property not used as a principal residence does not disqualify a buyer as a first-time home buyer.

But you have to provide the proof that you did not live in there as primary anytime.