You only have to pay gift tax if

your accumulated lifetime gift exceeds $1,000,000. The annual exemption is only used to calculate the lifetime gift amount, not the threshold to determine your tax liability. BTW, the current annual exemption is $13,000/person, not $12,000