You are probably right.

I had a similer experience, and the difference was that I didn't leave the job.

The hiring manager even called my supervisor but my supervisor told the hiring manager that he was not allowed to give any reference, however, the hiring manager could call HR to verify the validit of the start--end date. The hiring manager didn't go further since she got the other two references back with very positive comments.