Someone calling you a "racist" does not make you one.

 

His behavior indicates that he has an "entitled" mentality.

In the situation you described, you did the right thing. No need to apologise. Better yet, you should be alarmed and annoyed by such imposition.