Simple.

His ex still has a spot deep down in his heart. Taht is why this date is very special. It is a date that reminds him of good and bad times.

Let it go. Pick a different date and make it your special one with him.