If he needs a ride from you, let him take the lead, you just follow in silence. Let him have a chance to be fully responsible for the outcome.
By being anxious about being late, you are shouldering the burden for him. He might be addicted to the adrenaline rush created by your prodding. How old was he when this pattern started?
Let him feel the loneliness of having to take charge all by himself for a change.
(Not making a decision is also a decision. He may feel less blamed for the outcome, but he has paid a price in some other ways, even though he may not realize it until many years later, if ever.)
