My son is 9, in 4th grade. His birthday is behind cut-off date.

My son is 9, in 4th grade. His birthday is behind cut-off date. He passed an assessment test to get into 1st grade.
He is always the youngest in the class. Many kids are 1 year older than him. He is actually proud of this, although he never speaks out.

Boys at this age are proud. They understand many things better than parents think. They care about their friends, their class, their school, even their town. Holding your son back one year will make him think he fails, he is not smart, you are not happy with him. He will lose his old friends. Maybe he will find it is hard to make new ones in a new surrounding. The negative emotion could lead to disaster.
Instead, if I were you, I would spend sometime with him working on things you thought he fell behind. 4th graders don't learn much from school academically.