After graduating from college, Clifford became a teacher at Woodland High School in Maine. He also gained his first coaching experience at the school, serving as their head coach for two seasons while leading them to two tournaments.[2][5] He then served as an assistant coach at St. Anselm’s College, Fairfield University, Boston University and Siena College. In 1995, he assumed the head coaching duties at Adelphi University and coached for four seasons under Keith Dickson, leading their team to four appearances in the NCAA Division II Tournament, an 86–36 (.705) record and four consecutive 20-win seasons; he was the first coach in the school's history with back-to-back 20-plus win seasons.[5]
Clifford became an NBA assistant coach with the New York Knicks and Houston Rockets under Jeff Van Gundy and quickly developed a reputation as a defensive expert. He then was an assistant for Stan Van Gundy with the Orlando Magic.[2] He considers both the Van Gundy brothers as mentors.[6] He reached the NBA Playoffs in each of his five seasons with Orlando, appearing in the NBA Finals in 2009.
Clifford then joined the Los Angeles Lakers in 2012–13 as an assistant.