Elder was born in Los Angeles and grew up in the city's Pico-Union and South Central areas. His father Randolph (1915–2011), who was born in Athens, Georgia, was a sergeant in the United States Marine Corps during World War IIand moved to California from Georgia after the war during the Second Great Migration.[1] After working as a janitor at Nabisco, Randolph opened a cafe in Pico-Union around 1962.[1] Following his father's passing in 2011, Larry Elder recalled: "Gruff and blunt, my dad often intimidated my two brothers and me. But we never doubted his love or his commitment to his family."[1] In 2013, Elder and his brother Kirk accepted a Congressional Gold Medal from U.S. Representative Dana Rohrabacher on their father's behalf.[2] Larry Elder's mother Viola (née Conley, 1924–2006) was originally from Toney, Alabama, and she was a clerical worker for the United States Department of War during World War II.[3]
An honors student who also took advanced courses at Fairfax High School, Elder graduated from Crenshaw High School in 1970 and earned his B.A.in political science in 1974 from Brown University. He then earned his J.D. from University of Michigan Law School in 1977.[4]