Nelson was born on 4 June 1932 in St. Louis, Missouri. His family was musical: his brother was a saxophonist who played with Cootie Williams in the 1940s, and his sister sang and played piano. Nelson began learning to play the piano when he was six, and started on the saxophone at eleven. From 1947 he played in territory bands around Saint Louis, before joining the Louis Jordan big band from 1950 to 1951, playing alto sax and arranging. After military service in the marines, he returned to Missouri to study music composition and theory at Washington and Lincoln Universities, graduating in 1958.

After graduation, Nelson moved to New York, playing with Erskine Hawkins and Wild Bill Davis, and working as the house arranger for the Apollo Theater in Harlem. He also played on the West Coast briefly with the Louie Bellson band in 1959, and in the same year began recording as leader with small groups. From 1960 to 1961 he played tenor sax with Quincy Jones, both in the U.S. and on tour in Europe.