Longtime collaborator with Rock n Roll Hall of Famer Patti Smith since the early 1970's, Kaye has been the lead guitarist in the Patti Smith Group, and played key roles in her career since she released her debut LP Horses in 1975. In addition to Patti Smith, Kaye has collaborated in the studio with such artists as Suzanne Vega, Jim Carroll, Soul Asylum, Kristen Hersh of Throwing Muses, and late Beat poet Allen Ginsberg.

Kaye also has been a music writer, involved in compiling the seminal 60's garage rock LP reissue "Nuggets" for Elektra in 1972, Kaye wrote the liner notes, making one of earliest references to the term 'punk rock'. Kaye has written music realted books including bios of 1930's crooner Russ Colombo and worked closely with the late Waylon Jennings on his autobiography.