The Music Tapes started life as a creative outlet for the 16 year old Julian Koster (later of Neutral Milk Hotel), who would create tapes of songs and stories as means of escapism, which he would share with his friends, culminating in 'The American Phoam Rubber Co. Symphony Orchestra Proudly Presents the 2nd Silly Putty Symphony for Edison Wax Cylander', which was essentially tape collage. In 1989, Koster formed the band Chocolate U.S.A., a more conventional indie-pop band, which also included Bill Doss and Eric Harris, later (along with Koster, though somewhat unofficially) members of the Olivia Tremor Control. The band released two full albums, All Jets Are Gonna Fall Today and Smoke Machine, before splitting in 1994, after which Koster chose to focus his creative efforts on the Music Tapes once more.

In 1999 Julian released the album First Imaginary Symphony For Nomad, featuring a slew of musicians helping him, most notably Robbie Cucchiaro and Eric Harris.