Ellis shortly ran out of funds, dropping out of school in 1941 to tour with the
Soon thereafter, Ellis got his first big break, joining the Tommy Dorsey band, where he remained until 1947. It was this with this band that Ellis made his first recordings. Another break followed in 1953, when he replaced Barney Kessel in the Oscar Peterson Trio. The remained until 1958, longer than any other guitarist. It is in the trio that Ellis would meet bassist Ray Brown, someone who would turn out to be a long term friend and musical collaborator. As in Kessel’s career, Ellis’ position in the trio was somewhat controversial since he was white.
Oscar Peterson’s trio had, by this time, become a significant part of the house band for Norman Granz’s Verve Records, and the work afforded Ellis the chance to play and record with some of the most important musicians of the time including Ben Webster, Stan Getz, Dizzy Gillespie, Roy Eldridge, Sweets Edison, Buddy Rich, Ella Fitzgerald, and Louis Armstrong. From 1958 to 1960, Ellis toured with Fitzgerald.
His earliest musical influence was Jimmy Giuffre. His musical direction was cemented forever by hearing Charlie Christian. Ellis recounted in an interview with Leonard Feather Christian’s impact: "... the first time I heard Charlie Christian I thought he really wasn't so much, because I felt I could play faster than that. Then after a few more times it really hit me, and I realized that speed wasn't everything. I got quite emotional -- put my guitar away and said I'd never play again. But the next day I got it out and started to try to play like Charlie." Christian’s influence is everywhere evident in Ellis’ playing to this day, but also so are the blues and Texas Swing he heard growing up.
Ellis is still very much alive.