There were several double bass players in his family, but he did not meet them. He studied with several teachers. He went to the Juilliard School. In 1962 he became famous when he was a soloist in a nationally televised New York Philharmonic Young People's Concert, conducted by Leonard Bernstein. On that famous telecast, Karr performed "The Swan" from Carnival of the Animals by Camille Saint-Saëns.
Gary Karr has given performances in many parts of the world. He has published a number of books for learning to play the double bass. After 40 years as a concert artist he retired in 2001 to Victoria, British Columbia, where he lives with his dog Shin-Ju.
In 1967, Karr founded the International Society of Bassists (ISB), an organization which encourages the playing of the double bass all over the world. Every two years the ISB has a big international conference.
Olga Koussevitzky, widow of the conductor and double bass player Serge Koussevitzky (1874-1951) heard Karr play when he was young, and gave him the beautiful Amati double bass made in 1611 which her husband had played. Karr later gave this double bass to the ISB so that double bassists world-wide could play on it.
In 1984 the Karr Doublebass Foundation was established by Karr, which loans instruments to promising young double bassists to help them in their careers.