Krips was born into a Jewish family. He became a pupil of Eusebius Mandyczewski and Felix Weingartner. From 1921 to 1924, he was Weingartner's assistant at the Vienna Volksoper where he worked as repetiteur and chorus master. Afterwards he became conductor of several orchestras, including the orchestra in Karlsruhe from 1926 to 1933. In 1933 he returned to Vienna as conductor of the Volksoper. He also became a professor at the Vienna Academy of Fine Arts in 1935. He often conducted at the Salzburg Festival between 1935 and 1938.
In 1938, when the Nazi's took power in Austria, (the Anschluss), Krips had to leave the country because he was Jewish and Roman Catholic. He worked in Belgrade until Yugoslavia also became involved in World War II. For the rest of the war he worked in a food factory.
In 1945 he went back to Austria, where he was one of the few conductors who were allowed to work, because he had not worked under the Nazi regime. He was the first person after the war to conduct the Vienna Philharmonic and the Salzburg Festival where he conducted Mozart's opera Don Giovanni.
From 1950 to 1954 Krips was principal conductor of the London Symphony Orchestra. Afterwards he conducted the Buffalo Philharmonic Orchestra and the San Francisco Symphony (from 1963 to 1970). He gave his first performance at Covent Garden in 1963 and appeared at the Metropolitan Opera in 1966. From then on he often conducted there. In 1970, he became conductor of the Deutsche Oper in Berlin. Between 1970 and 1973, he was the principal conductor of the Vienna Symphony Orchestra.