Carroll was born in Manhattan and raised in Forest Hills, a community of Queens, New York. After high school in 1942, he joined the Merchant Marines and worked on ships in the Atlantic. In 1946, he enrolled at the University of Montana to study English. While there, he became interested in theater. During one of the amateur productions, he met his future wife, Nancy Fields, whom he married in 1951.
He moved to Ireland where he continued his theatrical studies at the National University of Ireland. He was discovered during one of his college productions and was signed to appear at the Dublin Gate Theater. He worked in theater in Europe until 1954 when he returned to New York. His attempts to land on Broadway failed and he taught high school until 1958. Finally in 1958, he landed an Off-Broadway production, "Ulysses In Downtown".
He followed that with a production that was directed by Peter Bogdanovich. At the same time, he was getting attention on TV. He worked in a great many character roles throughout the 1960s. A pilot for "Those Were The Days" was first shot in 1968 based on the English hit, "Till Death Do Us Part", but was rejected by the networks. In 1971, it was re-shot and re-cast as "All in the Family" (1971) and the rest is history.