Klein attended the High School of Art and Design in Manhattan and attended New York's Fashion Institute of Technology, leaving for six months after his first year before returning to finish his degree. He received an honorary doctorate from FIT in 2003. He did his apprenticeship in 1962 at an old line cloak-and-suit manufacturer, Dan Millstein, and spent five years designing at other New York City shops. In 1968, he launched his first company with his childhood best friend, Barry K. Schwartz.
He became a protégé of Baron de Gunzburg, who introduced him to the New York elite fashion scene before he had his first mainstream success with the launch of his first jeans line. He was immediately recognized for his talent after his first major showing at New York Fashion Week. He was hailed as the new Yves Saint Laurent, and was noted for his clean lines.
In 1974, Klein designed the tight-fitting signature jeans that went on to gross $200,000 in their first week of sales.