He began art studies at the Cincinnati Art Academy at age 14. In 1882 he was a pupil of Charles Verlat in Antwerp; the following year he made his first trip to the West to sketch the Indian tribes of New Mexico, California, and the Columbia River.
Sharp taught at the Cincinnati Art Academy from 1892 until 1902, and then resigned to devote full time to painting. Summers were spent in Montana at Crow Agency in a cabin and studio at the foot of the Custer Battlefield. In 1893, Joseph Sharp first travelled to Taos, and his sketches from that trip were published in Harper's Weekly. He began making summer trips West to sketch Indians, and in 1902, he painted in Arizona, California, Wyoming, and Montana.
By 1912, Sharp was a permanent resident of Taos, living across from Kit Carson's homesite and painting many of the Pueblo Indians in their daily activities. Sharp continued to travel, going frequently to Hawaii and California during the winters where he completed numerous floral landscapes.
Sharp died in Pasadena, CA on Aug. 29, 1953.