As a settlement, Holly Hill had several false starts beginning in the early 19th century. Spanish Governor Coppinger gave 4,500 acres of the area to Fernando de la Maza Arrendonda who then sold most of the property to one Thomas Fitch. Fitch then sold about 3,200 acres that land to William Samuel Fleming Sr. in 1817. Fleming abandoned the property during the Second Seminole War and the area was essentially devoid of people until after the Civil War.

William Wallace Ross arrived in the area sometime in the late 1860's and built a home at a place he named Palmetto Point. He was granted a post office there in 1868 but that office was discontinued in 1870. Ross and his brother-in-law put together an orange grove but that was destroyed in a freeze in the early 1870's.

William Samuel Fleming Sr. returned to the area and resumed his business in the early 1870's. In 1877, he sold a parcel of land to a family who became the first settlers in what is now officially Holly Hill. The first churches were built in 1883, the first school in 1885. Henry Flagler built the first train line through Holly Hill in 1887 and the town began to grow after that.

Holly Hill is located on the Indian River between Daytona Beach and Ormond Beach. Unlike the cities on either side, Holly Hill only includes a mainland section and not a piece of the barrier island across the Indian River. The population of Holly Hill has dropped almost 4% since 2000.