Nestled in rolling farmland, the home was built in the late 1700s, with a major addition constructed in the 1860s. The mansion, built in Georgian and Italianate styles, boasts nearly ten thousand square feet with modern innovations of the time such as central heating and indoor plumbing.

White Hall State Historic Site was the home of Cassius Marcellus Clay, 19th century emancipationist, politician, newspaper publisher, Ambassador to Russia, and friend to Abraham Lincoln.

General Green Clay, a decorated soldier who served in both the Revolutionary War and the War of 1812, built the original house. Green Clay was a very successful land surveyor who in addition to vast land holdings, also had several lucrative businesses including distilleries, taverns and ferries. Green Clay built his home in 1798 and called his two story brick Georgian house Clermont. Green Clays youngest son Cassius later inherited the home and several hundred acres surrounding it.
Although raised by one of the wealthiest landowners and largest slaveholders in Kentucky, Cassius Clay did not approve of the institution of slavery. Cassius was born in Clermont in 1810 and was well educated, attending both Transylvania University in Kentucky and Yale University. It was while at Yale that Cassius heard the famous abolitionist William Lloyd Garrison speak. Garrison had a heavy impact on Clay, and as a result he devoted a great portion of his life to speaking out against the peculiar institution and fought for the gradual emancipation of slaves. Cassius views did not meet with approval in his hometown, but Cassius did not let widespread opinion deter him. He traveled widely to political rallies speaking out against slavery. At one such rally Clay met Abraham Lincoln, for whom he eventually campaigned. After Lincoln became president he appointed Cassius Clay Minister to Russia, a post Clay served in for two terms.

In 1833, Cassius Clay married Mary Jane Warfield Clay. The marriage lasted 45 years and produced 10 children. It was Mary Jane who supervised the remodeling of Clermont in the 1860s while Clay was overseas in Russia. Prominent architect Thomas Lewinski and builder John McMurty designed the new addition, which was Italianate in style and boasted such modern amenities as central heating and indoor plumbing. It was renamed White Hall.

Upon returning from Russia in 1869, Cassius and Mary Jane met with marital problems possibly brought on by a compilation of factors such as long years of separation, money strains, and rumors. The couple divorced in 1878. Cassius remarried a second time at the age of 84 to a 15 year old, a marriage that was considered scandalous and caused national headlines. The scandal did not last, as the couple divorced after only a few years of marriage.

Cassius resided in the mansion until his death in 1903, after which the house went up for auction. Cassius grandson Warfield Bennett bought the home and rented it out to tenant farmers, who resided in the home until the mid 1960s, after which the home was left vacant and open to vandalism. In 1968 the Bennett family donated the house to the State of Kentucky. Thanks to the concentrated efforts of First Lady Beulah Nunn, wife of Governor Louie B. Nunn, the Madison County Garden Club, and the Kentucky Department of Parks, the mansion was restored to its former glory and opened to the public in September 1971.