Built in 1803, the grand Adam-style house was designed by Gabriel Manigault, a talented amateur architect, for his brother Joseph. The Charleston Museum purchased the house in 1933 and the Garden Club of South Carolina designated it as a project for restoration in 1947. The house opened to the public in 1949 and since that time The Garden Club of Charleston has maintained the large garden.

Each year at the beginning of December,  Club members decorate the exterior of the Joseph Manigault House and also each room with flower arrangements for Christmas.