Isabella Stewart Gardner was born in New York City on April 14, 1840. She was the child of David Stewart, of Scottish descent, who made his fortune in the Irish linen trade and later in mining investments. Her mother was Adelia Smith, descendant of Richard Smith, an Englishman who had settled in Boston in 1650. She was named for her beloved paternal grandmother, Isabella Tod Stewart, herself a remarkable woman and successful farmer. (A portrait of her hangs in the Museum.) Mrs. Gardner was also a descendant of royal Stuarts (although this genealogy is spurious) and took great pride in this lineage.

Isabella Stewart was educated at private schools in New York and Paris. Her first connection with Boston came through her schooling, between 1856 and 1858 in Paris, where a friendship with schoolmate Julia Gardner led to her eventual marriage to Julia's older brother John ("Jack") Lowell Gardner Jr. (1837-1898) on April 10, 1860. The couple was married in New York City and moved to Boston, Jack's hometown, where they settled into a house, a wedding gift from her father, at 152 Beacon Street in the Back Bay section of the city. In June 1863, Isabella Stewart Gardner gave birth to a son, John L. Gardner III, known as "Jackie." At just two years of age, Jackie died of pneumonia in March 1865, and during the two years that followed his death, Isabella Stewart Gardner endured depression and illness. At a doctor's suggestion, John Gardner took his wife to Europe to travel throughout Scandinavia, Russia, Vienna and Paris and, upon returning home, Isabella Stewart Gardner was in good health and spirits. Although the Gardners had no more children, they raised their three nephews following the death of Jack's widowed brother.