Thomas Wentworth, the 1st Earl of Strafford, designed the gardens with avenues of trees in what was, at the time, open countryside. In 1726, at the garden’s highest point, he had built the folly Stainborough Castle. Had it been built a few years earlier, it would have been the first such folly to be built in an English landscape garden.
In the central court of the castle he had placed his portrait statue (relocated, and believed now to be found near the front entrance to the college), and named its four towers for his children, William, Anne, Lucy, and Henrietta.
A folly is a building built for decoration, rather than the real purpose of the building it represents.