Here in 1797, Elizabeth Ten Broeck and her husband Abraham raised this house on land leased from her brother, the patroon Stephen Van Rensselaer. At the time this parcel was in the town of Watervliet, just north of the city of Albany. Originally called Prospect and built in the restrained Federal style, the house would hold Abraham for only twelve years before his death in 1810; his widow Elizabeth lived here for another three years until her death.
Over the following thirty years the building took on Greek‑revival details as tastes changed. In 1848 Thomas Worth Olcott bought the place, renamed it Arbor Hill (Arbour Hill), and added a first‑floor butler’s pantry and second‑floor bathrooms — Victorian touches that speak to the late 19th century. The mansion’s national recognition came when it was listed on the National Register of Historic Places in 1971, and a decade later it was included as a contributing property to the Arbor Hill Historic District–Ten Broeck Triangle when that neighborhood was listed on the Register.
The butler's pantry and second-floor bathrooms Thomas Worth Olcott added in 1848 sit inside walls Elizabeth and Abraham Ten Broeck raised in 1797.