The name Rotsoort is derived from the family name Van Oort, who had their country house (built in 1702 on the souther parts of the current Pastoe grounds) decorated with blue-black stone. The blue-black stone was melted too hotly in the oven and was therefore unsaleable. The patent and production of imitation marble gave them great riches.
With the horse powered grinding mills, the top of the marbled floor tiles was polished extra smoothly. The first property, as of 1668, was homestead Rotswijk, of which the windmill was taken down (on the corner of the nowadays Rotsoord/Diamantweg, previously the Hoogravensedijk). In the direction of the city centre, another country house was built: Rotsenburg. The internationally renown painter Willem Roelofs, part of The Hague School movement, spent his youth here. Another famous resident was the inventor Abraham Keer, who took to the skies by hot air balloon as the second person in Europe. Looking from the Rijndijk (nowadays Jutfaseweg), the panorama of Rotsoort strongly resembled the country houses along the river de Vecht. The factories and working-class cottages were out of view.