Not in Toronto but this building in East Peckham (Kent) England was in my family until the late 60s. It had 573 acres, 3 cottages, a gate house and a chapel. My great aunt sold it to the Harrier Krishna for only around 60,000 pounds. They abandoned it, George Harrison of the Beattle supposedly purchased it but it still sits neglected today. It had the ghost of a priest who hung himself it the attic that my Grandmother claims to have had many conversations with.
Also the grounds are supposed to be haunted by the Gate Keeper who was murdered protecting the property in the 1860's.
I wish it had stayed in the family and as the oldest Grandson it would have been willed to me with the stipulation I need to spend one night by myself alone to get it.
I only visited one time when I was 17 many many many years after it was no longer in the family.
Built in 1535
Roydon Hall, Kent by
Hugo Agogo, on Flickr
Roydon Hall by
Paul Grimwood, on Flickr