For comparison / contrast, here's some information about the areas demolished in South Parkdale for the Gardiner in 1956-58. The area was already pretty cut off from the rest of the neighbourhood by the rail tracks, and had seen significant demolition when Lakeshore Blvd was constructed in the 30s (I believe). About 150 houses demolished in total, at least some of which were more like cottages or considered substandard at the time. Other than this very little had to be destroyed for the Gardiner, as it generally threaded through mostly vacant industrial land created when the harbour was filled in.
All images from the Lost Toronto blog http://lost-toronto.blogspot.ca/2010_07_20_archive.html
The area before construction of Lakeshore, with houses to the south of the rail corridor.
In process of demolition in 1956 - you can see where the Lakeshore pushed through earlier. Probably would not have been a great area to live.