As I understand it from three generations of my family, the inner-city post-WWII population decline was due to wholesale 'urban renewal.' Innermost residential areas that would have still existed in 1950 were subsequently bulldozed and replaced with college campuses and medical facilities, freeways and interchanges, commercial/industrial buildings, and 'capital city' stuff (government buildings, museums, plazas, etc.). For example, almost all residential areas immediately west of the heart of the city disappeared.
Compare this post-WWII aerial with a current Google Maps aerial, with a focus on the periphery of the CBD: