The current system is essentially the spines of what was originally envisioned to be a sort of hub and spoke system.
The Blue and Orange Lines are the heavy rail subway lines, each originally meant to have several branches into various sections of the city as well as a feeder loop within Center City itself (The Red Line is to Philadelphia what PATH is to NYC and generally is not considered part of the city's subway system). As it is now, most trips in the city via transit involve a bus/subway transfer even for moderate distances. The ridership numbers are healthy but much lower than one would expect for a city this size.
In 1912 a comprehensive subway plan was presented to the city and small sections of it were constructed but thanks to the usual issues (funding, political will, greed) the rest never happened. SEPTA has, until recently, put the emphasis on suburban to city routings. Direct connections between various activity centers in the city are often indirect at best and cumbersome at worst, particularly depending on time of travel. This is not to say that it is hopelessly difficult to traverse Philadelphia via SEPTA but again it is more complex than it ought to be and the lack of a more extensive subway system is a big part of it.
This map, courtesy of Phillyskyline, depicts the original proposals:
There have been only reconstructions projects, slight route extensions and realignments since the 1950s. While this isn't terribly over par for other cities with systems of this age the fact that Philadelphia's subway never grew beyond the embryonic stage has really hurt the cohesiveness of Philadelphia. The economic and social geography of this city would be quite different today if even 1/2 of the original plan had been built.