I've been a tourist to all 3 of those cities. I haven't been to Asheville since the 1990s though.
Portland doesn't really struggle with the crowds though, it's easy to fly in to Portland, or drive up from Boston, or take the train from Boston. It's the points north of Portland that struggle.
Mid-Coast Maine in the summer is pretty awful, with heavy traffic all piled up on a 2 lane highway - US 1.
https://goo.gl/maps/c3B7vuzBV4J6UNzd6