I lived in New Haven in 2004-2005, and the area around Downtown has only gotten better since then. It always had a great restaurant scene due to Yale, but there are now two grocery stores downtown. Broadway is not the only student commercial area. Chapel Street is a few blocks to the south, and has a great selection of restaurants. Once you head west of Dwight Street (either on Chapel or Broadway) it basically turns to ghetto though.
New Haven is pretty unique for a Connecticut city as it maintained two middle class urban neighborhoods directly adjacent to Downtown. East Rock has always been a middle class enclave with a great local public school - very popular with Yale professors. It has its own walkable business district on State Street. Wooster Square was the traditional Italian neighborhood and has gentrified in recent years. I always loved the
Court Street rowhouses, which are (AFAIK) the only intact block of rowhouses outside of Boston in New England.