While Westlake does have some of the dumpiest buildings in LA, it also has some of the most ornate. Some investment recently has cleaned up some of these prewar buildings, returning them to their former glory.
If you are in Westlake during the day, there is nothing to worry about. There are so many people about and the festival like atmosphere of street vendors makes it safe enough. At night, its best to avoid.
LA has many areas that are worse than Westlake (like Skid Row and parts of South LA). The difference is that Westlake has the added benefit of history that makes it a bit more interesting than those other areas.
Echo Park was pretty sketchy only 15 years ago -- proof that areas can improve for the better.
For anyone interested, here's a decent article on the history of MacArthur Park:
https://la.curbed.com/2017/1/26/1439...gentrification