There are so many classical buildings with shitty proportions and poorly thought out details, or details that are completely oblivious to the location and context of the building, and yet they get a pass.
Maybe the problem isn't ugly buildings, but uneducated citizens?
Part of the reason we designate buildings as historic or protected is to teach people about the importance of the buildings. This is exactly why good examples of Brutalism must be preserved: To teach people about an architectural style that they know so little about.