In essence it depends on what the goal of your pages is :
i would prefer 'can be harmful' ... say i'm creating an audiovisual-experience like a game that works in a browser, the use of labels would be detrimental to the layout and people with visual impairment (or others) might not be able to enjoy the game anyway. Downplaying the game to a point where anyone with any impairment can enjoy it would require me to have a team that consists of at least one person with one impairment of each type to make sure everyone can have the right experience. Most of it would be cut out in most places to adapt and the whole experience would change so.
I think this is something that comes at your own discretion depending on what you want to achieve , but this certainly answers my question on the matter