[Capsicum]

Isn't that the logic behind the term "visible" minority, as opposed to minorities that are linguistic, religious etc. that are not based on physical appearance (unless they choose to dress differently).

In the US, though, minority without qualifier (as "visible" in the Canadian case) seems to default to mean racial minority, though things like religion, sexual orientation etc. are often included too in discussions of diversity.