In addition to the history of cultural stereotyping, which certainly plays a role, it’s the comparison of entire sound systems that produces a phonaesthetic character in the mind of the listener: the sounds present, the way they’re combined, the intonational phrasing, and the rate of speech—plus a number of sociological factors.
Source: wiktionary