In my opinion, in the majority of cases these days, the cause of harshness is the loudspeaker drivers and crossovers. Especially, when used with tube amplifiers.
Drivers exhibit rising impedance as frequency increases as a function of the voice coil inductance. This impedance rise will obviously be at its maximum right below the frequency where a driver is crossed over. At any rate, tube (and, solid state, according to Ralph Karsten) amplifiers prefer higher impedances. What this means is that they put more power into those higher impedances, resulting in increased (and, out of balance with the rest of the musical spectrum) volumes in those regions.
Given that the preponderance of loudspeakers these days are using a crossover between the midwoofer/midrange driver and tweeter in the presence range (upper midrange/lower treble), usually falling somewhere between 1500 Hz and 3000 Hz, which is precisely where the lion's share of complaints of brightness and harshness are centered, it all seems pretty elementary to me.
The classic way around this is a Zobel (resitor - capacitor in parallel with the respective driver) network in the loudspeaker crossover, which flattens the impedance rise due to voice coil inductance. However, in my opinion, the Zobel causes at least as many problems than it solves ala robbing music of immediacy and drama, in addition to making the loudspeaker that much more difficult for the partnering tube amplifier to drive. At any rate, since most current loudspeakers do not implement the Zobel (or, at least, in this manner), here we are with this issue being exhibited in the majority of high-end audio systems of the current times.