Yes, generally its grain, grittiness, forward high frequencies, as you say. But personally I also find, that anything in your setup, which keeps you from being drawn into the music, so that you are consciously or unconsciously forced to listen to your system instead of the music is eventually very fatiguing.
As far as speakers are concerned, I think we should know your setup and which kind of music you prefer.
Regards