Likely culprit for harshness at higher volumes?

Almarg's suggestion about headphones may apply here. Anthem says the phone output is "in parallel with main path". I take this to mean it is the preamp circuit. Many components that offer phone output use a separate cheap circuit which would not reflect the sound you get from your speakers.

Saru - There are differing views on jitter, and digital in general, that go to the extremes on both end. This is just another example of an audio topic where you need to be open to the idea that very few of us are correct about everything and the truth is likely somewhere in between the extremes. Some will argue that digital comes down to a 0 and a 1 in the signal and quality only matters once it becomes an analog signal and others will spend thousands on digital components and such. Keep an open mind.

The guys at my local Hi Fi dealer were always telling me to ignor the specs and just listen. The owner gave me some specific examples of equipment that had specs that were the opposite of what you'd expect based on the sound. I'm an engineer so I like specs, but like all metrics they can be made to say whatever the manufacture thinks you want to see.

Check temp at idle, not while playing music since program material varies from one side to the other. Let it sit for an hour. DC offset is easy. You need a multi-meter. Preamp set on anything other than phono, volume at o, speakers disconnected from amp. Set your meter at 2v or preferably lower if possible, ie: 200mv, probes on speaker terminals, red on pos, black on neg. Ideally, it should read o. Anything up to 50mv is acceptable, however any decent amp should be in the 20mv or lower region on each side. It's okay if they vary within this range. A fast amp is going to have a rise time of at least 1.5u seconds(micro-seconds), and a slewing rate of at least 40volts per u second. Linearity has to do with timing of each frequency as it reaches your ears. Flat response is just that, equal amplitude of all frequencies. Not all amps' DC offset is adjustable. Some have potentiometers for this purpose in which case it is, some have DC servo circuits, and some have discreet components that control this, in which cases it is not. Without experience, I would not attempt to adjust, but simply check and double check. If it is suspect, and typically could very well be, you need to take it to a reputable tech and have it looked at. O DC offset results in a voice coil resting where it should, in it's mid position. The farther off it is, either + or -, the more distortion the speakers will produce. Most amps have potentiometers to adjust bias or in other words, quiescent current. This procedure is more involved except to say 150ma is optimum per ch.. TNT audio has a great article on this which I've read and followed. Just Google "adjust the bias of your amp". I currently have some Bedinis, Ampzillas, Leach Low Tim, and Meridian amps. They're all great amps, but the best of the lot are the Leach amps. Interestingly enough, they are the ones that have no vested interest to compete and they are the only amps I've heard that "disappear".

Left: 7.2mV; Right 13.2mV. Same reading with volume at -95db (minimum) and 0db (wasn't sure what you meant by 'volume at 0'). So, I'm assuming that the DC offset on my amp is acceptable, even though left and right are a little different?

I don't really have any idea how to measure fast and linear but I assume that's not something adjustable (w/out changing internal components?) so I have what I have. I'm not even sure how to adjust for DC offset so it's a good thing it's within the range you suggested. :)

Planning to order the Sony headphones recommended by almarg when I get the money.....

Volume at minimum. Yes, very acceptable. Without bench equipment you cannot get specs for speed, linearity or frequency response but the best way to check those aspects is with your ears. It's really those characteristics we're dealing with when we judge and come to a conclusion on how an amp sounds, hence my original suspicion your amp is having difficulty with transient performance, referring to it's "speed". The poorer the recording, the more difficult it is for an amp to reproduce especially vocal transients ie: s, z, ch, j, v. Probably because one of the main ways to describe a poor recording is compression when the transients become very fine and close together. This can result in some pretty harsh sounds when the amp can't deal with them effectively.