Looking for Better Jitter Reduction in a DAC

I do indeed think you may well have a gain mismatch between the DAC and the amplifier. What you described is precisely how this would sound like: clipping of louder passages.
Part of the traditional criticism of digital comes from precisely this, as the 2V output for cd players as specified in the Red Book was too high for many amplifiers of the day. Hence Quad after the introduction of CD produced a special input board for their existing amplifiers. These days, there are still many amplifiers that cannot quite handle this signal level. In your case, the situation is aggravated because your DAC has a 2.8V output (what is your amplifier’s input sensitivity?). A telltale sign would be that you do not have to open the volume control very much to get a loud signal. Now you may ask why manufacturers would specify a DAC or CD player with an output level that is too high, or amplifiers with a high input sensitivity. The answer is that the human brain interprets louder as better. So in a demo room, the louder player/DAC or the more sensitive amplifier will always be interpreted as the better.
So how do you find out if this is the case (after all, the problem may be different from what I suggest, mine is only an hypothesis)? The first test is the volume control setting. If this suggests that you may have a gain mismatch issue, there are two options. The technical one is to have a look with a scope. Such clipping is easily visible. The second is to use a clean test tone and raise the volume until you hear trouble. The third one is to insert inline attenuators, and see if that improves the situation. If it does, you also have your (fortunately very cheap) solution.
See here for some demo material and discussion: http://www.harbeth.co.uk/usergroup/forum/the-science-of-audio/amplifier-matching-mismatching-and-cli...

Harshness could also be your power amp or your speakers.

@willemj 

Higher voltages on line level devices has nothing to do with making louder volumes to sound nice. (although loud does sound nice as you hear more low level detail as it lifts the low level sounds above the ambient noise floor)

Generally you only see high voltage levels on Class A+ and pro audio equipment. 

The use of higher voltage more expensive components in a line level device is to achieve better THD+N and better SNR. Better performance.

Consumer RCA = cheap crap that is popular because it is cheap manufacture and can often sound OK to non-discerning listeners

Pro Audio balanced is the gold standard = higher signal level and shielded conne tions with signal wires that arent grounded. Expensive and always equal or better than RCA. 

My Benchmark DAC 3 was recently testsd by Stereophile - it delivers 18 volts on XLR at max volume. This high cost high spec analog output circuitry is how Benchmark achieve market leading specifications.

I am not talking about absolute levels, but relative ones, i.e. a mismatch between output level of the source and input sensitivity of the amp. Pro audio is very good with a properly matching gain structure, but home hifi is often not. I agree about XLR.

It is entirely possible that the harshness could be from the speakers. Tuner and TT sound fine, but are of course lower than 2.8V output gain. I'm using Gallo floorstanders; 88dB, 8 ohms, a benign impedance curve between 4 and 8 ohms. Gallo uses a very revealing 180 degree radiating tweeter, piezoelectric, no crossover.
I have also tried some soft dome bookshelves; 88dB, 8 ohms, and they presented the same harshness or distortion in the highs from digital.

Preamp is Atma-Shere UV-1; Line stage output...12V, 17V RMS at clipping. Output impedance... 400 Ohms.
No specs for input sensitivity. It must be low since VC is at about 8 to 9 o'clock, 11 o'clock for TT. Ralph installed a 6 dB pad in the AUX input for a digital device.

I am now bypassing the preamp and DAC is going direct to amp. Amp is a TAD-60 (Tube Audio Design); Input Sensitivity is 1.0V with variable gain control, Input Impedance…100K ohms.
TAD recommends that the gain control be turned to max when using a preamp. I need to leave it at 12:00 on the dial.
The DAC has a fixed output.

@willemj , thanks for that very interesting Harbeth link.

So:
1 problem also exists with other speakers
2 problem disappears using sources with lower output levels
3 already powerful signal with VC at 8 or 9 o’clock
4 problem disappears when pre amp is out of the chain.

4 has the greatest practical importance. For those who want to know the cause of the problem, I think it is now almost certain that the input of the pre amplifier was clipping because the input sensitivity was too high/ the source too hot. If this analysis is indeed correct, and if you want to keep your pre amplifier in the chain, all you need to do is replace the 6 dB pad with a 12 or even 18 dB pad.
You may wel have stumbled on what I think was the reason digital had/has a bad reputation. It was not the quality of the signal, but the gain mismatch between the players’ output and the amplifiers that people were using. Are you happy with the sound you are using now?
I am not quite sure what your current gear does or does not do, but an outboard volume control (passive amplifier by an inappropriate name) could be useful, and if not for you maybe for others. Here is one: http://www.tcelectronic.com/level-pilot/