VAC - any comments? Good or bad.

Lrsky, one could make that argument for any component. All we can offer is our impressions based upon our experiences.

To reiterate, ask any manufacturer about the validity of this experiment. I am sure it was done with good intentions, but it is flawed, not because I personally think so. Call ANY electronics manufacturer and ask. PLEASE DO NOT TAKE MY INPUT SERIOUSLY. I am just one of the guys.
Ask Dave Gordon at ARC, Bill Conrad of conrad johnson, Dan D'Agistino of Krell, Jeff Rowland. Need I go on?
If it was important enough for Subaru to go to all that time and trouble to do this in the first place, surely he won't mind getting input from the actual manufacturers of the gear, who try so desparately to create these products, why this form of testing is invalid.
They WILL be forthcoming if you just ask.
Best,
Lrsky

Lrsky, we're way past your suggestion. MANY manufacturers of audio components devise simplified controlled test methodologies to evaluate product tweaks and of course during ongoing manufacturing QC. You don't have to reinvent the wheel every time you want to look at a performance criterion. I spent a former professional life devising test methodology for laboratory equipment evaluation and manufacturing processes, culminating in chairing ASTM/ANSI and ISO subcommittees in the late 80s. The subsets centered around piston-operated volumetric-ware: not unlike tiny "tweeters" for liquids, currently used in all labs tosample, measure and move around small (1 uL - 1ml) aliquots of liquids hither and yon. You see these little pipettors (The "Pipetman" is the one I co-invented) on TV as reportes think they're camera-friendly for the general public....
The point is that there is NO DIRECT WAY to measure a uL of volume in a short amount of time! You have to rely on an indirect means, perhaps such as radiometrics, spectrophotometry, or im my developed expertise, gravimetry of water. By knowing a LOT about what happens when you move volumetrically and then weigh tiny amounts of water drops you can very noiselessly measure true volume. That's how nearly ALL the volumetric lab equipment in everly lab in the world was calibrated. And I wrote the friggin methodology. Now doing the ALL the reference measurements for imprecision and bias (barometric pressure, temperature, evaporative blanking, time-clock matching, microgram balance calibrating, operator bias calibrating (these are often hand-held devices), and a few I've plumb forgotten, for EACH string of measurements would be preposterously inefficient. Especially for $100-500 hand-tools. So manufacturers have devise highly-controlled procedures (or at least that's what my publications were supposed to teach them to do!) to shorten these "controls" to only several minutes per device. It was this kind of atmosphere in the late 80s in Geneva that spawned ISO9000/1/2etc. Unfortunately that's become more of a paper-cover set of machinatuions rather than necessarily a raising of quality level. But I digress.
It onlly takes a cursory reading of a few back issues of Speaker Builder et al to uncover manufacturers who've used simple SPL meters in real-world acoustic setups to uncover non-linearities in switched-component analyses via differential testing. I casually mentioned my testing to a chief designer (ex-KEF, now Boston Audio), as well an ex- BBN master acoustician, who implicitly trusted the soundness of matched-reference technique analysis for uncovering non-linearity of a suspected "unorthodox behavior of a gain device". Indeed, when I mentioned my results to VAC's folks, after a cursory description of technique, their concern was NOT my procedure, or its validity, but the degree of nonlinearity of the results, which they said again seemed surprising, even for the admittedly unruly output stage of the AVATAR.
So Lrsky, I kindly suggest that you step back a bit here, as my technical prowess as organizing a valid scientific inquiry is not truly in question...I've got the backing to pass ANY scrutiny of methodology despite not having conventional DIRECT high performance traditional electrical instrumentation, and please not be coy about summoning the industry gods to pass judgement on what anyone of reasonable facility with the scientific method can discern is a well-run set of differential tests. As well, the amplitude of the non-linearity difference to noise ratio is VERY high (although not calculated), so my experience tells me I'm on solid ground with the stat calcs too.
But again, rather than pee back at you, I'd rather educate.
What you simply do NOT understand is that calibrated speakers, room, mic, cable, etc., are NOT needed to run this type of test, as these items are held STABLE through all test runs. Repeat runs (controls) proved that system imprecision was extremely small (which I think you may not have a problem with); indeed the raw data is NOT flat in dB across the 27 or so test frequency points for any situation, because of ANY and ALL of the effects of the system. But they are STABLE as a rock (well within 0.5dB).
Again, the biggest sources of imprecicion are human headmovements and parallax error in eyesight.
This test can, and has been used validly by INNUMERABLE MANUFACTURERS to assess changes in one variable at a time at ANY component position in the system chain. I think that's part of what you don't get.... Whereas there is NO VALIDITY in quoting actual raw data in dB vs frequency because of no pure reference scale calibrations (that and efficiency are the reasons why I didn't post it); because of the stable RELATIVE REFERENCE, there can be great validity in concluding statistical significance from differential calculation of this "humpy data".

Part II (Sorry!)
I guess another part of this is that manufacturers generally use resistive loads for amplifier measurement, wherein linearity usually appears reasonably flat. We're now used to seeing a curve now and again from some testers (like Atkinson at Stereophile) who throw in a "projected behaviour with an 8-ohm real speaker" curve, which, especially with a high output impedence amplifier, will show some non-linear frequency dependent behaviour.
In a way, there's some useful "single-blindness" in having very raw data. Indeed, after running the first SS1 vs VAC test, one can't tell by looking at the raw data anything.
I then ran the second test: SS2 vs VAC. A cursory examination of the two sets showed that SS1 very nearly equalled SS2, and that VAC very nearly equalled VAC. I then tried SS3 by itself, which equalled the two SS. The amplitude of difference between the VAC and the three SS was quite large (see the curves again), and correlated nicely with the gross differences in sound differences. Certainly the three SS amps had differences in sound, but these had to do with grittiness, edginess, decay, dimensionality, and all that typical SS stuff. The Audio refinement Complete Integrated clearly sounded the best. The Acurus and NAD I didn't like much. But ALL THREE SS AMPS sounded TIMBRALLY EQUAL, and indeed measured spectrally the same. It all makes sense. (Now do you understand, Balekan?) The three SS curves are very tight (intertest imprecision mainly given only by pink noise bias difference set by gain control). The two VAC curves are quite tight, only being a bit at variance at the 16k and 20k points where both are at highest slope, and therefore coarsest measurement...but also greatest non-linearity--quite wild rides....
The only almost-sily fly in the soup here is that someone could actually postulate that since the raw data is relative to an unknown (yet stable) reference base, isn't it possible that it's the VAC that is linear, and that the THREE SS AMPS are all wrong? Yup. Entirely possible. And if there was only one SS amp I'd have to agree, and would never have published these results. But when THREE different SS amps agree so highly, and one TUBE amp disagrees so greatly, TWICE, and sounds horrible, to boot, my money's on something terribly wrong with that output stage...when mated with the supposedly benign Fidelios. Ern

Ern,
"...and please not be coy about summoning the industry gods to pass judgement on what anyone of reasonable facility with the scientific method can discern is a well-run set of differential tests..."
Why would you refer to industry "experts", as "Gods" in such a sarcastic manner? What ax are you grinding, and why?
My comments were aimed at the potential of some possible unusual load of the the speaker, and possibly room effects to skew any results, and that if you used a host of speakers, that the test results would be different with each differing reactive, rather than resistive load.
There are numerous questions, but a few come to mind. Did you use calibrated mic's? Have you previously plotted the room for additive frequency anomolies, or did you do as most real qualified scientists do, and use an anechoic chamber, or a faux chamber, designed with the flaws input into the response curve of the room,in order to factor those errors out? Again, I would ask if you use more than one set of speakers, if so were the results of the amps exactly the same?
No one is peeing here, I simply said I thought you did the tests with sincerity, but that I doubted the validity because the reaction of only one set of speakers, with one Avatar amp, and no pre test,test data on it, to compare with others of its ilk,(meaning, was the amp tested, and confirmed to be in perfect, or its intended operating condition and certified to be so?).
Speakers present reactive not resistive loads, as everyone knows, hence, tests of an amplifiers output, is done with various resistors, (still flawed) yes but equally so, since this test is still into resistors, even though of varying values.
Cable capacitance, inductance, any of that factored in, or would you say that this doesn't make any difference whatsoever in your tests?
Why are you buying such flawed gear? If you have doubts about a product and its output, why not try before you buy; why lie in the bushes (metaphorically) and ambush any manufacturer with such (imperfect, incomplete) testing. Even though I have written many press releases that were released nationally, that does not make me Ernest Hemingway.
And just so you'll know, even though the folks at VAC didn't really call you to task on this test, I know personally, that a coffee mug was offered as a prize to anyone who could explain your tests, and how they relate to real world.
The implication of your test is that the Avatar is flawed in its output, and that Solid State amps are not. I don't believe that, as a narrow finding, that this is correct. I don't know, and neither does anyone reading this, that the Avatar in question was working correctly when you "tested" it.
Also, a little presumptious to think you can educate someone you don't know, and even offensive based on this writing you give as support for that claim.
If only Kevin Hayes weren't so concerned about public image he would join this discussion and "enlighten" you, as to the variables which are not dealt with or even mentioned in your diatribe.
Again, I am only one person. Ask any (and they aren't God's that is insulting) industry expert how valid these tests are, don't ask or try to convince me.
By the way, what audio products have you designed? How well do they sell? With your obvious tallent you should teach all these manufacturers the error of their ways. You seem to be a disgruntled, anti audio person, on some not so obvious quest to discredit VAC. If that is the case, I suggest you move on, sell it and forget it. If not, please be more respectful.