How to assess and quantify system improvement and changes?

good luck with this one.

really! I wish you do figure it out, but the variables in this quest are innumerable.

the predominate issue is Audiophillia Nervousa is a self diagnosed disease. 

as such, EVERYTHING then becomes purely the beholder’s viewpoint or assessment on every account.

in fact, the. variable factors extend farther than the equipment itself. one can have the exact same gear in one setting, remove it from that venue, transport it to some other home, or setting, in some other likely dissimilar room, and the results will probably be perceived differently. further, merely purchasing either a single piece, or even a ’turn key’ setup you have auditioned at a dealership, and put it into your own arrangement , and chances are it will sound different.

another separating issue is simply the fact few if any outfits will have identical pieces in tow. how then, can a proper account of how ABC aids or detracts from, any particular rig? or for that matter how good is the ABC gizmo on its own merits  ever to be reviewed?

if the aforementioned items are not enough to frustrate this affair, add in our ears are not hearing things exactly the same way or as well, as other ears!

OK, then bring in the meters and micrometers. of course! but which meters? which rulers? which scales? how will this or that actually get measured?

then too, which words will. be used in surveying or appraising the thing? humans are barely able to communicate these days and suddenly we are all supposed to be crystal clear oh the piece, context, syntax, and findings?

synergy. is yet another factor which will ever detract from acquiring a status quo for quantifying performance in terms of gains and losses. its a proven fact, not everything ever made plays well with every other thing ever made.

I’m not even mentioning circumstances surrounding electrical power anomalies which vary  from one installation to another.  . that is another bag of worms all by itself.

in years past, telephones were in short supply. regularly, to talk to a friend or relative, you very often  had to  travel to their home. In so doing one experienced at least in one fashion, the ‘charm of distance’. 

back then distance was often a desired boon for its yield was greater privacy. 

privacy used to be highly valued, now everyone wants everyone else to know what they ate for breakfast, or where their cat  coughed up a fur ball. and when.  

in the often tail chasing hobby of higher end audio, the ‘charm’ lies in the preferences and perceptions of the one paying the bills to be the boss.  his or her insights on individual items, or as an ensemble are the only views that matter.

the only rule of thumb I’ve discovered which generally points to  performance, though not empirically, is simply the ‘cost of likewise  goods’.

gear makers on average have been quite studious on pricing their products, mostly.  subtracting those whose view points lend to them over valuing their equipment, tier pricing seems to equate to tier performance. IMHO

paying to get to some of these tiers of upper end  performance can  bring on many tears. gnashing of teeth and wringing of hands are other participants in the aftermath.  especially if system synergy is not elevated from the alteration or addition of the newly arriving  new bright and shiny gizmo

exactly where these supposed ‘tiers’ separations exist is yet one more facet that lends itself to the charm’ or chagrin, of this hobby.

bear in mind, the notation on the ‘self diagnosed’ theme. 

charming, isn’t it?

There's a fundamental problem here, as there's no benchmark that any of the judgments can be keyed to.  These are subjective judgments being made by individuals about unique combinations of components in unique listening spaces.  Try codifying that!

From a different perspective.  Replace a small, inexpensive 2-way with a large, high-quality floorstander.  More bass, right?, and better sound generally.  How much more bass?  How much does the increased bass enhance the overall listening experience?  (These are two different judgments.)

Then replace some throw-away 39c ICs with really expensive ICs.  Is there are much of a change?  Where?  In what areas do you hear it, and how do you quantify it?  Compare experiment 1 and 2.  Does the change in 2 equal that in 1?

Most improvements are incremental. People exaggerate, often without knowing they're doing it, to make a point.

A few years ago I dropped a pair of $200 speakers into a system that I'd previously been running with 5K speakers.  What struck me most was how decent, relatively speaking, they sounded.  No, not as good as the 5K, but as everyone always points out, not nearly 25x as good.

What I will say is that reviewers' hyperbole set people up for hearing more significant improvements than they actually achieve when they bring the component home.  Then the search for mitigating circumstances begins...