almarg
7,946 posts
04-12-2018 1:19pm
I believe that the main point Mapman was making about attribution of differences that may be measured in the proposed test was simply that the measurement and comparison process should be repeated multiple times. That would presumably eliminate the possibilities that the measured differences, if any, are the result of imprecision or lack of repeatability in the test hardware and/or software itself, or are the result of extraneous variables such as differences in the warmup state of the equipment.
That makes a lot of sense to me, especially when the measured differences can be expected to be small.
Actually, repeating the test doesn’t necessary make it more valid. The same mistakes that affect the first one, assuming there are any mistakes, would affect the rest of them. That’s why “repeatability” means having a totally different test and test conductor. That should be pretty obvious. Otherwise unscrupulous or uncareful test conductors - even careful test conductors - could have unfortunate consequences. Mistakes in a system may be beyond the control of the test conductor. So, any test result is only a single data point. It has little meaning if the test results are negative because - as has been pointed out a number of times - there are many reasons why a test might fail to show differences, or audibility or wherever was being tested when in fact there are.
7,946 posts
04-12-2018 1:19pm
I believe that the main point Mapman was making about attribution of differences that may be measured in the proposed test was simply that the measurement and comparison process should be repeated multiple times. That would presumably eliminate the possibilities that the measured differences, if any, are the result of imprecision or lack of repeatability in the test hardware and/or software itself, or are the result of extraneous variables such as differences in the warmup state of the equipment.
That makes a lot of sense to me, especially when the measured differences can be expected to be small.
Actually, repeating the test doesn’t necessary make it more valid. The same mistakes that affect the first one, assuming there are any mistakes, would affect the rest of them. That’s why “repeatability” means having a totally different test and test conductor. That should be pretty obvious. Otherwise unscrupulous or uncareful test conductors - even careful test conductors - could have unfortunate consequences. Mistakes in a system may be beyond the control of the test conductor. So, any test result is only a single data point. It has little meaning if the test results are negative because - as has been pointed out a number of times - there are many reasons why a test might fail to show differences, or audibility or wherever was being tested when in fact there are.