Clever Little Clock - high-end audio insanity?

Zaikesman and Tonnesen, as you know, tests of statistical significance are sensitive to how big the sample is. With a sample of 25,000, any relationship will prove statistically significant. Since we are all too willing to say that a relationship that is statistically significant is also significant, we are in danger of saying as you think I am saying that you can prove anything with statistical significance tests. These tests were developed to answer the simple question of whether an unusual random sample from a population where there was no difference could have given us sample results where there is a difference.

In the tests that you both propose as to whether subject hear or don't hear the CLC is present, a large sample of say 10,000 would achieve statistical significance even were there no difference, although I would not predict in which direction, such as whether the CLS helped or hurt.

I am not being anti-science or anti-logic, I am merely saying that such tests may not be a valid method to prove or disprove whether the CLC does anything. I am also saying that those who claim it does nothing cannot claim the high ground by saying that those hearing a difference are delusional as those hearing no difference may also be affected by prior conceptions.

Tbg: I'm still not following your point about sample size, but it sounds like you are saying that it is possible to manipulate statistical methods to prove anything one wants to by the way one chooses the sample size...?

If the effect of the CLC were profound, as some people claim, you could prove this with a very high level of confidence with a very small sample size - there is a 1 in a million probability that someone could correctly guess, purely by chance, whether the CLC was in the house in 21 tries.

A larger sample size is only required if the positive effect is very small, in which case you need a large sample size to show that the small benefit is statistically significant.

I expect that 10 tests would be enough to convince most people that this device does not have much benefit, and a larger number of test would only show with increasing certainty that it has no benefit at all.

Tonnesen, you are actually arguing against statistical significance but are right. If you get a very large random sample of people, there will be statistically significant differences heard between with and without the CLC. But you are wrong that 10 subjects should be enough to convince people that the device has no effect. If you had 10 non-randomly chosen individuals with "good ears'" you might well question whether their hearing a difference can be generalized to all listeners. Similarly, were you to have 10 who doubt the benefit, others might well legitimately question your findings. Even with 10 randomly chosen individuals much would depend on the strength of the treatment effect.

I am not arguing that one should not attempt such tests, but I am arguing that they may not necessitate others heeding them as proof that the CLC does nothing.

Tbg, I'm afraid I hope you're more clear (and relevant, I dare say) with your students...

From what I can tell though, you've got at least one thing basically wrong there:

"I am merely saying that such tests may not be a valid method to prove or disprove whether the CLC does anything"

To me this isn't correct. If enough trials are run, with the clock randomly inserted or removed from the listening environment without the subject seeing which condition it is (I think at least 30 trials would be preferable, which could be divided between 2 or 3 different sessions on different days), its absence or presence should be correctly reported at a rate significantly higher than just 50/50 chance if it's actually doing anything like what the believers maintain. If on the other hand the results hewed pretty close an even 50/50 split, it would be strongly indicative that nothing is audible.

This is a different question than whether whatever the clock may do is 'good' or 'bad', which is a subjective judgement, and not important as to whether the thing can be *detected*. For illustration, I just flipped a coin 30 times, twice. The first set of 30 I got a 17/13 split, the second set a 16/14 split. Whether the splits favored heads or tails is unimportant (as it happens, it was one of each), the relevant point is that my splits only deviated from the mean of 15 by 1 or 2, strongly indicating random outcomes.

Oh, you're talking about random choosing of listeners? I don't think this is necessary, or even desirable. I'd rather limit the listener pool to audiophiles who believe in the audibility of these tweaks.