• David Colquhoun supplied calculated beyesian confidence limits for the Which? study on 17th, This calculation assumes and requires the sample to be random.
• Expert reviewer and dietitian Catherine Collins confirmed the sampling was random on 18th January but giving no details of methods used.
• Shefalee Loth (Which? staffer) confirmed the sampling was random, also on the 18th, again giving no details of sampling methodology.
• Around mid-day on 18th January we see the first signs of backtracking. Avoiding the direct issue of sampling methodology and without confirming whether the sample was random or not, Colquhoun states that the expert panel had ‘nothing to do with selection’.
• Finally on the 19th January we see Colquhoun’s acceptance that the ‘the sample was not a random sample of all nutritional therapists.’ (Posted 19th January 1:26am)

At the time of writing we see that of the three expert reviewers, one reviewer (Colquhoun) says its definitely not random sampling. Another reviewer (Collins) says it definitely is random sampling, and the third reviewer Margaret McCartney is wisely keeping very quiet on the subject.

In David Colquhoun’s excellent published book on the subject (‘Lectures On Bio-statistics’ p16) he explains in his own words, why randomness is fundamental:“Any sort of statistical analysis (and any sort of intuitive analysis) of observations depends on random selection and allocation having been properly done.”

Looking more deeply into what we know about the random (or not random) sampling methodology actually used in the study: there is little information available other than this offering from Colquhoun’s discussion comments on 19th January:

“I understand that Which? staff selected the therapists in a way that mimicked what a present day consumer might do. They used google and online directories, and chose the therapists that were at the top of the search results. Thus, the sample was not a random sample of all nutritional therapists, but would be like the therapists that were chosen by googling. It would be biased towards the more popular (and therefore perhaps better?) therapists. It is a worry that less popular therapists might have given worse advice than was found in this case.”

That any scientist, especially a dietitian such as Catherine Collins could ever consider this as a random selection is interesting. Colquhoun concedes it is not random, then introduces as a fact the unsupported conjecture that the study now becomes biased in favour of nutritionists. It’s not random any more – it’s now better than random! Colquhoun is also clearly saying that clinical competence of nutritional therapists declines with your google search position.*(see note). One waits to hear more of this ground-breaking discovery in the scientific press.

So how to make sense of all this? It is probable that, at the start of the Which? project, the reviewers either assumed or were told by Which? that the sample was random. Some time after publication, the truth emerges which destroys the entire legitimacy of the report’s findings. The expert panel are in disagreement with each other as to the validity of the sampling methodology used, and now the Which? discussion board has been closed down (now re-opened).

In summary, any conclusions drawn from a non-random study are simply meaningless. They cannot be relied on to suggest anything at all about the population under investigation. The Which? report can therefore be revealed as a study whose sole basis for legitimacy is the stridency of its own conclusions (see Salem Witch Trials).

Clearly, the panel of experts should all now disassociate themselves from the Which? report and its conclusions. Which? should also retract the report; whether it does or not may well depend on the weight of any defamation actions. Which? should at least acknowledge that a selection methodology more suitable to assessing the relative merits of washing machines should never be applied to a study of this type.

Note:
*The raft of problems associated with this statement and this type of sample selection are worthy of much discussion, but not here! As a taster, the notion that google listings are necessarily biased towards popularity is generally false, (confusion of cause and effect), the association of practitioner competence with google ranking is unsubstantiated and patently absurd, search results differ depending on the geo-location of the initial google enquiry, which search terms were used, and so on…

So the first time I just asked what was the chance of picking one therapist out of 1805 and then picking the same one again. This is the 0.06%. This is wrong because you don’t just pick two, you have another 14 chances to pick the same one.

So the next time I simply multiplied 0.06 by 14 to account for the extra chances to pick the same practitioner. This comes to 0.8%. However this is also wrong as it is still the chance of picking that one initial therapist again, not all the others that have also been picked in subsequent selections.

The correct calculation (I think – please correct me if I’m wrong again) is 1/1805 + 2/1805 + 3/1805 and so on all the way up to + 14/1805. This is quite a lot higher again at 5.8%

So (pending further correction!) we can say is there around a 6% chance that the same practitioner could be picked twice. Apologies for my initial bungling.

Despite this, I still think a value of 6% makes it rather unlikely that this would have occurred. I look forward to confirmation of methodology from Which? at some point. I woulds argue though, that a study of this kind would be better suited to random selection without replacement (i.e. the same practitioner cannot be picked twice) to avoid the consequent effects of pseudoreplication.

‘the chance is actually 0.8% (not 0.06%, that is for two in a row not two out of 15)…’ Will you please explain how you calculated this? Just curious…thanks!

Secondly, the practice of medicine has become over dependent on technology. People have been promised improved clinical care where MRI scans, ultrasound, laser and all the other new innovatory tools that we believe are so indispensable, allow us to forget the human dimension. Emergency medicine requires fast diagnosis and treatment protocols, which technology provides. However this form of medicine is entirely different from the degenerative illnesses that take the lion’s share of doctor’s time and finances.

Lastly, our dabbling with DNA synthesis, gene transmission and nano technology is flying in the face of evolutionary processes. We want quick fixes and fail to appreciate the colossal contribution evolution has made to our species’ hardiness to survive. By removing so-called threats such as viral, bacterial and other organisms that have always been co-partners in our world, we are damaging the natural balance of our environment.

We are guilty of fooling ourselves over our abilities to make the world a safer and healthier place for ourselves and future generations. If the doctors go on strike over their concerns over reduced pension rights, then wait and see. Overall mortality figures will drop over the duration–not rise–which is what happened when Israeli medics went on strike in March 2000:

‘Industrial action by doctors in Israel seems to be good for their patients’ health. Death rates have dropped considerably in most of the country since physicians in public hospitals implemented a programme of sanctions three months ago, according to a survey of burial societies.

The Israel Medical Association began the action on 9 March [2000]… Since then, hundreds of thousands of visits to outpatient clinics have been cancelled or postponed along with tens of thousands of elective operations.

In the absence of official figures, the Jerusalem Post surveyed non-profit making Jewish burial societies, which perform funerals for the vast majority of Israelis, to find out whether the industrial action was affecting deaths in the country.

“The number of funerals we have performed has fallen drastically,” said Hananya Shahor, the veteran director of Jerusalem’s Kehilat Yerushalayim burial society. “This month, there were only 93 funerals compared with 153 in May 1999, 133 in the same month in 1998, and 139 in May 1997,” he said.

Meir Adler, manager of the Shamgar Funeral Parlour, which buries most other residents of Jerusalem, declared with much more certainty: “There definitely is a connection between the doctors’ sanctions and fewer deaths. We saw the same thing in 1983 [when the Israel Medical Association applied sanctions for four and a half months].”

Doctors’ strike in Israel may be good for health. BMJ 2000;320:1561.1

So counter intuitively, the benefits of our expensive medical system may be seen in a different perspective should a protracted strike begin.