Which? magazine, a consumer comparison and review publication has just published a study on the dangers posed by nutritional therapists. A difficult leap from its traditional role of testing washing machines and vacuum cleaners – nevertheless the headline scoop “Are Nutritional Therapists Gambling With Your Health?” shows a robustness of tone and a certainty not often seen in scientific papers. So just how well was this supposed ‘scientific study’ designed and executed?
Which’s usual techniques involve comparing a variety of solutions, to whatever problem interests them, and determining the best option on offer. The validity and legitimacy of the product is on the whole, taken as a given. We need to wash our clothes, we need to vacuum up dust, we need a TV set and so on. This form of review journalism, whilst not easy, is inherently simple in purpose and straightforward in execution.
Should Which? ever decide to ask “Are Washing Machines Really Necessary At All”, then a completely different style of study is clearly required, engaging broadly different contributors across related fields. Stuffing a review panel with individuals with a track record of hostility to personal hygiene would introduce an unwelcome bias into the conclusions. Lastly, if the study excluded all representation by those being studied – if the washing machine manufacturers and retailers were denied any input in evaluating the results, then those same results would show a clear review bias.
Nevertheless this is how the Which? study has been structured.
In this particular case, whilst there was no nutritional therapist representative on the review panel, Which? attempts to cover itself by the simple expedient of ‘deeming’ that the panel are experts in the field of nutritional therapy and therefore qualified to sit in judgment on nutritional therapists. This expert panel consists of three individuals with a long history of collaboration with each other, and a well established antipathy to Nutritional Medicine including owning websites whose sole purpose appears to be the disparagement of all nutritional therapists and related academic qualifications up to and including MSc degrees. This clearly indicates a bias of authority.
Nevertheless this is how the Which? study has been structured.
Part of their conclusion states that GP’s and dietitians should be consulted for any medical condition. No evidence is offered for this assertion, whilst both a Dietitian and a GP are represented on the review panel. There is a clear conflict of interest bias.
Nevertheless this is how the Which? study has been structured.
One supposed aim of the study was to measure the quality of advice on offer by nutritional therapists. The questions were designed specifically to tempt the subjects into indiscretion and error, which clearly implies the covert real aim of this project was to ‘expose’ supposed charlatans much in the way the late and unlamented News Of The World might have approached the question. Secretly recording advice given in good faith in response to questions partly designed by those with a conflict of interest is good tabloid journalism, but extremely bad science. This is a fine example of measurement bias.
Nevertheless this is how the Which? study has been structured.
Which? studied a sample of 15 nutritional therapists out of a total population of over 2,400. In order for any valid results to be considered representative of all nutritional therapists (with a reasonable degree of certainty) then rigorous scientific methodologies exist covering sample size and selection methods, and much else besides. In summary, for such a study to claim any representative value at all, a sample size of around 100 would be the bare minimum required. A sample size of 15 is not even ‘bad science’ – it’s no science at all.
Nevertheless this is how the Which? study has been structured.
Which? offers no insight into how the sample was selected – it should of course be completely random but in the absence of confirmation doubts must remain. Certainly if it transpires the nutritional therapists were individually selected by the review panel, the entire basis of the study is undermined. This is known as selection bias.
So what did the ‘study’ really measure?
As pseudo-science, it quite clearly measures nothing at all. However, it certainly does illustrate that an unknown percentage of nutritional therapists may be incompetent, and that BANT may be ineffective in policing it’s members’ competencies. But are there more useless nutritional therapists than there are useless dietitians? Or useless GP’s? And who is the most dangerous? The study carefully avoids these important questions.
I‘m disappointed to note that dietitians, who claim to place reliance on evidence-based science, should contribute to such bad science in a non-peer reviewed article in a consumer magazine traditionally more concerned with vacuum cleaners. I certainly hope that the damage to their professional reputations is only temporary.
I await a well-designed comparison study in the more honorable tradition of Which? magazine. But next time please – be careful how you choose your ‘experts’.
Note:
For more information on sources of bias, please refer to:
http://www.umdnj.edu/idsweb/shared/biases.htm
I am not at all surprised at these “cientific findings”. Ever since I became a Nutritional Therapist, the doctors have seen me as a threat to their profession. Firstly, they don’t seem to understand anything related to nutrition, let alone functional tests and nutrients. However, I have seen, time after time, clients that have come to me as a last resort to deal with their health problems. Back in 1997, one of my first clients was a lady who had been told by her GP that due to her arthritic condition, she would probably be in a wheel-chair in 5 years’ time. I am glad to report She is still walking and doing fine, and she still consults me wherever I go (initially in London then in Spain and now in Colombia where I live). Closer to my heart, I have to report that my 84-year old mother doesn’t take one single medication (only nutritients and good diet) and is in perfect health, and her old arthritis, incipient osteoporosis and many aches and pains have long dissapeared. Who needs proves when you see your family, friends and clients recover from usual ailments, just by following good sound nutritional advice??
Hi Clara, thank you for sharing your positive experiences with Nutritional Therapy all the way from Colombia! It is the case that the current scientific paradigm dictates that ‘proof’ is required for any health advice given. However, this is not easy to accomplish in this field and the scientific methods to properly evaluate all the multifactors involved do not exist yet, and will take some time to evolve.
On the particular question of sample size, may I draw your attention to the confidence limits calculated at http://www.dcscience.net/?p=4997#170112
These show that. although small, the sample was quite big enough to draw useful conclusions. Perhaps you you should have done this calculation yourself before accusing Which? of being unscientific.
My observation that the Which? article is unscientific is clearly not based purely upon sample size; that is one of a number of factors and is by no means the most important. It is a small sample, and you are quite entitled to consider it sufficient to support your conclusions. I don’t agree that it does. I do, however, agree with you that BANT need to sharpen up their procedures regarding incompetent practitioners if the transcripts support your findings. This does not necessarily appear to be the case according to the updated press release from BANT http://www.bant.org.uk/bant/pdf/WHICH/UPDATED_RESPONSE_FROM_BANT_TO_WHICH_ARTICLE_20120118.pdf
I make no claim that my article would pass the rigorous criteria required for publishing in a scientific journal. It is after all a blog post. The real question is, would yours? When the Which? report is published, word for word, headline title and all, with all its conclusions intact, in the BMJ or a journal of similar standing, then I shall gracefully concede defeat and would be delighted to invite you to a healthy lunch (preferably without the tape recorder).
It is shameful that you attack the authors and advisers on the report rather than address the serious concerns raised by the report. You do not even acknowledge the issues.
Which? found that Nutritional Therapists were routinely giving bad advice, some dangerous advice. They used unscientific diagnostic techniques and recommend their customer not to visit their GP and to delay cancer treatment.
By any objective standard, this is appalling.
What would an ‘expert’ nutritionist have done if on the review panel? Claim that all these things are OK?
Nutirional Therapists rapidly need to get their house in order before it is done for them.
Hi Andy from Quackometer,
Very sorry you feel as you do! Disagreement is not attack. Illuminating inherent bias is not accusatory. We are all biased.
Any report claiming scientific credentials needs to minimise the possibility of bias. Failure to minimise bias simply produces results which may be justifiable, but not justified.
Perhaps re-reading my original article may serve to clarify my position which is not as you state. Also see updated BANT response which in any case disagrees with the report’s findings.
My major concern here is not the sample size, but rather the method of selection of the nutritional therapists investigated. There was no mention of how they were selected, an appropriate way would be randomly from some sort of nutritional therapy register. If we give Which? the benefit of the doubt and assume that is what they did – we see a small mention of the fact that one practitioner was visited by two of the participants. If we take the conservative measure of the number of nutritional therapists as 1,805 (not 2400+ see http://www.dcscience.net/?p=4997#170112) then the chance of the same therapist being randomly selected twice is 0.06%. Personally, I would take this as reasonable evidence to support the notion that the sample of therapists was not selected randomly. If your sample is not randomly taken from the population you are trying to study, then this violates the underlying assumptions of any whiz-bang stats you might like to apply, Bayesian or otherwise. I’m not defending people giving out crap advice, but to damn an entire profession on the basis of this investigation (not scientific study) seems like a Which? hunt to me.
Another cause for concern: As I have just pointed out, one therapist was visited by two researchers yet their consultations are considered to be independent data points (as evidenced by all results being given as out of 15). Confidence intervals as put forward in the above post http://www.dcscience.net/?p=4997#170112 use this total figure of 15. This is pseudoreplication. To illustrate the invalidity of pseudoreplicated results, you can ask yourself whether visiting the same therapist 15 times and getting 15 bits of bad advice would be sufficient evidence to judge the rest of the therapist population. Obviously not. This is the same thing, just at a smaller scale. No less invalid. I can’t recalculate the confidence intervals correctly because I don’t know how this particular therapist fared in either consultation, it likely does not alter the results greatly but in general this is considered very bad science and certainly rings alarm bells. On that basis I would not be so quick to offer the benefit of the doubt on unanswered methodological questions.
Have to correct my own calculations, the chance is actually 0.8% (not 0.06%, that is for two in a row not two out of 15). Still less than 1% though, so pretty small..
Eleanor,
‘the chance is actually 0.8% (not 0.06%, that is for two in a row not two out of 15)…’ Will you please explain how you calculated this? Just curious…thanks!
hmmm…. I’ve done it wrong again! Well, at least this goes to show (to me!) that these things are generally not as straight forward as they initially seem.
So the first time I just asked what was the chance of picking one therapist out of 1805 and then picking the same one again. This is the 0.06%. This is wrong because you don’t just pick two, you have another 14 chances to pick the same one.
So the next time I simply multiplied 0.06 by 14 to account for the extra chances to pick the same practitioner. This comes to 0.8%. However this is also wrong as it is still the chance of picking that one initial therapist again, not all the others that have also been picked in subsequent selections.
The correct calculation (I think – please correct me if I’m wrong again) is 1/1805 + 2/1805 + 3/1805 and so on all the way up to + 14/1805. This is quite a lot higher again at 5.8%
So (pending further correction!) we can say is there around a 6% chance that the same practitioner could be picked twice. Apologies for my initial bungling.
Despite this, I still think a value of 6% makes it rather unlikely that this would have occurred. I look forward to confirmation of methodology from Which? at some point. I woulds argue though, that a study of this kind would be better suited to random selection without replacement (i.e. the same practitioner cannot be picked twice) to avoid the consequent effects of pseudoreplication.
Since the report was published on 16th January, one or two quiet voices have been persistently asking for details of the sampling methods used in the study. Particularly crucial are Eleanor Dormontt’s observations on pseudoreplication and the probability of randomly selecting the same practitioner twice – which the study did. Answers have been hard to come by, but bland unsupported assertion is much in evidence in the comments sections at which.co.uk:
• David Colquhoun supplied calculated beyesian confidence limits for the Which? study on 17th, This calculation assumes and requires the sample to be random.
• Expert reviewer and dietitian Catherine Collins confirmed the sampling was random on 18th January but giving no details of methods used.
• Shefalee Loth (Which? staffer) confirmed the sampling was random, also on the 18th, again giving no details of sampling methodology.
• Around mid-day on 18th January we see the first signs of backtracking. Avoiding the direct issue of sampling methodology and without confirming whether the sample was random or not, Colquhoun states that the expert panel had ‘nothing to do with selection’.
• Finally on the 19th January we see Colquhoun’s acceptance that the ‘the sample was not a random sample of all nutritional therapists.’ (Posted 19th January 1:26am)
At the time of writing we see that of the three expert reviewers, one reviewer (Colquhoun) says its definitely not random sampling. Another reviewer (Collins) says it definitely is random sampling, and the third reviewer Margaret McCartney is wisely keeping very quiet on the subject.
In David Colquhoun’s excellent published book on the subject (‘Lectures On Bio-statistics’ p16) he explains in his own words, why randomness is fundamental:“Any sort of statistical analysis (and any sort of intuitive analysis) of observations depends on random selection and allocation having been properly done.”
Looking more deeply into what we know about the random (or not random) sampling methodology actually used in the study: there is little information available other than this offering from Colquhoun’s discussion comments on 19th January:
“I understand that Which? staff selected the therapists in a way that mimicked what a present day consumer might do. They used google and online directories, and chose the therapists that were at the top of the search results. Thus, the sample was not a random sample of all nutritional therapists, but would be like the therapists that were chosen by googling. It would be biased towards the more popular (and therefore perhaps better?) therapists. It is a worry that less popular therapists might have given worse advice than was found in this case.”
That any scientist, especially a dietitian such as Catherine Collins could ever consider this as a random selection is interesting. Colquhoun concedes it is not random, then introduces as a fact the unsupported conjecture that the study now becomes biased in favour of nutritionists. It’s not random any more – it’s now better than random! Colquhoun is also clearly saying that clinical competence of nutritional therapists declines with your google search position.*(see note). One waits to hear more of this ground-breaking discovery in the scientific press.
So how to make sense of all this? It is probable that, at the start of the Which? project, the reviewers either assumed or were told by Which? that the sample was random. Some time after publication, the truth emerges which destroys the entire legitimacy of the report’s findings. The expert panel are in disagreement with each other as to the validity of the sampling methodology used, and now the Which? discussion board has been closed down (now re-opened).
In summary, any conclusions drawn from a non-random study are simply meaningless. They cannot be relied on to suggest anything at all about the population under investigation. The Which? report can therefore be revealed as a study whose sole basis for legitimacy is the stridency of its own conclusions (see Salem Witch Trials).
Clearly, the panel of experts should all now disassociate themselves from the Which? report and its conclusions. Which? should also retract the report; whether it does or not may well depend on the weight of any defamation actions. Which? should at least acknowledge that a selection methodology more suitable to assessing the relative merits of washing machines should never be applied to a study of this type.
Note:
*The raft of problems associated with this statement and this type of sample selection are worthy of much discussion, but not here! As a taster, the notion that google listings are necessarily biased towards popularity is generally false, (confusion of cause and effect), the association of practitioner competence with google ranking is unsubstantiated and patently absurd, search results differ depending on the geo-location of the initial google enquiry, which search terms were used, and so on…