Nice summary. I don't see a link to the original preprint you're discussing? Also what areas do you have in mind: "There are areas where fraud is almost unknown in medical science"?
WHOOPS! Thanks for pointing that out, I totally forgot to link to James' preprint!
I think most areas where auditability is required have a low rate of fraud. Trials of novel pharmaceuticals where independent bodies look at the raw participant data. Large multicentre studies across countries where an independent statistician (or more than one) reviews results from multiple places and will notice any issues. Even smaller studies in areas like cancer research where every patient is carefully monitored and it's very hard to pretend that you have more people enrolled than you see in your clinic. Oversight is the biggest fraud prevention device, because if more people see the data then more people can notice issues!
From my years in molecular biology, it was quite the opposite: mine and competing labs operated on the system of *mistrust*: we treated everything others published as definitely wrong and they treated our research the same way. We repeated their steps and wrote how wrong they were in their conclusions. They repeated ours and wrote how wrong we were.
An outsider following the network of publications on the topic could think all that research was "wrong", but it was actually all fundamentally correct and supporting across the labs, polishing the theory with every year of seemingly vicious back-and-forth. Science is truly powered by our nerd drive to correct each other, same thing that powers wikipedia.
But we also all knew who sold their integrity, abandoned pursuit of knowledge, and publishes paid BS in MDPI journals. No lab would bother with anything they wrote let alone rely on their conclusions in our own work, but again, unfortunately very hard for the public, or even for scientists working on unrelated subjects, to spot those internal, often unspoken, dynamics.
We know that very few researchers are willing to provide raw data alongside their papers and although they claim confidentiality for participants, in reality the overwhelming majority of these can and should be anonymised. But that's just the manipulation/fraud side of things. When we get to sheer, unbridled incompetence, that's got to be off the charts, especially in the social sciences, where all kinds of bullshittery gets past ethics review, gets funded, gets published, and ends up crapping all over the poor clients, via therapists who fervently believe in the gold-plated guru they've invested themselves in.
The Carlisle study you mentioned in the beginning of your article (https://doi.org/10.1111/anae.15263) in of itself is shoddy research IMHO.
As a research engineer at a top biomechanics research center, I am thoroughly confused why the author manually reviewed tabular data. The tabular data should have been analyzed by automated statistical methods. The author readily admits he missed data until the 3rd or 4th iteration and even more astonishing used exceptionally primitive tools for his analysis. Our center employs statisticians and epidemiologists for statistical analysis. After 20yrs in this field I cannot come up with a reason for manual statistical analysis by an MD. Particularly when the given subject is shoddy science.
Is there a breakdown anywhere of which areas of medicine are the worst, and which are the least fraudulent? A league table perhaps?
My impression is that areas afflicted by pseudoscience and integrative/CAM medicine rank highly in this regard, but that might be a biased conclusion on my part.
I don’t doubt there’s a problem but it’s hard to envision how this could be so high for experimental science that requires a lot of investigators. There’s a whole team of people. Then everyone tries to replicate so then this would also be a red flag. Everyone is talking to everyone else. They would have to collude to fake the data wouldn’t they? This is how a number of the frauds have been exposed, isn’t it? The PI does fishy stuff and the others working on it report him or her. But there could be fudging and so on and low-quality journals that just post fabricated studies because they are not interested in any of the standards but only at getting money for the journal. There’s so many review processes in medicine it’s probably not that easy to put forward too many fraudulent results?
The issue doesn’t seem to be primarily one of quantity but of impact. Many studies do not have impact but there are so many studies. The really bad thing would be to get research money if you are not on the level but as for your study making a bunch of things happen—it’s not so clear, is it?
Fraud is indeed much less common in places where there are many investigators who all have access to the data. But there are many areas of medicine - like anaesthetics - where it's not uncommon to see only one or two researchers being in charge of all of the numbers. And what we tend to find is that those areas of research have a very large quantity of dodgy shit, while things like novel pharmaceutical research have far fewer fake studies.
We really do have to talk some time. This is awesome.
Nice summary. I don't see a link to the original preprint you're discussing? Also what areas do you have in mind: "There are areas where fraud is almost unknown in medical science"?
WHOOPS! Thanks for pointing that out, I totally forgot to link to James' preprint!
I think most areas where auditability is required have a low rate of fraud. Trials of novel pharmaceuticals where independent bodies look at the raw participant data. Large multicentre studies across countries where an independent statistician (or more than one) reviews results from multiple places and will notice any issues. Even smaller studies in areas like cancer research where every patient is carefully monitored and it's very hard to pretend that you have more people enrolled than you see in your clinic. Oversight is the biggest fraud prevention device, because if more people see the data then more people can notice issues!
From my years in molecular biology, it was quite the opposite: mine and competing labs operated on the system of *mistrust*: we treated everything others published as definitely wrong and they treated our research the same way. We repeated their steps and wrote how wrong they were in their conclusions. They repeated ours and wrote how wrong we were.
An outsider following the network of publications on the topic could think all that research was "wrong", but it was actually all fundamentally correct and supporting across the labs, polishing the theory with every year of seemingly vicious back-and-forth. Science is truly powered by our nerd drive to correct each other, same thing that powers wikipedia.
But we also all knew who sold their integrity, abandoned pursuit of knowledge, and publishes paid BS in MDPI journals. No lab would bother with anything they wrote let alone rely on their conclusions in our own work, but again, unfortunately very hard for the public, or even for scientists working on unrelated subjects, to spot those internal, often unspoken, dynamics.
We know that very few researchers are willing to provide raw data alongside their papers and although they claim confidentiality for participants, in reality the overwhelming majority of these can and should be anonymised. But that's just the manipulation/fraud side of things. When we get to sheer, unbridled incompetence, that's got to be off the charts, especially in the social sciences, where all kinds of bullshittery gets past ethics review, gets funded, gets published, and ends up crapping all over the poor clients, via therapists who fervently believe in the gold-plated guru they've invested themselves in.
The Carlisle study you mentioned in the beginning of your article (https://doi.org/10.1111/anae.15263) in of itself is shoddy research IMHO.
As a research engineer at a top biomechanics research center, I am thoroughly confused why the author manually reviewed tabular data. The tabular data should have been analyzed by automated statistical methods. The author readily admits he missed data until the 3rd or 4th iteration and even more astonishing used exceptionally primitive tools for his analysis. Our center employs statisticians and epidemiologists for statistical analysis. After 20yrs in this field I cannot come up with a reason for manual statistical analysis by an MD. Particularly when the given subject is shoddy science.
Is there a breakdown anywhere of which areas of medicine are the worst, and which are the least fraudulent? A league table perhaps?
My impression is that areas afflicted by pseudoscience and integrative/CAM medicine rank highly in this regard, but that might be a biased conclusion on my part.
I don’t doubt there’s a problem but it’s hard to envision how this could be so high for experimental science that requires a lot of investigators. There’s a whole team of people. Then everyone tries to replicate so then this would also be a red flag. Everyone is talking to everyone else. They would have to collude to fake the data wouldn’t they? This is how a number of the frauds have been exposed, isn’t it? The PI does fishy stuff and the others working on it report him or her. But there could be fudging and so on and low-quality journals that just post fabricated studies because they are not interested in any of the standards but only at getting money for the journal. There’s so many review processes in medicine it’s probably not that easy to put forward too many fraudulent results?
The issue doesn’t seem to be primarily one of quantity but of impact. Many studies do not have impact but there are so many studies. The really bad thing would be to get research money if you are not on the level but as for your study making a bunch of things happen—it’s not so clear, is it?
Fraud is indeed much less common in places where there are many investigators who all have access to the data. But there are many areas of medicine - like anaesthetics - where it's not uncommon to see only one or two researchers being in charge of all of the numbers. And what we tend to find is that those areas of research have a very large quantity of dodgy shit, while things like novel pharmaceutical research have far fewer fake studies.
Oh, dear! Wow.