Misdiagnoses Aren't Killing Or Disabling 800,000 People A Year
Why the evidence doesn't really show that misdiagnoses are a leading cause of death in the United States
As a society, we absolutely love a story where healthcare is the villain. There's a regular parade of news where some research study seems to indicate that healthcare professionals are destroying lives. Every once in a while, the myth that medical error is the third leading cause of death rears its head, despite the obvious fact that this cannot possibly be true.
This week, a new spate of headlines has popped up with a very similar vibe. If you believe the news, missed diagnoses - or misdiagnosis, where doctors fail to diagnose people correctly in a timely manner - is causing a truly astonishing amount of suffering in the United States. According to a recently published piece of research, nearly 800,000 people either die or are permanently disabled each year due to misdiagnosis in the US.
Fortunately for those of us who see doctors regularly, this estimate is extremely unreliable. The true number of people harmed by misdiagnosis is probably drastically lower than 800k a year.
The Study
The new study, published in the British Medical Journal, is the third in a series of papers. This series was aimed at estimating precisely what the headlines are reporting on - how many people are harmed because their diagnoses are missed in the United States?
The authors did this in a three-part investigation - the first step looked at which diseases were likely to be responsible for the majority of misdiagnosis harms. The second step looked at how common those harms might be for people who had the diseases, getting a very rough estimate of how many people with, say, a stroke might have their diagnosis delayed or missed. Finally, the third step applied those results to an epidemiological database, generating an estimate of how often people in the general population actually experienced the harms that they had described.
And so we get the final estimate that’s been hitting the headlines. If you look at the table below, you can see exactly how this works - multiply the number of cases of the disease diagnosed each year (incidence) by the rate of ‘diagnostic error’, and then by the rate of serious harm associated with diagnostic error, and you get to the far right column which estimates the number of harms per year in the United States. Simple!
The problem here is that none of this is simple, and none of those steps are in any way reasonable. When you look at the underlying evidence here, the argument that these numbers reflect harms caused by misdiagnosis fall apart entirely.
Citation Laundering
I’ve discussed citation laundering before, but it’s a really key concept for this study so let’s go over the basics:
A group of scientists make a claim once, in a very vague and defensible way in an academic journal, where it can be cited. “A might be associated with B”.
That paper is cited, using slightly stronger language, removing the uncertainty but keeping the language. but still phrased somewhat reasonably. “A is associated with B”.
The above is repeated, maybe more than once.
Now, in a citation chain with two or more links, the original claim starts to get cited as proof of the original idea, no matter how strong the initial claim might have been. “A causes B”
It’s basically a game of Telephone, which for some inexplicable reason is on Wikipedia under Chinese Whispers. It’s a very effective tool, because by the time you’re 2-3 citations down the list it’s incredibly difficult for anyone to check whether the things you’ve said make sense.
Does A Cause B?
Note: this next section is long. It is detailed. It has to be, because there’s a lot to cover here. If you don’t want all the wonderful pedantry, skip to the end from here.
Getting back to the paper at hand, have a think through of the methodology. The authors have taken two previous literature reviews they did, which themselves were based on studies which were mostly reviews of one kind or another, and aggregated all of this into one central estimate. We are at least 3-4 steps away from the original research here, which makes citation laundering a potential problem.
So, why don’t we look at some of the source data for the claims made? Take lung cancer. The authors have estimated that there are 224k lung cancers in the US each year, and that of these 50k (22.5%) will experience a misdiagnosis event, and 32k (14.2%) will experience a serious diagnostic harm, which they define as permanent disability or death.
Taken as a single claim, this is a huge fucking deal. 1 in 8 people with lung cancer are permanently disabled or die because of delayed or missed diagnosis? If that’s true, the entire basis for lung cancer treatment in the US is obviously flawed.
So what’s the estimate based on? The authors cite their own step two review as the basis for the claim. That review has several citations for lung cancer, but if you go down the daisy chain to the original evidence you get to this study: Guideline-concordant Timely Lung Cancer Care and Prognosis among Elderly Patients in the United States: A Population-based Study.
In this paper, a group of researchers estimated how many of the people who were diagnosed with lung cancer in a very large database in the US between 2002 and 2007 received timely and appropriate care. They also looked at how long before a diagnosis people had at least one symptom that might have been associated with the cancer.
Of the 48,850 people in this database, 22.5% received “guideline-concordant timely lung cancer care”. In the study, this was defined as an acceptable time between diagnosis and care, which for lung cancer in the 00s meant how long between the initial diagnostic x-ray and surgery, radiotherapy, or chemo.
There are a few issues here. Firstly, there’s no argument from the authors of this study that waiting longer for lung cancer care directly caused harm. In fact, quite the opposite - people with delayed care survived for longer than people whose care happened quickly. Secondly, this was not under any definition an assessment of diagnosis-related harm. Every person in this study was diagnosed. There was no assessment of misdiagnosis at all. The actual 22.5% figure comes from an estimate of potentially delayed treatment, which means that it is incorrect to use it to look at harms potentially associated with diagnosis.
This means that the lung cancer estimate - that big fucking deal from before - is completely meaningless. It is not just unlikely that 1 in 8 people with lung cancer experience death or disability due to misdiagnosis - it is directly contradicted by the original research. If anything, there is an association with better outcomes for delays in treatment in the original study, although as the authors note that’s very unlikely to be causal.
Ok, so that’s one part of the authors big table which is wrong. What about the others? Well, fully 10% of the entire 800k big number is made up of 94k stroke-related harms. According to the authors, for every 100 strokes in the US there are about 4 people killed and 6 people permanently disabled by the direct mistakes of clinical staff.
Again, the authors cite themselves. They first go back to the same step two study, which itself cites another review that they conducted in 2017. This review tried to estimate the proportion of strokes and similar conditions which were picked up correctly by physicians in Emergency Departments in the US.
When you go down to the original research it’s blindingly obvious that again this has very little to do with misdiagnosis of any reasonable definition. One of the bigger studies that was included in this review looked at people who turned up at ED in 2000-2002, and got some neurologists to look retrospectively at the decisions made by the ED physicians at the time. They found that most of the ED judgements were correct, and disagreed with the ED diagnosis of stroke about 10% of the time.
Is this misdiagnosis? Certainly, neurologists are better at diagnosing strokes, but most of the disagreements happened in people with very generalized symptoms anyway. There’s also no gold standard here, because the neurologist was looking back at the cases with the benefit of hindsight, while the ED physicians were making decisions in the moment. We don’t know whether neurologists working in a busy ED would actually make different decisions to ED physicians - we just know that they disagree sometimes after.
You see the same thing when you look at some of the smaller studies that this review picked up. One study of 189 people discharged with a diagnosis of stroke found that 29 of them did not have the possibility of stroke mentioned in their clinical records by the admitting ED team. But all 189 people were referred to neurology and had an MRI and the correct diagnosis within a day or two of arriving. The study did not record whether those who didn’t have stroke mentioned in their ED records had a longer time to diagnosis than those who did. Again, this is a clinical process that could probably be improved - ED clinicians were found to be under-recognizing stroke patients with nontraditional symptoms - but it’s a complex question whether this is misdiagnosis.
In the initial review, the authors actually acknowledge this issue quite frankly, stating that “it is difficult to know if some errors simply reflected an initial, rough diagnostic assessment or a reasoned decision in an uncertain case to consult neurology, rather than directly ordering an MRI”. There’s no way of knowing if these ‘misdiagnoses’ were incidents where patients had delayed or missed diagnoses, or simply places where ED physicians were correctly referring to neurology services when they weren’t sure what to do with someone.
There’s also a remarkable numeric inconsistency between the author’s reviews. In their first review, they found that 10.4% of strokes were misdiagnosed. When this was aggregated for the step two review paper for this project, the “diagnostic error rate” was given as 8.7% for stroke. But if you look at that table above, in the final estimate this has jumped up to 17.5%, which doubles the number of stroke-related harms that the authors estimate occur. I can’t find any explanation for this doubling of harms due to stroke between papers, but it does have the effect of increasing the topline figure by a good 50,000 people or so.
This is where the citation laundering comes in. The nuance is completely discarded in favor of defining the overall point estimate for stroke as a “Literature-derived missed or delayed diagnosis rate” in the next paper upstream. The final estimate posits a direct causal relationship between this uncertain, vague delay - that may in many cases not be a delay at all - and the eventual deaths and permanent disabilities of 94,000 people a year.
If the authors had laid this out in a single paper, everyone would point and laugh. It’s a ridiculous suggestion. Some people who have strokes may have had their diagnosis delayed. This may have contributed to further ill health, and maybe their death. No one in their right mind would accept this series of assumptions if it was clearly laid out, but when it’s all hidden behind “As reported previously” and some superscript numbers very few will care enough to check.
Down The Rabbit Hole
The problem with a paper like this is that there is just so much to cover. For example, take sepsis. The estimate of missed diagnoses for sepsis was based on four studies:
A case series of 487 patients where the authors concluded that “Non-recognition of sepsis in ED patients with serious infections who formally meet organizational sepsis definitions seems to have no deleterious impact on initial therapy adequacy”.
A randomly-selected retrospective cohort study where the authors found that out of 300 people who died of sepsis, 16 were potentially preventable due to “delays in recognition and treatment of sepsis”.
A large population cohort where the authors reviewed every pediatric admission for sepsis in Ontario between 2005-2010, and checked whether those children had been to an ED in the 5 days before getting admitted. However, these authors specifically did not address whether these were misdiagnoses, acknowledging that it may have been clinically appropriate to send the kids home at the time they first attended. They also found no clinical harm associated with delayed admission in this patient group.
Another population cohort which looked at all children in a large clinical registry. Researchers conducted a post-hoc chart review on an arbitrarily selected group of children with suspected and confirmed sepsis. They found that there were some children who possibly had sepsis based on this retrospective chart review who did not have this noted in their clinical record - it was about 2% (20/996) of the children in an acute setting, but 12% (12/98) in a community setting.
From these four studies, the authors summed the total number of cases of ‘missed’ sepsis and the total cases of sepsis. This gave them 169 missed vs 1,780 total, from which they derived a point estimate of 9.5%.
These papers didn’t really measure ‘missed’ diagnosis of sepsis. They measured whether ED physicians had appropriately documented sepsis in their notes. The treatment that the patients received in all of these studies did not differ between those who had adequate notes and those who didn’t. You could argue that this is simply evidence that ED physicians are not great at accurately recording sepsis diagnoses, and has nothing to do with misdiagnosis at all.
The authors also somehow managed to extract the wrong numbers from one of these papers. In the retrospective cohort study, there were 36 sepsis-associated deaths out of 300 sepsis cases which were considered ‘potentially preventable’. This is the number that appears in the review paper. But only 16 of these were due to delayed diagnosis - the remaining 20 were because people received the wrong dose of antibiotics, had hospital acquired infections, or other issues with their treatment.
It’s also absurd to use these four studies as a basis to make any statements about the US as a whole. 80% of the estimate comes from two large studies that only looked at children, and one of those was in Canada. The one study with a wildly high estimate of missed cases of sepsis, where researchers found that 59% of cases were missed, was based on a tiny sample of people in Germany. The only study that examined adults living in the United States - who make up most of the deaths from sepsis - found that just 5% of cases had a delayed diagnosis and that this didn’t impact treatment or outcomes in any measurable way.
And beyond that, how do we know how much harm missing a case of sepsis causes? Remember - three out of these four studies found that the ‘missed’ diagnosis, however defined, was not associated with worse outcomes.
The final step three paper didn’t actually assess how harmful any particular misdiagnosis would be. They just used five separate papers to come to an overall estimate of how likely it was that a ‘serious harm’ would result from a misdiagnosis, then applied a weighting factor to estimate how much more/less harmful this issue would be for each disease.
And, again, we can go through those five studies, briefly:
A retrospective survey asking doctors to remember a case where a patient was misdiagnosed and what happened.
A fairly large study looking at voluntary reports made by physicians about errors.
Another reasonably big review of medical records from 2004.
A smaller retrospective review of medical records from the mid 00s.
This is ridiculous. You can’t reasonably estimate the rate of harms caused by misdiagnosis from retrospective surveys asking doctors to remember a time when someone was misdiagnosed. People remember the patient who was misdiagnosed and later died, but no one remembers the person who had to wait an extra six hours for an MRI because the ED resident referred them to neurology rather than diagnosing them straight away.
Even the bigger epidemiological studies don’t really make sense to use. These are retrospective reviews of negative incidents that happened in hospitals. But again, people may not report misdiagnoses when nothing bad happens to the patient. By design, this will always result in an overestimate of the rate. What you need is an ongoing, longitudinal sample that measures every diagnosis and what happens to them.
So the authors overall estimate - that nearly 60% of all ‘misdiagnosis’ would result in disability or death - is very unlikely to be accurate. Most of the sepsis papers didn’t find any negative health impacts to ‘misdiagnosis’ as they defined it. The lung cancer paper found that ‘misdiagnosed’ patients did better.
The True Burden Of Misdiagnosis
Unfortunately, there’s a real problem here under some really mediocre science. Misdiagnosis is a common issue, especially for disadvantaged populations, and it’s absolutely true that it’s something the health systems of the world need to work on.
But none of this is helped with ludicrously high estimates of the role that medical decision-making has in people’s death. According to the CDC, there were about 130,000 stroke-related deaths in the US in 2021. If this paper was correct, about one third of those deaths were directly caused by ‘misdiagnosis’.
But that’s not even the most astonishing number - the paper implies that there were 271k serious harms related to infectious disease misdiagnosis, of which about 126k resulted in death. This review of misdiagnosis mortality actually looked at deaths in 2014, not 2021. In 2014, the number of deaths due to all infectious diseases in the United States was 113k according to a large review published recently.
In other words, taking the estimate from this paper at face value there were more infectious disease deaths caused by misdiagnosis than there were people who died due to infectious diseases. Part of this is to do with definitions, but even the individual disease categories are quite clearly impossible - the authors report that there were around 3,000 deaths caused by misdiagnosis of meningitis and encephalitis in 2014, but CDC Wonder, a database of mortality for the US, puts the total number of deaths at only 1,500.
Hopefully, I don’t need to explain why this is a problem. You can’t have more people killed by misdiagnosed meningitis than people who died of the disease.
So how many people are killed by misdiagnosis every year? Well, the obvious answer is none, because people die of diseases, not misdiagnosis. People who are diagnosed incorrectly missed out on the chance to be saved, but their deaths are still ultimately caused by the heart attack, cancer, stroke, etc.
That’s why “Burden of serious harms from diagnostic error”, the title of this paper, makes no sense to me. What the authors are ultimately looking at are areas within the healthcare system where improvements could potentially improve patient care. But they might not. Neurologists looking at charts post-hoc may disagree with ED physicians about who likely had a stroke, but if the neurology registrar was the one up at 2am reviewing the 30th person to turn up to ED with a headache and blurry vision that day there’s no guarantee they’d do anything different in their management. Mistakes in the medical system are potentially preventable, but only to a point.
There are definitely things that we need to improve on. There are people who are mistreated in hospitals, who wait far too long to be diagnosed, and those people too often are already disadvantaged populations. But the true number of people who are killed or disabled by misdiagnosis in the United States is likely to be orders of magnitude lower than 800,000 a year. At the very least, the number that’s hit the headlines is almost certainly a massive overestimate of the truth.
Nice article!
I would love to know what you think about studies showing benefit (cognitive, immunologic, etc.) from near infrared radiation treatment/exposure. I hope it's beneficial, but understand that studies cited in the media may be significantly flawed.
I'd also like to hear what you think about recent research showing melatonin possibly harming the microbiome and immune function (in mice?).
Thanks!