Tuesday, October 23, 2012

More DSM-5 confusion

One of the big concerns about forthcoming changes to autism diagnosis in DSM-5 is that the new rules may miss out a sizeable chunk of the autism population, particularly those who, under current diagnostic guidelines, would be considered to have Asperger’s Disorder or PDD-NOS. A number of studies published over the last year or so seemed to confirm those fears, although each of those studies had its limitations.

Earlier this month, a study in the American Journal of Psychiatry by Marisela Huerta and colleagues appeared to show quite conclusively that almost every autistic person, including people in the PDD-NOS bracket, would qualify for an Autism Spectrum Disorder diagnosis in DSM-5.

Emily Singer has a nice summary on the SFARI blog, complete with interviews with various researchers all agreeing that the study really seems to have settled the issue.

However, I'm not so sure. Here's my attempt to explain why.

Huerta et al.'s approach was to map items from the ADI-R and ADOS diagnostic tests onto "symptoms" in the DSM-IV and DSM-5 criteria. They then looked to see whether, for each individual assessed, the combination of symptoms qualified them for a diagnosis under DSM-IV or DSM-5. It's important to note that they didn't use the official algorithms for translating item scores into diagnoses. Instead, they asked whether, for each symptom, there was evidence from at least one item in ADI-R or ADOS.

Next, they compared the diagnostic outcome of the DSM-IV and DSM-5 recodings to what they call "clinical best estimate diagnosis". I was a little unsure what this meant, so I emailed Dr Huerta, who  confirmed that this was essentially the opinion of experienced clinicians at the time of the original assessment. The diagnosis was guided by DSM-IV and was based partly on outcomes of the ADOS and ADI-R, as well as developmental history and performance on some standardized tests.

Have a look at Table 2 below. This is actually just the top third of the table as the data Huerta et al used comes from three different sources. For now, we're just looking at data supplied by the Collaborative Programs of Excellence in Autism.

Table 2 from Huerta et al (2012). Click to enlarge.

We're interested here in the kids with best estimate diagnoses of PDD-NOS or Asperger syndrome. I've highlighted in red the row entitled "Either parent or clinical measure". This means that a symptom box could be ticked if it was present in either the parent measure (ADI-R) or clinical measure (ADOS).

The first column of figures shows the sensitivity and specificity according to the proposed DSM-5 criteria (columns to the right of this show outcomes for potential tweaks to the DSM-5 criteria). In this sample, 94% of kids with best estimate diagnoses of Asperger’s or PDD-NOS met the DSM-5 criteria for Autism Spectrum Disorder (see blue oval). Across the three datasets, the value was 96%.

But now look at the column headed "Autistic Disorder". This shows the proportion of kids meeting DSM-IV criteria for Autistic Disorder according to the recoding. In this case, the figure is 84% (green oval).

This is distinctly odd. According to DSM-IV, PDD-NOS and Asperger's disorder should only be diagnosed if Autistic Disorder has been ruled out. But the authors are reporting that, of the kids with a best estimate diagnosis of PDD-NOS or Asperger's disorder, 84% actually met criteria for Autistic Disorder under the recoding.

For the other two datasets, it's even more extreme. The corresponding figures are 90% and 97%. Overall, it works out that 94% of kids diagnosed with PDD-NOS or Asperger's actually meet criteria for Autistic Disorder.

Now look at the specificity, highlighted in orange. Across the three datasets, specificity for DSM-5 was 33% (purple oval). In other words, two thirds of the non-PDD kids met DSM-5 criteria for Autism Spectrum Disorder. We're not talking about autism risk here. This relates to actual diagnosis. 33% specificity would be terrible.

For DSM-IV it's even worse. Specificity is 10%. That means that 90% of the non-PDD kids met criteria for PDD-NOS (in fact two thirds of them met criteria for Autistic Disorder). 

Something is not right. 94% of kids who were considered by clinicians to have PDD-NOS or Asperger's ended up meeting criteria for Autistic Disorder under the recoding. And 90% of kids who weren't considered autistic at all ended up meeting criteria for PDD-NOS.

One possibility is that the clinical best estimates are wrong. If this were the case, it would completely undermine the claim that most kids with PDD-NOS or Asperger's will meet DSM-5 criteria - because the study didn't actually include any such kids. Or at least, if it did, they were all in the non-PDD group!

More likely I think is that the coding scheme used in this study is far more liberal than best estimate clinical diagnosis. Getting a diagnosis from a real life clinician requires more than just a single example of each relevant behaviour. If this is the case then it again undermines the study's conclusions. If the recoding is too liberal, if boxes are being ticked that would not ever be ticked by a clinician conducting a real best estimate diagnosis, then we can't trust the DSM-IV coding. And by the same token, we can't trust the DSM-5 coding either.

Emily Singer's article has an interesting quote from Cathy Lord, who's a member of the workgroup responsible the DSM-5 changes to autism diagnosis, as well as being a co-author on the Huerta et al. paper.

"It's still not the same thing as taking the new criteria and testing them out, which is why we didn't do this analysis before," says Lord. "But clearly people have been analyzing much more restricted datasets, so we thought we better get in here and do it."

I think she's absolutely right to worry about the limitations of recoded data. But we need to be skeptical whether or not the results support DSM-5.


Huerta M, Bishop SL, Duncan A, Hus V, & Lord C (2012). Application of DSM-5 Criteria for Autism Spectrum Disorder to Three Samples of Children With DSM-IV Diagnoses of Pervasive Developmental Disorders. The American journal of psychiatry, 169 (10), 1056-64 PMID: 23032385

No comments:

Post a Comment