Using symptom and interference questionnaires to identify recovery among children with anxiety disorders.

Procedure

All children were referred by primary and secondary NHS/education services to Berkshire Child Anxiety Clinic, University of Reading, UK. The children in the first trial ( Thirlwall et al., 2013 ) were assessed before (Time 1) and after (Time 2) eight-session GPD-CBT ( n = 50), four-session GPD-CBT ( N = 46), or a 12-week wait-list ( n = 63). The children in the second trial ( Creswell et al., 2015 ) were assessed before (Time 1) and after (Time 2) eight-session CCBT ( n = 56), eight-session CCBT + MCBT ( n = 60), or 10-session CCBT + MCI ( n = 62). All children and primary caregivers completed the ADIS-IV-C/P at Time 1 and Time 2 in all conditions. However, not all respondents completed the CAIS-C/P and SCAS-C/P at both time points (total n for each measure ranged from n = 247–320).

Spence Children’s Anxiety Scales

The Spence Children’s Anxiety Scales (SCAS-C/P; Nauta et al., 2004 ; Spence, 1998 ) are parent- and child-report scales comprising 38 items referring to common symptoms of child anxiety disorders. A number of subscales are formed from these 38 items. Relevant to the present study, six of the items form a separation anxiety subscale (SCAS-C/P-SA), six form a social phobia subscale (SCAS-C/P-SP), six form a generalized anxiety subscale (SCAS-C/P-GA), and five form a physical injury fears subscale (SCAS-C/P-PI) which is relevant to specific phobias. Respondents are asked to indicate how accurate each of the 38 statements are on a 4-point scale of never , sometimes , often , and always . The child and parent versions are identical apart from adjustments to make the wording suitable for each and the inclusion of ‘filler’ items in the child version. The SCAS-C/P has been widely used and established as having good concurrent validity ( Whiteside & Brown, 2008 ), discriminant validity ( Nauta et al., 2004 ; Spence, 1998 ; Whiteside & Brown, 2008 ), convergent validity with other commonly used measures ( Essau, Muris, & Ederer, 2002 ; Nauta et al., 2004 ; Spence, 1998 ; Whiteside & Brown, 2008 ), and acceptable test–retest reliability ( Spence, 1998 ). The good psychometric properties of the SCAS-C/P have been established in children aged 6 to 18 years ( Spence, 1998 ; Nauta et al., 2004 ), and it has been widely used with children of the age range considered here (e.g., Hudson et al., 2014 ; Vigerland et al., 2016 ). In the present sample, internal consistency was good for total SCAS-C/P total scores at Time 1 and Time 2 (α = .88–.93), although was more variable for subscales (see Table 2 ).

Child Anxiety Impact Scales

The Child Anxiety Impact Scales (CAIS-C/P; Langley et al., 2004 ; Langley et al., 2014 ) are 33-item parent- and child-report measures, containing 27 items which refer to impact within three categories: school activities, social activities, and home/family activities. Four items relate to global impact. Respondents are asked to indicate on a 4-point scale the extent to which anxiety has caused trouble with each activity (i.e., ‘not at all,’ ‘just a little,’ ‘pretty much,’ or ‘very much’). Parent and child versions are identical apart from minor changes to wording. The CAIS-C/P has been shown to have good internal consistency and convergent validity with other measures of child anxiety ( Langley et al., 2014 ). The psychometric properties of the CAIS-C/P have been established in children aged 7 to 17 years ( Langley et al., 2004 , 2014 ). Internal consistency was good for CAIS-P at Time 1 and 2 and for CAIS-C at Time 2 in the present sample (α = .70-.94) but not Time 1 (α = .52; see Table 2 ).

Data Analysis

Consistent with existing literature evaluating the accuracy of measures at identifying diagnoses by the ADIS-IV-C/P ( Bailey et al., 2006 ; Caporino et al., 2013 ; Villabø et al., 2012 ), and recommendations for this area of research ( Silverman & Ollendick, 2005 ), ROC ( Swets, 1988 ; Zweig & Campbell, 1993 ) methods were used to determine the ability of Time 2 total scores on the parent and child versions of the CAIS and SCAS, plus subscales of the SCAS, to identify recovery according to the ADIS-IV-C/P. Analyses were also conducted using change scores between Time 1 and Time 2, but no clear advantage was found of looking at these over Time 2 scores alone so we have focused our report on Time 2 scores. Furthermore, analyses were conducted separately on the two trials and just on the children who received treatment (i.e., excluding those in wait-list conditions). No clear or systematic differences were found in the results of any of these groups compared with the overall 337 children (see the Appendix for results with treated group only). Therefore, only the results of analyses with these 337 children are presented.

A number of values are generated from ROC methods. The AUC is a single global indicator of a test’s ability to correctly classify individuals’ diagnostic status, with an AUC of 1.0 indicating a perfect test and .50 indicating a test which classifies individuals at chance level. AUCs of .70 to .74 have been described variously as moderate ( van Gastel & Ferdinand, 2008 ), fair ( Villabø et al., 2012 ), and as representing a large effect size ( Rice & Harris, 2005 ). The present study therefore used an AUC of .70 as the minimum value for a measure to be considered at least moderately accurate at identifying recovery from diagnoses according to ADIS-IV-C/P. For each diagnostic outcome, results of analyses using ROC methods are only presented for measures which achieved an AUC exceeding .70. MedCalc for Windows (Version 15.6) was used to establish that all ROC analyses included sufficient numbers of children with both positive and negative outcomes to achieve an AUC of .70.

ROC analyses also generate sensitivity and specificity values which, in the context of this study, indicate the ability of each questionnaire (CAIS/SCAS) to correctly identify children who have recovered from a diagnosis and children who have retained a diagnosis, respectively, on a scale of 0 to 1. In the present context, a higher cut-off score leads to greater sensitivity and lower specificity whereas a lower cut-off score results in lower sensitivity but greater specificity. To minimize the risk of children who retained diagnoses being incorrectly classified as recovered, the highest cut-off score with a specificity of at least .70 was selected for each measure.

Identifying Recovery From Primary Diagnosis With Time 2 SCAS-C/P and CAIS-C/P Scores

As shown in Table 3 , the only measure to achieve an AUC exceeding .70 for identifying loss of primary diagnosis was the CAIS-P at a cut-off score of 8. However, the relatively poor sensitivity at this cut-off (.61) limits the extent to which it can be considered useful for this outcome as it indicates that a considerable proportion who recovered from their primary diagnosis actually had CAIS-P scores exceeding 8.

Identifying Recovery From Particular Anxiety Diagnoses With Time 2 SCAS-C/P and CAIS-C/P Scores

As shown in Table 3 , recovery from separation anxiety disorder was successfully identified by both child (AUC = .80) and parent (AUC = .82) reports on the SCAS separation anxiety subscale. The (total) CAIS-P score was only marginally less successful (AUC = .79) at a cut-off point of 12. All three of these measures also achieved acceptable sensitivity (.70–.76) and specificity (.71–.73) at their respective cut-off scores, meaning they can be considered as moderate to good at identifying recovery from separation anxiety disorder. Although both SCAS-C and SCAS-P total scores achieved moderate AUC values for identifying loss of separation anxiety disorder, the relatively poor sensitivity at their cut-off scores (.61 and .62, respectively) limits the extent to which they can be considered useful for assessing this outcome.

Recovery from social phobia was identified well by CAIS-P (AUC = .78) with good sensitivity (.78) and acceptable specificity (.71) at a cut-off score of 13. However, both SCAS-P total and social phobia subscales had poor sensitivity (.55 and .56, respectively) at their respective cut-off scores.

The only measure to achieve an AUC exceeding .70 for identifying recovery from generalized anxiety disorder was the CAIS-P (AUC = .76), at a cut-off score of 12. This was associated with acceptable sensitivity (.70) and specificity (.72), suggesting that this measure is useful for identifying recovery from generalized anxiety disorder.

Although child and parent reports on the SCAS physical injury subscale achieved moderate AUC values for identifying recovery from specific phobias, the poor sensitivity (.46 and .48, respectively) at a cut-off score of 2.5 prevents the conclusion that either accurately identify this outcome. No other measure accurately identified recovery from specific phobias.

Discussion

The parent-report CAIS score successfully identified absence of any anxiety diagnosis as established using the standard administration of the ADIS-C/P, albeit with a low cut-off score. Although no measure was particularly successful at identifying whether children recovered from their primary diagnosis, at least one of the questionnaire measures successfully identified recovery from each of the specific diagnoses considered here (apart from specific phobia). In general parent-report scores tended to be better at identifying diagnostic outcomes from the ADIS-C/P than child scores, with the CAIS-P performing best.

There is a paucity of research using ROC methods to evaluate self-report measures as indices of diagnostic outcomes, limiting comparison with other findings. The superior AUC, sensitivity, and specificity values found for the PARS at identifying recovery according to ADIS-IV-C/P ( Caporino et al., 2013 ) are understandable given that they are both clinician-rated measures. However, the present findings demonstrate that questionnaire measures can be used as acceptable indicators of some ADIS-C/P diagnostic outcomes on the basis of having AUC values of at least 0.70 and acceptable sensitivity and specificity. Specifically, based on the data sets used here, cut off-scores of the CAIS-P and separation anxiety subscale of the SCAS-C/P can be recommended to assess recovery for children with separation anxiety disorder (CAIS-P < 12; SCAS-C-SA <6; SCAS-P-SA <7), with CAIS-P cut-off scores recommended to assess recovery from social phobia (CAIS-P < 13) and generalized anxiety disorder (CAIS-P < 12). Notably, although the separation anxiety subscale of the SCAS-C/P performed well at identifying recovery from separation anxiety disorder, the social phobia subscale of the SCAS-C/P was not a good indicator of recovery from social phobia and nor was the generalized anxiety subscale a good indicator of recovery from generalized anxiety disorder. Furthermore, neither the SCAS-C/P (total or physical injury subscale) nor the CAIS-C/P was accurate at identifying recovery from specific phobias. This may well be accounted for by the fact that impairment relates to restricted fear domains and contexts and more closely tailored measures may be required, which is also likely to account for the relatively low internal consistency of the SCAS specific phobia scale.

The superior performance of the CAIS over the SCAS for the majority of outcomes may well reflect the fact that meeting diagnostic criteria is dependent on there being significant functional impairment ( APA, 1994 , 2013 ). It is also important to note that the CAIS-P cut-off point for identifying children with no anxiety diagnoses (i.e., complete recovery) was markedly lower than those for the individual anxiety diagnoses. This is likely to reflect the high comorbidity of anxiety disorders in children ( Waite & Creswell, 2014 ), meaning that a child who has recovered from their primary or any specific diagnosis may well retain at least one other anxiety diagnosis.

Parent reports were generally better at identifying diagnostic outcomes on the ADIS-C/P than children’s self-reports: consistent with previous findings of superior accuracy of mother rather than child report as screening tools for child anxiety ( Villabø et al., 2012 ). This may potentially relate to limitations associated with children’s reflective abilities, recognition of the broader interference (e.g., to family life) of their symptoms, and/or to anxious children’s tendency to ‘fake good’ ( Kendall & Chansky, 1991 ). Although we restricted our analyses to a relatively narrow age group (7–12 years), it is likely that the utility of child report will vary substantially within this age group. We would also anticipate that quite different rates of diagnostic accuracy would be found with older populations (i.e., adolescents) and future studies are required with this age group. It is also the case that the superiority of parent report may reflect the fact that because this was a referred sample parents will have been aware of their child’s symptoms and the associated functional impairments. Whether similar findings would be found in a longitudinal community study remains unclear.

A further possibility is that the greater agreement between parent (compared with child) report questionnaire scores and diagnostic outcomes reflected the tendency for poor parent–child agreement on the ADIS ( Choudhury, Pimentel, & Kendall, 2003 ; Comer & Kendall, 2004 ; Grills & Ollendick, 2003 ) and for diagnostic outcomes of the ADIS interviews to be more closely related to parent reports ( Grills & Ollendick, 2003 ). We used the standard administration of the ADIS-IV-C/P, that is, diagnoses were given if either parent or child report met diagnostic criteria. As highlighted by existing literature (e.g., Grills & Ollendick, 2003 ; Hawley & Weisz, 2003 ) it is possible, or indeed likely, that the stronger performance of parent reports on questionnaires in predicting recovery in the present study reflects a tendency for final ADIS-C/P diagnoses to reflect parent reports to a greater extent than child reports. In the present study, children and parents showed moderate agreement (62% to 81% across diagnoses) although final clinician-awarded ADIS-IV-C/P diagnoses did more closely reflect parent reports (87% to 92% across diagnoses) than child reports (68% to 86% across diagnoses). This is not to say that child reports are unimportant in assessing recovery from anxiety disorders, and the results of the present study should be considered in the context of the greater parental influence on ADIS-IV-C/P outcomes when administered in its standard form with preadolescent children. Further research would benefit from the addition of more objective assessments of recovery from child anxiety disorders which are independent of child and parent reports (e.g., using behavioral observations).

A number of study limitations