The diagnostic adaptive behavior scale: Evaluating its diagnostic sensitivity and specificity

See discussions, stats, and author profiles for this publication at: http://www.researchgate.net/publication/264629826

The Diagnostic Adaptive Behavior Scale: Evaluating its diagnostic sensitivity and specificity

ARTICLE in RESEARCH IN DEVELOPMENTAL DISABILITIES · AUGUST 2014

Impact Factor: 4.41 · DOI: 10.1016/j.ridd.2014.07.032 · Source: PubMed

DOWNLOADS

VIEWS

278

288

9 AUTHORS, INCLUDING:

Giulia Balboni

Scott Spreat

Università di Pisa

Woods Services, Inc.

36 PUBLICATIONS 149 CITATIONS

74 PUBLICATIONS 771 CITATIONS

SEE PROFILE

Keith Widaman

Patricia Navas

University of California, Davis

University of Zaragoza

174 PUBLICATIONS 9,896 CITATIONS

19 PUBLICATIONS 44 CITATIONS

SEE PROFILE

Available from: Giulia Balboni Retrieved on: 31 July 2015

Contents lists available at ScienceDirect

Research in Developmental Disabilities 35 (2014) 2884–2893

Research in Developmental Disabilities

The Diagnostic Adaptive Behavior Scale: Evaluating its diagnostic sensitivity and speciﬁcity

Giulia Balboni a,*, Marc J. Tasse´ b, Robert L. Schalock c, Sharon A. Borthwick-Duffy d, Scott Spreat e, David Thissen f, Keith F. Widaman g, Dalun Zhang h, Patricia Navas b

a University of Pisa, Via Roma, 67, 56126, Pisa, Italy b The Ohio State University, 1581 Dodd Dr., Columbus, OH 43210, USA c Hastings College, P.O. Box 285, Chewalah, WA 99190-0285, USA d University of California – Riverside, 20500 Yate Circle, Riverside, CA 92508-3126, USA e Woodland Center for Challenging Behaviors, 9 Imlaystown-Hightstown Road, Allentown, NJ 08501-2011, USA f University of North Carolina – Chapel Hill, CB #3270, Chapel Hill, NC 23599-3270, USA g University of California – Davis, One Shields Avenue, Davis, CA 95616-8686, USA h Texas A&M University, 4225 TAMU, College Station, TX 77843-4225, USA

A R T I C L E I N F O

A B S T R A C T

Article history: Received 5 June 2014 Accepted 14 July 2014 Available online

The Diagnostic Adaptive Behavior Scale (DABS) was constructed with items across three domains – conceptual, social, and practical adaptive skills – and normed on a representative sample of American individuals from 4 to 21 years of age. The DABS was developed to focus its assessment around the decision point for determining the presence or absence of signiﬁcant limitations of adaptive behavior for the diagnosis of Intellectual Disability (ID). The purpose of this study, which was composed of 125 individuals with and 933 without an ID-related diagnosis, was to determine the ability of the DABS to correctly identify the individuals with and without ID (i.e., sensitivity and speciﬁcity). The results indicate that the DABS sensitivity coefﬁcients ranged from 81% to 98%, speciﬁcity coefﬁcients ranged from 89% to 91%, and that the Area Under the Receiver Operating Characteristic Curve were excellent or good. These results indicate that the DABS has very good levels of diagnostic efﬁciency.

(cid:2) 2014 Published by Elsevier Ltd.

1. Introduction

All current diagnostic and classiﬁcation systems (Diagnostic and Statistical Manual of Mental Disorders, DSM-5; American Association on Intellectual and Developmental Disabilities’ [AAIDD] Terminology & Classiﬁcation Manual; and International Statistical Classiﬁcation of Diseases and Related Health Problems, ICD-10) have three common criteria to rule- in or rule-out a diagnosis of Intellectual Disability (ID): (a) signiﬁcant limitations in intellectual functioning, (b) signiﬁcant limitations in adaptive behavior, and (c) an age of onset during the developmental period (American Psychiatric Association[APA] 2013; Schalock et al., 2010; World Health Organization, 1993).

Keywords: Adaptive behavior Intellectual disability Sensitivity Speciﬁcity

* Corresponding author at: Department of Surgery, Medical, Molecular & Critical Area Pathology, University of Pisa, Via Roma, 67, 56126 Pisa, Italy. Tel.: +39 050 992370; fax: +39 050 992658. E-mail address: giulia.balboni@med.unipi.it (G. Balboni).

http://dx.doi.org/10.1016/j.ridd.2014.07.032 0891-4222/(cid:2) 2014 Published by Elsevier Ltd.

Adaptive behavior has been formally included in deﬁning the condition of ID for at least the past 50 years (see Heber, 1959, 1961; Schalock et al., 2010). As discussed in Tasse´ et al. (2012), we have today essentially the same deﬁnition of adaptive behavior as originally proposed by Heber (1959) where adaptive behavior was deﬁned as the collection of conceptual, practical, and social adaptive skills. Both the AAIDD (Schalock et al., 2010) and the DSM-5 (APA, 2013) deﬁne adaptive behavior similarly. Moreover, there is agreement in deﬁning signiﬁcant limitations in adaptive behavior as performance that is approximately two standard deviations below the mean (i.e., represented by a standard score of approximately 70 or less) in at least one of the three adaptive behavior domains of conceptual, social, or practical adaptive skill or in a total score from an adaptive behavior scale (APA, 2013; Schalock et al., 2010).

With the advent of the intelligence tests movement, Intelligence Quotient (IQ) scores quickly became the predominant criterion for diagnosing persons as having ID as well as planning interventions and services for this population, overshadowing consideration of the person’s adaptive behavior skills and deﬁcits. Perhaps due to debate over the precise structure of adaptive behavior (Bruininks, McGrew, & Maruyama, 1988; Harrison, 1989; Meyers, Nihira, & Zetlin, 1979; Widaman, Borthwick-Duffy, & Little, 1991; Widaman & McGrew, 1996) and even its conceptualization (e.g., Greenspan & Granﬁeld, 1992), the importance of the role of IQ scores increased. However, since 2002 (Luckasson et al., 2002), the relevance of the construct of adaptive behavior and the role it should play in a diagnosis of ID has clearly been reafﬁrmed. Despite being an integral part of diagnosing intellectual disability, a relative paucity of research studies have investigated the accuracy with which standardized adaptive behavior measures accurately identify persons with ID as having signiﬁcant limitations in adaptive behavior. A key issue in such studies in identifying those behaviors that best distinguish individuals with ID from those without ID. Although more than 200 adaptive behavior measures have previously been identiﬁed (Schalock, 1999), only four of them are normed on a representative U.S. sample of the general population, and few have been developed speciﬁcally for the purpose of ruling in/out a diagnosis of ID (Tasse´ et al., 2012). The four instruments normed on representative US samples are: (1) Vineland Adaptive Behavior Scales-2nd edition, VABS-II (Sparrow, Cicchetti, & Balla, 2005); (2) Adaptive Behavior Assessment System-II, ABAS-II (Harrison & Oakland, 2003); (3) Scales of Independent Behavior- Revised, SIB-R (Bruininks, Woodcock, Weatherman, & Hill, 1996); and (4) Adaptive Behavior Scale-School Version, ABS-S:2 (Lambert, Nihira, & Leland, 1993). Some of these (e.g., ABS & SIB-R) have not been re-normed in 2 decades.

When reviewing the psychometric properties of these measures, accuracy to correctly identify persons with ID from individuals without ID is a key aspect. The accuracy of differential diagnosis may be evaluated comparing the standard scores of assessed individuals with and without and ID diagnosis and computing diagnostic efﬁciency (or validity) statistics (e.g., Streiner, 2003) such as sensitivity and speciﬁcity. Sensitivity is deﬁned as the proportion of true positives (i.e., person having ID) that are correctly identiﬁed as having ID by the test (Altman & Bland, 1994), reﬂecting how good a test is at correctly identifying people who have the condition the test is intended to measure (Loong, 2003). Speciﬁcity refers to the ability to correctly reject a diagnosis or identify true negatives, when the person is known not to have the said diagnosis (Altman & Bland, 1994).

Information regarding the sensitivity and speciﬁcity of an adaptive behavior assessment instrument is critical in establishing the valid use of the instrument. For example, in the case of the ABS-S: 2, the authors stated simply that ‘‘mean scores for the groups with developmental disabilities are sufﬁciently below the average scores of the normal group’’ (Lambert et al., 1993, p. 50). However, the actual percentages of people correctly classiﬁed were not provided, so the precision of the ABS-S: 2 to accurately measure signiﬁcant limitations in adaptive behavior and correctly identify someone as having an ID is not known.

The SIB-R Comprehensive Manual (Bruininks et al., 1996) provided a more detailed description of sensitivity and speciﬁcity. In the manual, the authors stated that 76% of the individuals within the standardization sample were correctly classiﬁed into their original groups (51% mild ID; 74% moderate ID; and 82% non-ID). From a decision-making perspective, the fact that the lowest degree of accuracy (i.e., 51%) of the SIB-R is for the group with mild ID is not surprising, as this group is the closest to the actual diagnostic cut-off score. However, if only about half of persons diagnosed with mild ID have scores in a range that would support diagnosis, the SIB-R may not be sufﬁciently sensitive to identify correctly persons with mild forms of ID. Furthermore, although the SIB-R covers the age span (from infancy and up), sensitivity and speciﬁcity data were not provided for separate age groups.

For the VABS-II (Sparrow et al., 2005), the percentages of people correctly classiﬁed ranged from 71% to 100% for those aged 6–18 years old (71% mild ID; 87% moderate ID; and 100% severe and profound ID), and from 97% to 100% for individuals aged 19–86 (97% mild ID and 100% for individuals with moderate and with severe and profound ID). Data regarding the sensitivity of the VABS-II were also available by domain (i.e., communication, daily living skills, socialization, and motor skills). However, the proportion of people who were correctly excluded from the diagnosis of ID (i.e., speciﬁcity) was not reported.

The ABAS-II (Harrison & Oakland, 2003) offers data on both sensitivity and speciﬁcity across four infant-preschool samples, ﬁve school-age samples, and one adult sample. The results for the infant-preschool samples indicate that 58–73% of those with ID (sensitivity), and 0–5% without ID (speciﬁcity) scored at least 2 standard deviations below the mean on the General Adaptive Composite (GAC) score, whereas 77–86% children diagnosed with ID and 3–19% of the matched control group scored at least 2 standard deviations below the mean on one or more adaptive domains or the GAC. Across the ﬁve school-age samples and the adult sample 50–87% of those with ID and 0–17% of the matched control group scored at least 2 standard deviations bellow the mean on the GAC, and 62–100% with ID and 5–19% without ID scored at least 2 standard deviations below the mean on one or more adaptive domains or the GAC.

G. Balboni et al. / Research in Developmental Disabilities 35 (2014) 2884–2893 2885

Although the four standardized measures of adaptive behavior brieﬂy reviewed above are appropriate instruments for making a determination of ID, all were generally developed using Classical Test Theory (CTT) models (Tasse´ et al., 2012). In distinction, the Diagnostic Adaptive Behavior Scale (DABS; Tasse´ et al., in preparation) used an Item Response Theory (IRT) model. Unlike CTT, IRT models provide guidance in test development and allow scale developers to tailor the efﬁciency of the instrument for a speciﬁc ability level of the trait that is measured. For example, given the intended use of an adaptive behavior to provide accurate and precise scores near the cutoff point for diagnosis (i.e., 2 SD below the mean), an IRT approach to scale construction can build up precision at a given point on the standard score scale, which should yield more sensitive measures for diagnostic decisions (Hays, Morales, & Reise, 2000). Moreover, IRT models provide a better estimate of the individual’s true score (Santor & Ramsay, 1998), which is described as a function of the trait (e.g., adaptive behavior) being measured and test item parameters (Thissen, Nelson, Rosa, & McLeod, 2001), such as each item’s level of difﬁculty and discrimination strength.

2. Study purpose

The purpose of this study was to evaluate the DABS with regard to its sensitivity and speciﬁcity. The DABS is an adaptive behavior assessment instrument that was developed using IRT (Graded Response Model for ordinal polytomous data [Samejima, 1969]), focusing its item pool on the items that provide the most precise information for the purpose of making a diagnosis of ID. IRT models were used to estimate the person’s adaptive functioning and item parameters to identify a ﬁnal set of items that provide the most information around the cut-off point for determining signiﬁcant limitations in adaptive behavior (i.e., approximately 2 standard deviations below the population mean) from 4 to 21 years old (Tasse´ et al., in preparation).

The goals of the present paper include:

(1) comparing the DABS standard score of assessed individuals with and without and ID diagnosis and determining sensitivity and speciﬁcity of the DABS to correctly identify persons with an ID diagnosis from individuals who do not have an ID diagnosis; and

(2) evaluating the sensitivity and speciﬁcity across age groups: 4–21 years old.

3. Method

3.1. Participants

The participants comprised individuals assessed on the DABS during the standardization phase. The sample was composed of 1058 persons, aged between 4 and 21 years old (M = 11.1, SD = 4.9), and 51% were male. Individuals in the standardization sample came from 46 American states, with none from Alaska, Arkansas, North Dakota, or West Virginia, and 71.5% were white.

As can be seen in Table 1, 125 individuals (12%) had a formal diagnosis of mental retardation/ID or developmental delay and were classiﬁed as ‘‘participants with an ID-related diagnosis’’ or the ‘‘ID group.’’ The remaining 933 (88%) were reported as not having any signiﬁcant deﬁcits in intellectual functioning and adaptive behavior. Of these individuals, 20% had another veriﬁed condition that was different from an ID-related diagnosis (e.g., ADHD, learning disability, etc.). These individuals were classiﬁed as ‘‘participants without an ID-related diagnosis’’ or the ‘‘non-ID group.’’

The participants were divided into three age groups: 4–8, 9–15, and 16–21 years old (Table 2). In each age group, about 50% were male, and 8%, 9%, and 21%, respectively, had a formal ID-related diagnosis. No signiﬁcant age or gender differences

G. Balboni et al. / Research in Developmental Disabilities 35 (2014) 2884–2893 2886

Table 1 Conditions of the participants. n Condition Percentage

125 933 ID-related diagnosis Non-ID 12 88

Typically developing Other veriﬁed conditions 80 20

Prevalence (non-cumulative) ADHD Autism Spectrum Disorder Learning disability Language impairment Emotional disturbance Hearing impairment Visual impairment Other health impairment 6 4 4 3 3 1 1 3

G. Balboni et al. / Research in Developmental Disabilities 35 (2014) 2884–2893 2887

Table 2 Age and gender of participants with and without and ID-related diagnosis within each of the three age groups. Age groups

4–8 (n = 388) 9–15 (n = 432) 16–21 (n = 238)

between the ID and non-ID groups were found, with two exceptions: (a) a gender difference (more male participants) in the ID versus the non-ID participants in the 9–15 year old group, x2 (1) = 6.92, p < .01; and (b) an age difference in the 16–21 year old group, in which participants with ID were slightly older than their peers without ID, t(236) = 1.99, p < .05.

3.2. Instrument

The DABS was developed over a 6-year period with the aim of making available an adaptive behavior test that is focused on providing diagnostic information for ID. ID was deﬁned as a condition that is: ‘‘. . . characterized by signiﬁcant limitations in both intellectual functioning and in adaptive behavior as expressed in conceptual, social, and practical adaptive sills. This disability originates before age 18’’ (Schalock et al., 2010, p. 1). The DABS reﬂects both the theoretical and empirical work done over the last two decades, which has supported the 3-factor structure of adaptive behavior as deﬁned in Schalock et al. (2010) as: ‘‘the collection of conceptual, social, and practical skills that have been learned by people in order to function in their everyday lives’’ (Schalock et al., 2010, p. 15) (Arias, Verdugo, Navas, & Go´ mez, 2013; Navas, Verdugo, Arias, & Go´ mez, 2012).

The DABS administration is done via a face-to face interview between a professional and a respondent who knows the assessed person very well. The individual’s performance of each adaptive skill is assessed according to a four-point rating scale from: 0 (rarely or never does it) to 3 (does it always or almost always independently) and all the items on the DABS must receive a rating.

The interviewer of the DABS is described as a professional (e.g., psychologist, teacher, case manager, social worker) or other professional who has completed at least a Bachelor’s degree, has had direct work experience with people with an intellectual or closely related developmental disability, and who has had previous individual assessment experience. The DABS should be completed with as many respondents as the interviewer deems necessary to provide the most valid and complete assessment of the assessed person’s adaptive behavior. Respondents should inform about individual’s typical performance during daily routines and changing circumstances.

As described further in Tasse´ et al. (submitted for publication), IRT analyses yielded a DABS ﬁnal version with a total of 75 items for each of the three age groups (4–8; 9–15; 16–21 years old), consisting of 25 items per domain (conceptual, social, and practical skills). Domain scores for each of the three adaptive skill areas as well as an Overall Adaptive Behavior Standard Score (OABSS) are reported on a standardized scale with a mean of 100 and standard deviation of 15. Therefore, the ‘‘2 SDs below the mean’’ cut-off score for the diagnosis of ID is equal to a standard score of approximately 70. However, the evaluator should also consider the variability of the individual’s observed score caused by several potential sources of measurement errors. The average SEM of the DABS is equal to approximately (cid:2)3 points for each domain standard score (i.e., conceptual, social, and practical skills) and (cid:2)2 points for the OABSS. Using a 95% conﬁdence interval, the ID diagnosis cut-off represents domain standard score that might go up to 76 (70 + 2 SEM) and 74 (70 + 2 SEM) for the OABSS.

3.3. Procedure

Data collection for the DABS standardization was conducted between 2008 and 2011. Participants were recruited nationally through several waves using electronic media, listservs, mailings, and web postings from various professional societies and national disability groups, including: AAIDD, Association of University Centers on Disabilities (AUCD), National Association of School Psychologists (NASP), and APA – Division 33, to name a few.

DABS accuracy in the measurement of signiﬁcant limitations in adaptive behavior was investigated using: (1) comparison of the standard score of assessed individuals with and without an ID-related diagnosis on each domain (i.e., conceptual, social, and practical skills) and the OABSS; and (2) estimation of sensitivity and speciﬁcity in correctly identifying assessed individuals with and without an ID-related diagnosis based on their DABS scores (i.e., a standard score on one of the three domains or the OABSS that was at or below 76 or 74, respectively).

ID-related (n = 32) Non ID (n = 356) Non ID (n = 390) Non ID (n = 187) ID-related (n = 42) ID-related (n = 51) Age Mean (SD) 6.25 (1.32) 5.98 (1.41) 12.00 (2.17) 11.65 (1.98) 18.59 (1.56) 18.08 (1.63) Gender (%) M–F 66–34 48–52 69–31 48–52 53–47 51–49

3.4. Analyses

Comparisons of standard scores were performed using t-tests. When statistically signiﬁcant differences were found, Cohen’s d was computed (Cohen, 1988), and the effect sizes were interpreted according to the following criteria: negligible (<.20), small (.20–.49), medium (.50–.79), and large ((cid:3).80) effect size.

Test sensitivity was calculated as the proportion of individuals with an ID-related diagnosis who were correctly identiﬁed by the DABS (i.e., who obtained a standard score on one of the three domains that was at or below 76 or an OABSS that was at or below 74). Test speciﬁcity was calculated as the proportion of individuals without an ID-related diagnosis who did not have a domain standard score at or below 76 or an OABSS at or below 74.

One can combine the sensitivity and speciﬁcity into a single statistic called Overall Correct Classiﬁcation (OCC). The OCC is deﬁned as the proportion of all individuals with and without an ID-related diagnosis who were correctly identiﬁed and detected by the DABS, respectively (e.g., McFall & Treat, 1999; Streiner, 2003).

If expressed in percentage form, sensitivity, speciﬁcity, and OCC are determined to be statistically signiﬁcant if their CIs do not include the value of 50% (i.e., if they are more accurate than a random classiﬁcation). Matthey and Petrovski (2002) suggested that sensitivity coefﬁcients (cid:3)70% and speciﬁcity coefﬁcients (cid:3)80% are considered appropriate benchmarks to be attained by diagnostic tests. Higher values of sensitivity, speciﬁcity, and OCC are always desirable.

Another way to measure sensitivity and speciﬁcity in discriminating participants with and without ID based on their DABS scores was estimated using Receiver Operating Characteristic (ROC) curves analyses. The ROC is computed by plotting the true positive rate (sensitivity) against the false positive rate (1 – true negatives or 1 – speciﬁcity) for different possible cut points on the DABS (e.g., Altman & Bland, 1994; McFall & Treat, 1999; Streiner, 2003). The Area Under the Curve (AUC) is equal to the probability of a test giving a correct diagnosis (i.e., correctly identifying individuals with and without ID). In other words, the AUC reﬂects the probability that a randomly selected person from the ID group will have a standard score below the standard score of a randomly selected person from the non-ID group. AUC values range from 0 to 1 and are statistically signiﬁcant if their CIs do not include the value of .5. Swets (1998) suggested the following parameters when interpreting AUC values: .5–.6 = fail; .6–.7 = poor; .7–.8 = fair; .8–.9 = good; and .9–1.0 = excellent. For each age group, comparisons within the AUCs of the standard scores for the DABS Overall Adaptive Behavior and the conceptual, social, and practical domains were conﬁrmed by analyzing the presence or absence of overlaps between the corresponding CIs.

4. Results

4.1. Comparisons of DABS standard score obtained by participants with and without an ID-related diagnosis

As shown in Table 3, the mean OABSS for those within the ID group aged 4–21 years old was signiﬁcantly below (p < .001) the mean OABSS of the assessed individuals within the non-ID group aged 4–21 years old (with effect sizes greater than 2.20, meaning that the difference is larger than two standard deviations). As expected (see Fig. 1), all the individuals with an ID- related diagnosis, with the exception of those who were 10 years old, had a mean OABSS below the ID diagnosis cut-off standard score of 74. The mean OABSS was approximately 100 for those without an ID-related diagnosis, regardless of their age range. Standard deviations (see Table 3) were approximately 15 for the non-ID groups (i.e., 4–8; 9–15; 16–21 years old). Consistent with the greater variability in adaptive behavior presented by individuals with ID, standard deviations were larger for individuals within ID-groups (ranging from 17.05 to 19.34). Also, on the conceptual, social and practical skills domains (see Table 3), participants within the ID group obtained mean standard scores for the conceptual, social and practical skills domains that were signiﬁcantly lower (p < .001) than the mean standard score obtained by the participants without an ID-related diagnosis (with large effect sizes) at each age level. Fig. 2 presents the mean standard scores for

2888 G. Balboni et al. / Research in Developmental Disabilities 35 (2014) 2884–2893

Table 3 Comparisons of the standard score obtained by the participants of each of the three age groups with and without and ID-Related diagnosis on the domains and on the Overall Adaptive Behavior scale: standard score Mean (SD), t-test and Cohen’s d values. Age groups 4–8 16–21 9–15 ID (n = 32) ID (n = 51) Non-ID (n = 356) t value (Cohen’s d) Non-ID (n = 390) t value (Cohen’s d) Non-ID (n = 187) t value (Cohen’s d) ID (n = 42) Domains Conceptual 9.00 (2.33) 13.99 (2.29) 18.56 (2.91)

Social 10.11 (1.87) 11.21 (1.83) 14.65 (2.33)

Practical 7.87 (1.84) 10.93 (1.79) 15.43 (2.47)

OABSS 12.23 (2.26) 13.56 (2.22) 18.46 (2.97) 64.22 (22.39) 73.27 (13.27) 72.72 (19.23) 66.49 (17.94) 100.55 (14.89) 100.81 (14.87) 100.17 (14.55) 100.60 (14.83) 99.93 (15.04) 99.91 (15.09) 100.39 (15.37) 100.03 (15.17) 54.70 (14.44) 63.73 (15.99) 60.88 (16.61) 53.33 (17.05) 99.54 (15.51) 99.44 (15.27) 99.16 (15.45) 99.30 (15.40) 64.88 (18.69) 72.10 (16.88) 72.16 (20.28) 65.64 (19.34) Note. All the differences are statistically signiﬁcant (p < .001); OABSS, Overall Adaptive Behavior standard score.

[ ( F i g . _ 1 ) T D $ F I G ] G. Balboni et al. / Research in Developmental Disabilities 35 (2014) 2884–2893 2889

participants with and without an ID-related diagnosis. This ﬁgure shows that, generally, the standard scores for individuals with an ID-related diagnosis fell under the ID cut-off score of 76. The only exceptions noted were for 4, 5, 9, and 10 years old participants for the Social skills domains and for the 4, 10 and 15 years old participants for the Practical skills domain, for which the mean standard score was higher than the cut-off point. However, for these age groups the mean OABSS was generally below the ID cut-off point.

4.2. Sensitivity and speciﬁcity of the DABS

False negatives (i.e., participants with an ID-related diagnosis who obtained DABS standard scores above the ID diagnosis cut-off point) and false positives (i.e., participants without an ID-related diagnosis who obtained DABS standard scores below the ID diagnosis cut-off point), sensitivity, speciﬁcity, and OCC are reported in Table 4. The DABS yielded sensitivity coefﬁcients ranging from 81% to 98%, speciﬁcity coefﬁcients ranging from 89% to 91%, and OCC coefﬁcients ranging from 89% to 92%. Their CIs did not include the value of 50% (i.e., they were statistically signiﬁcant) for any age group. Sensitivity and speciﬁcity were higher than the clinical signiﬁcance value of 70% (except one value of 68%) and 80%, respectively.

Fig. 1. DABS Overall Adaptive Behavior standard score of participants with and without an ID-related diagnosis (ID diagnosis cut-off standard score is 74 standard score).

[ ( F i g . _ 2 ) T D $ F I G ]

Fig. 2. DABS conceptual, social and practical skills standard score of participants with and without an ID-related diagnosis (ID diagnosis cut-off standard score is 76 standard score).

G. Balboni et al. / Research in Developmental Disabilities 35 (2014) 2884–2893 2890

Table 4 False negative and positive, sensitivity, speciﬁcity and OCC (CI) of the DABS for the three age groups(%). Age groups False negative (CI) False positive (CI) Sensitivity (CI) Speciﬁcity (CI) OCC (CI) n

Moreover, 48% (4–8 years old), 74% (9–15 years old), and 74% (16–21 years old) of the cases that were identiﬁed as false positives had some veriﬁed condition other than ID (for example, learning disability) that might explain the signiﬁcant limitations in adaptive behavior.

The ROC Curves and AUCs are reported in Figs. 3–5, and in Table 5. The CIs of the AUCs for all ages assessed on the DABS were all above .5. Therefore, the DABS is signiﬁcantly more accurate than a random classiﬁcation in identifying participants with and without ID. Clinical signiﬁcance was interpreted using the CIs of the AUCs. The OABSS for the 4–8 and 16–21 year- old groups and all three domains on the 16–21 year-old group had ‘‘excellent’’ discriminant ability in identifying individuals with and without ID (i.e., AUC CI was on the right side of .90). The OABSS for the 9–15 year-old age group and the other remaining domains had discriminant ability that was ‘‘good’’ (i.e., AUC CI was on the right side of .80 and included the value of .90). Only the practical domain for the 9–15 year-old age group had discriminant ability that was in the ‘‘fair’’ range (i.e., AUC CI was on the right side of .70 and included the value of .80).

The CIs of AUCs for the OABSS and the conceptual, social, and practical domains showed clear overlap across all three DABS forms, indicating that no statistically signiﬁcant differences were found and that all forms have comparable discriminant ability in correctly differentiating the participants into the ID versus non-ID groups.

5. Discussion

18.75 (5.23–22.95) 14.29 (3.70–17.94) 1.96 (0–4.05) 8.71 (5.78–11.64) 11.03 (7.92–14.13) 10.16 (5.83–14.49) 81 (68–95) 86 (75–96) 98 (94–100) 91 (88–94) 89 (86–92) 90 (85–94) 90 (87–93) 89 (86–92) 92 (88–95) 4–8 9–15 16–21 338 432 238 Note. CI, 95% conﬁdence interval; OCC, overall correct classiﬁcation.

The multidimensional framework for understanding adaptive behavior (Schalock et al., 2010) has signiﬁcant implications for researchers, disability-related government agencies, and services providers. Making a diagnosis of ID inevitably is a high stakes decision for various outcomes (e.g., eligibility for accommodations or special education services and supports, waiver services, social security beneﬁts, exemption from the death penalty) (e.g., Pogge, Stokes, Buccolo, Pappalardo, & Harvey, 2014). Therefore, it is crucial to use valid adaptive behavior scales that enable accurate assessments of signiﬁcant adaptive skill limitations necessary for an ID diagnosis. A standard score of approximately 70 (plus or minus the standard error of measurement) obtained on a comprehensive adaptive behavior test is the cut-score often used in deﬁning signiﬁcant deﬁcits in adaptive behavior and a critical element in making a determination of ID (APA, 2013; Schalock et al., 2010). The four other, well-accepted standardized adaptive behavior tests used in making a diagnosis of ID (Vineland Adaptive Behavior Scale-II, Adaptive Behavior Assessment System-II, Scales of Independent Behavior-Revised, Adaptive Behavior Scale: School) were developed to provide adaptive behavior information for both (1) the diagnosis of ID, and (2) planning intervention (Tasse´ et al., 2012). The DABS was developed with the more restricted focus on providing the most robust and reliable information

1.0

Conceptual

Social

Practical

Total AB

[ ( F i g . _ 3 ) T D $ F I G ]

y t i v i t i s n e S

1.0

1 - Specificity

Fig. 3. Receiver operating characteristics (ROC) curves of DABS in 4–8 years old group.

Conceptual

1.0

Social

Practical

Total AB

[ ( F i g . _ 4 ) T D $ F I G ] G. Balboni et al. / Research in Developmental Disabilities 35 (2014) 2884–2893 2891

y t i v

i t i

s n e S

1.0

.4 .6 .5 1 - Specificity

at the cut-score of approximately 2 standard deviations below the population mean for the purpose of ruling-in or ruling-out a diagnosis of ID.

The aim of this paper was to analyze the accuracy of the DABS in providing critical adaptive behavior information for the purpose of ruling-in or out a diagnosis of ID in individuals aged 4–21 years old. Hence, we used two different methods to accomplish this goal: (1) comparing group means on the standard scores; and (2) estimating the sensitivity (i.e., the discriminant ability in correctly identify the individuals with an ID-related diagnosis) and speciﬁcity (i.e., the discriminant ability in correctly identify the individuals without an ID-related diagnosis) of the DABS. Sensitivity and speciﬁcity were estimated ﬁrst by computing the percentage of participants with and without ID who had DABS scores that were below or above the mean, respectively. The sensitivity and speciﬁcity were also estimated using ROC analyses, which yield AUC values which indicate the probability that a randomly selected person from the ID group will have a DABS standard score below the standard score of a randomly selected person from the non-ID group.

Based on the results obtained, the DABS has good to excellent efﬁciency in correctly identifying whether individuals aged 4–21 years old do or do not have a diagnosis of ID. Of the four commonly accepted, well-standardized adaptive behavior scales used in making a diagnosis of ID, the accuracy in identifying individual with ID was investigated estimating only sensitivity and speciﬁcity. When evaluating the accuracy of the DABS, we extended the sensitivity-speciﬁcity analyses with ROC analyses. Our analyses showed that the DABS correctly identiﬁed 81–98% of individuals with ID and 89–91% of

Fig. 4. Receiver operating characteristics (ROC) curves of DABS in 9–15 years old group.

1.0

Conceptual Social Prac(cid:2)cal Total AB

[ ( F i g . _ 5 ) T D $ F I G ]

y t i v i t i s n e S

1.0

1 - Specificity

Fig. 5. Receiver operating characteristics (ROC) curves of DABS in 16–21 years old group.

G. Balboni et al. / Research in Developmental Disabilities 35 (2014) 2884–2893 2892

Table 5 Area under the curve (AUC) (CI) of the DABS in the three age groups. Age groups 4–8 9–15 16–21 Domains

individuals without ID. These coefﬁcients are comparable to or exceed those reported with the other standardized adaptive behavior scales for individuals 4–21 years old.

The ABAS-II (Harrison & Oakland, 2003) reported that school age subjects with an ID scored signiﬁcantly lower (range 56– 73) on the ABAS-II GAC than their typical peers (range 93–101). Harrison and Oakland also reported 50%-87% of the subjects assessed on the ABAS-II who had a diagnosis of ID scored at least 2 standard deviations below the mean on the GAC. They did not, however, report the sensitivity of the ABAS-II in identifying subjects with ID using 2 standard deviations below the mean on at least 1 of the 3 domains independently from the GAC. The ABS:School-2 (Lambert, Nihira, & Leland, 1993) did not report data on its accuracy rate in correctly distinguishing students known to have an ID diagnosis from those who do not.

The DABS does not attempt to provide an exhaustive assessment of an individual’s strengths and limitations in adaptive functioning, and therefore should not be used for the purposes of programming or identifying areas of intervention or needed supports. Other instruments are available and may be used for planning intervention and measure adaptive behavior proﬁle (e.g., Brian, Belva, & Matson, 2013; Matson, Dempsey, & Fodstad, 2009; Matson, Rivet, Fodstad, Dempsey, & Boisjoli, 2009; Soenen, Van Berckelaer-Onnes, Scholte, 2009). The DABS was developed based on IRT model, and contains items that maximally differentiate between individuals 4 and 21 years old with and without ID, and provides reliable behavioral information around the cutoff point for determining signiﬁcant limitations in adaptive behavior. Dedicated efforts were made during scale development to obtain an instrument focused on identifying signiﬁcant limitations in adaptive behavior to aid in the diagnosis of ID. Based on the results of this investigation, the DABS performs as well as, and at times better than, the other four existing adaptive behavior scales used in the diagnosis of ID (i.e., Vineland Adaptive Behavior Scale-II, Adaptive Behavior Assessment System-II, Scales of Independent Behavior-Revised, Adaptive Behavior Scale: School), while having substantially fewer items and reduced administration time.

Acknowledgment

The investigation was partially supported by the Research Grant: ‘‘Development of a Diagnostic Assessment Instrument to Assess Adaptive Behavior.’’ funded by the American Association on Intellectual and Developmental Disabilities (Marc J. Tasse´ , Principal Investigator).

References

Conceptual Social Practical OABSS .929 (.886–.972) .925 (.884–.965) .888 (.836–.940) .942 (.908–.976) .931 (.883–.979) .886 (.833–.938) .866 (.795–.937) .916 (.863–.969) .991 (.983–1.000) .948 (.916–.979) .967 (.947–.987) .984 (.971–.997) Note. CI, 95% conﬁdence interval; OABSS, Overall Adaptive Behavior SS.

Altman, D. G., & Bland, J. M. (1994). Diagnostic tests 2: Predictive values. British Medical Journal, 309(6947), 102. American Psychiatric Association (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Arlington, VA: American Psychiatric Publishing. Arias, B., Verdugo, M. A., Navas, P., & Go´ mez, L. E. (2013). Factor structure of the construct of adaptive behavior in children with and without intellectual disability. International Journal of Clinical and Health Psychology, 13, 155–166. Brian, C., Belva, B. C., & Matson, J. L. (2013). An examination of speciﬁc daily living skills deﬁcits in adults with profound intellectual disabilities. Research in Developmental Disabilities, 34, 596–604. Bruininks, R. H., McGrew, K., & Maruyama, G. (1988). Structure of adaptive behavior in samples with and without mental retardation. American Journal on Mental Retardation, 93, 265–272.

Bruininks, R. H., Woodcock, R., Weatherman, R., & Hill, B. K. (1996). Scales of independent behaviour-revised. Park Allen, TX: DLM Teaching Resources. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: LEA. Greenspan, S., & Grandﬁeld, J. M. (1992). Reconsidering the construct of mental retardation: Implications of a model social competence. American Journal of Mental Retardation, 96, 442–453.

Harrison, P. L. (1989). Adaptive behavior: Research to practice. Journal of School Psychology, 23, 301–313. Harrison, P. L., & Oakland, T. (2003). Adaptive Behavior Assessment System (2nd ed.). San Antonio, TX: Harcourt Assessment. Hays, R. D., Morales, L. S., & Reise, S. P. (2000). Item response theory and health outcomes measurement in the 21st century. Medical Care, 38(Suppl. 9), 28. Heber, R. (1959). A manual on terminology and classiﬁcation in mental retardation. American Journal on Mental Deﬁciency Monographs, 64(2), 1–111. Heber, R. (1961). A manual on terminology and classiﬁcation on mental. Washington, DC: American Association on Mental Deﬁciency. Lambert, N., Nihira, K., & Leland, H. (1993). AAMR adaptive behavior scales – School (ABS-S:2): Examiner’s manual (2nd ed.). Austin, TX: Pro-Ed. Loong, T. W. (2003). Understanding sensitivity and speciﬁcity with the right side of the brain. British Medical Journal, 327(7417), 716. Luckasson, R., Borthwick-Duffy, S., Buntinx, W. H., Coulter, D. L., Craig, E. M. P., Reeve, A., et al. (2002). Mental retardation: Deﬁnition, classiﬁcation, and systems of supports (10th ed.). Washington, DC: American Association on Mental Retardation. Matson, J. L., Dempsey, T., & Fodstad, J. C. (2009). The effect of Autism Spectrum Disorders on adaptive independent living skills in adults with severe intellectual disability. Research in Developmental Disabilities, 30, 1203–1211.

G. Balboni et al. / Research in Developmental Disabilities 35 (2014) 2884–2893 2893

Matson, J. L., Rivet, T. T., Fodstad, J. C., Dempsey, T., & Boisjoli, J. A. (2009). Examination of adaptive behavior differences in adults with autism spectrum disorders and intellectual disability. Research in Developmental Disabilities, 30, 1317–1325.

Matthey, S., & Petrovski, P. (2002). The Children’s Depression Inventory: Error in cutoff scores for screening purposes. Psychological assessment, 14(2), 146. McFall, R. M., & Treat, T. A. (1999). Quantifying the information value of clinical assessments with signal detection theory. Annual Review of Psychology, 50, 215– 241. Meyers, C., Nihira, K., & Zetlin, A. (1979). The measurement of adaptive behavior. In N. R. Ellis (Ed.), Handbook of mental deﬁciency: Psychological theory and research (2nd ed., pp. 431–481). Hillsdale, NJ: Lawrence Erlbaum Associates. Navas, P., Verdugo, M. A., Arias, B., & Go´ mez, L. E. (2012). Development of an instrument for diagnosing signiﬁcant limitations in adaptive behavior in early childhood. Research in Developmental Disabilities, 33, 1551–1559. Pogge, D. L., Stokes, J., Buccolo, M. L., Pappalardo, S., & Harvey, P. D. (2014). Discovery of previously undetected intellectual disability by psychological assessment: A study of consecutively referred child and adolescent psychiatric inpatients. Research in Developmental Disabilities, 35, 1705–1710. Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores (Psychometric Monograph No. 17). Richmond, VA: Psychometric Society Retrieved from http://www.psychometrika.org/journal/online/MN17.pdf.

Santor, D. A., & Ramsay, J. O. (1998). Progress in the technology of measurement: Applications of item response models. Psychological Assessment, 10, 345–359. Schalock, R. L. (1999). Adaptive behavior and its measurement: Implications for the ﬁeld of mental retardation. Washington, DC: American Association on Mental Retardation. Schalock, R. L., Borthwick-Duffy, S. A., Bradley, V. J., Buntinx, W. H., Coulter, D. L., Craig, E. M., et al. (2010). Intellectual disability: Deﬁnition, classiﬁcation, and systems of supports (11th ed.). Washington, DC: American Association on Intellectual and Developmental Disabilities. Soenen, S., Van Berckelaer-Onnes, I., & Scholte, E. (2009). Patterns of intellectual, adaptive and behavioral functioning in individuals with mild mental retardation. Research in Developmental Disabilities, 30, 433–444.

Sparrow, S., Cicchetti, D., & Balla, D. (2005). Vineland-II: Vineland adaptive behavior scales. Survey forms manual. Minneapolis: NCS Pearson Inc. Streiner, D. L. (2003). Diagnosing tests: Using and misusing diagnostic and screening tests. Journal of Personality Assessment, 81(3), 209–219. Swets, J. A. (1998). Measuring the accuracy of diagnostic systems. Science, 240, 1285–1293. Tasse´ , M. J., Schalock, R. L., Balboni, G., Bersani, H., Borthwick-Duffy, S. A., Spreat, S., et al. (2012). The construct of adaptive behavior: Its conceptualization, measurement, and use in the ﬁeld of intellectual disability. American Journal on Intellectual and Developmental Disabilities, 117, 291–303. Tasse´ , M. J., Schalock, R. L., Balboni, G., Bersani, H., Borthwick-Duffy, S. A., Spreat, S., et al. (2014). Diagnostic adaptive behavior scale (DABS). Washington, DC: American Association on Intellectual and Developmental Disabilities (in preparation). Tasse´ , M. J., Schalock, R. L., Balboni, G., Bersani, H., Borthwick-Duffy, S. A., Spreat, S., et al. (2014). Developmental and standardization of the Diagnostic Adaptive Behavior Scale. (submitted for publication). Thissen, D., Nelson, L., Rosa, K., & McLeod, L. D. (2001). Item response theory for items scored in more than two categories. In D. Thissen & H. Wainer (Eds.), Test scoring (pp. 141–186). Mahwah, NJ: Lawrence Erlbaum Associates. Widaman, K. F., & McGrew, K. S. (1996). The structure of adaptive behavior. In W. J. Jacobson & J. A. Mulick (Eds.), Manual of diagnosis and professional practice in mental retardation (pp. 97–110). Washington, DC: American Psychological Association. Widaman, K. F., Borthwick-Duffy, S. A., & Little, T. D. (1991). The structure and development of adaptive behaviors. International Review of Research in Mental Retardation, 17, 1–54. World Health Organization (1993). International statistical classiﬁcation of diseases and related health problems (ICD-10). Geneva, Switzerland: WHO.

The diagnostic adaptive behavior scale: Evaluating its diagnostic sensitivity and specificity