intTypePromotion=1
zunia.vn Tuyển sinh 2024 dành cho Gen-Z zunia.vn zunia.vn
ADSENSE
 » 
 » 

Báo cáo y học: "Are there valid proxy measures of clinical behaviour? a systematic review"

Chia sẻ: Nguyen Minh Thang | Ngày: | Loại File: PDF | Số trang:20

Thêm vào BST
Báo xấu
46
lượt xem 3
download
 
  Download Vui lòng tải xuống để xem tài liệu đầy đủ

Tuyển tập các báo cáo nghiên cứu về y học được đăng trên tạp chí y học quốc tế cung cấp cho các bạn kiến thức về ngành y đề tài: Are there valid proxy measures of clinical behaviour? a systematic review

Chủ đề:

Bình luận(0) Đăng nhập để gửi bình luận!

Lưu

Nội dung Text: Báo cáo y học: "Are there valid proxy measures of clinical behaviour? a systematic review"

  1. Implementation Science BioMed Central Open Access Systematic Review Are there valid proxy measures of clinical behaviour? a systematic review Susan Hrisos*1, Martin P Eccles1, Jill J Francis2, Heather O Dickinson1, Eileen FS Kaner1, Fiona Beyer1 and Marie Johnston3 Address: 1Institute of Health and Society, Newcastle University, 21 Claremont Place, Newcastle upon Tyne, NE2 4AA, UK, 2Health Services Research Unit, University of Aberdeen, Health Sciences Building, Foresterhill, Aberdeen AB25 2ZD, UK and 3Department of Psychology, University of Aberdeen, Health Sciences Building, Foresterhill, Aberdeen AB25 2ZD, UK Email: Susan Hrisos* - susan.hrisos@ncl.ac.uk; Martin P Eccles - martin.eccles@ncl.ac.uk; Jill J Francis - j.francis@abdn.ac.uk; Heather O Dickinson - heather.dickinson@ncl.ac.uk; Eileen FS Kaner - e.f.s.kaner@ncl.ac.uk; Fiona Beyer - fiona.beyer@ncl.ac.uk; Marie Johnston - m.johnston@abdn.ac.uk * Corresponding author Published: 3 July 2009 Received: 14 January 2009 Accepted: 3 July 2009 Implementation Science 2009, 4:37 doi:10.1186/1748-5908-4-37 This article is available from: http://www.implementationscience.com/content/4/1/37 © 2009 Hrisos et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Abstract Background: Accurate measures of health professionals' clinical practice are critically important to guide health policy decisions, as well as for professional self-evaluation and for research-based investigation of clinical practice and process of care. It is often not feasible or ethical to measure behaviour through direct observation, and rigorous behavioural measures are difficult and costly to use. The aim of this review was to identify the current evidence relating to the relationships between proxy measures and direct measures of clinical behaviour. In particular, the accuracy of medical record review, clinician self-reported and patient-reported behaviour was assessed relative to directly observed behaviour. Methods: We searched: PsycINFO; MEDLINE; EMBASE; CINAHL; Cochrane Central Register of Controlled Trials; science/social science citation index; Current contents (social & behavioural med/clinical med); ISI conference proceedings; and Index to Theses. Inclusion criteria: empirical, quantitative studies; and examining clinical behaviours. An independent, direct measure of behaviour (by standardised patient, other trained observer or by video/audio recording) was considered the 'gold standard' for comparison. Proxy measures of behaviour included: retrospective self-report; patient-report; or chart-review. All titles, abstracts, and full text articles retrieved by electronic searching were screened for inclusion and abstracted independently by two reviewers. Disagreements were resolved by discussion with a third reviewer where necessary. Results: Fifteen reports originating from 11 studies met the inclusion criteria. The method of direct measurement was by standardised patient in six reports, trained observer in three reports, and audio/video recording in six reports. Multiple proxy measures of behaviour were compared in five of 15 reports. Only four of 15 reports used appropriate statistical methods to compare measures. Some direct measures failed to meet our validity criteria. The accuracy of patient report and chart review as proxy measures varied considerably across a wide range of clinical actions. The evidence for clinician self-report was inconclusive. Conclusion: Valid measures of clinical behaviour are of fundamental importance to accurately identify gaps in care delivery, improve quality of care, and ultimately to improve patient care. However, the evidence base for three commonly used proxy measures of clinicians' behaviour is very limited. Further research is needed to better establish the methods of development, application, and analysis for a range of both direct and proxy measures of behaviour. Page 1 of 20 (page number not for citation purposes)
  2. Implementation Science 2009, 4:37 http://www.implementationscience.com/content/4/1/37 iour and proxy measures of the same behaviour, and how Background The measurement, reporting and improvement of the this relationship can best be described both on average quality of health care provision are central to many cur- and for individual clinicians. rent health care initiatives that aim to increase the delivery of optimal, evidence-based care to patients (e.g., quality Methods and outcomes framework (QOF) [1], new GMS contract Inclusion and exclusion criteria [2]). In the UK, the new GMS contract [2] introduced in We included any study that examined clinical behaviour 2004 represents a growing trend towards pay-for-perform- (behaviour enacted by a clinician – doctor, nurses and ance incentives in primary care, delivered through the allied health professionals – with respect to a patient or QOF. Accurate measures of health professionals' clinical their care) within a clinical context. Studies were included practice are therefore critically important not only to pol- if they reported a quantitative evaluation of the relation- icy makers in guiding health policy decisions but also to ship between a direct measure representing actual behav- practitioners in the evaluation of their own practice and to iour and an indirect, proxy measure of the same researchers both in identifying deficits and evaluating behaviour. We excluded studies of undergraduate stu- changes in the process of care. dents. A direct measure of behaviour was defined as one based on direct observation of a clinician's actual behav- Clinical practice can be measured directly – by actual iour in a clinical context by either a trained observer or a observation of clinicians while practicing, or indirectly – simulated patient, or of a video- or audio-recording of it. by the use of a proxy measure, such as a review of medical A proxy measure of behaviour was defined as one based records or interviewing the clinician. Direct measures on clinician self-report of recent or usual behaviour in a include observation by a trained observer, video- or specified clinical situation, or patient-report of clinicians' audio-recording of consultations, and the use of 'stand- behaviour or medical record review. ardised' or 'simulated' patients. These are generally con- sidered to provide an accurate reflection of the behaviour Search strategy for identification of studies under observation, and as such represent a 'gold standard' The following databases were searched: PsycINFO (1840 measure of performance. However, direct measures are to Aug 2004), MEDLINE (1966 to Aug wk 3 2004), intrusive, can promote (unrepresentative) socially-desira- EMBASE (1980 to Aug wk 34), CINAHL (1982 to Aug wk ble behaviour in the individuals being observed, and are 3 2004), Cochrane central register of controlled trials time-consuming and costly to use, placing significant lim- (2004 issue 2), science/social science citation index (1970 itations on their use in any context other than small stud- to Aug 2004), current contents (social and behavioural ies. Thus, they are not always a feasible option. med/clinical med) (1998 to Aug 2004), ISI conference proceedings (1990 to Aug 2004), and Index to Theses Measurement of clinical behaviour has therefore com- (1716 to Aug 2004). The search terms for behaviour, monly relied on less costly and more readily available health professionals, and scenarios are shown in Table 1. indirect sources of performance data, including review of The search strategy was devised to also identify studies for medical records (chart review), clinician self-report, and a related review that examined the relationship between patient report. Having effective and less costly proxy intention and clinical behaviour, and hence contained the measures of behaviour could expand both the policy and additional search term 'intention' [3]. The search domains research agendas to include important clinical behaviours were combined as follows: (Intention) AND (Behaviour) that might otherwise go unexamined because of measure- AND (health professionals), (Intention-behaviour) AND ment difficulties. However, despite their widespread use, (health professionals), (behaviour) AND (outcomes) the extent to which these proxy measures of clinical AND (health professionals). The reference lists of all behaviour accurately reflect a clinician's actual behaviour included papers were checked manually. is unclear. Review methods The aim of this review was to identify the current evidence All titles and abstracts retrieved by electronic searching relating to the relationships between direct measures and were downloaded to a reference management database; proxy measures of clinical behaviour. In order to establish duplicates were removed, the remaining references were whether any indirect measures can be used as proxies for screened independently by two reviewers, and those stud- actual clinical behaviour, the accuracy of medical record ies which did not meet the inclusion criteria were review, clinician self-reported and patient-reported behav- excluded. Where it was not possible to exclude articles iour were assessed relative to a direct measure of behaviour. based on title and abstract, full text versions were obtained and their eligibility was assessed by two review- ers. Full text versions of all potentially relevant articles Objective The objective of the review was to assess whether there is identified from the reference lists of included articles were a relationship between measures of actual clinical behav- obtained. The eligibility of each full text article was Page 2 of 20 (page number not for citation purposes)
  3. Implementation Science 2009, 4:37 http://www.implementationscience.com/content/4/1/37 Table 1: Keyword combinations for three domains, combined for the database search Behaviour Health professionals Intention Thesaurus headings: (Intention or intend*) near behaviour?* Thesaurus heading: • BEHAVIOR Thesaurus headings: INTENTION • CHOICE BEHAVIOR • HEALTH PERSONNEL • Intend* or intention* • PLANNED BEHAVIOR • ATTITUDE OF HEALTH PERSONNEL • Inclin* or disinclin* • Behaviour?* • CLINICIANS • Clinician performance* Clinician* • (Actor or abstainer) near behaviur* Counsellor* Dentist* Doctor* Family practition* General practition* GP*/FP* Gynaecologist* Haematologist* Health professional* Internist* Neurologist* Nurse* Obstetrician* Occupational therapist* Optometrist* OT* Paediatrician* Paramedic* Pharmacist* Physician* Physiotherapist* Primary care Psychiatrist* Psychologist* Radiologist* Social worker* Surgeon*/surgery Therapist* Example thesaurus headings are given for the PsycINFO database and were adjusted and exploded as appropriate for other databases. assessed independently by two reviewers. Disagreements 4. the percentage of participants enrolled for whom the were resolved by discussion or were adjudicated by a third relationship between direct and proxy measures of behav- reviewer. iour was analysed (attrition bias). Quality assessment Internal validity Internal validity relates to the rigor with which a study was External validity External validity relates to the generalisability of study conducted, and how confident we can be about any infer- findings. We assessed this for included studies on the ences that are subsequently made [4]. Important aspects basis of: of internal validity that are particularly relevant to the included studies are the reliability and validity of the 1. whether the target population of clinicians was local, measurement methods used to assess the performance of regional, or national. clinical behaviours. We therefore assessed internal validity on the basis of the psychometric evaluations performed 2. whether the target population of clinicians was sam- by each study: pled or whether the entire population was approached – and if the population was sampled, whether it was a valid Reliability random (or systematic) sample – in order to assess the 1. Measurement of inter-rater and intra-rater reliability for potential for selection bias. checklist scoring by trained observers and simulated patients. 3. the number of clinicians recruited and the total number of consultations assessed. 2. Test re-test reliability of either direct or indirect meas- ures. Page 3 of 20 (page number not for citation purposes)
  4. Implementation Science 2009, 4:37 http://www.implementationscience.com/content/4/1/37 and specificity; total agreement; total disagreement; and Validity of the scoring checklist Content and face validity of the scoring checklist: e.g., the kappa coefficients. For these studies, we concluded that rationale and process for the choice of items included and sensitivity and specificity were generally the best statistics for any weights assigned to them; to assess the performance of a proxy measure, provided these statistics were not based on a combination of items describing different clinical actions. Validity of the direct measure method General: The ability of the direct measure to accurately detect the aspects of behaviour under scrutiny (e.g., the Statistical methods used by studies that compared sum- range of clinical actions on the scoring checklist). mary scores included: comparisons of means; analysis of variance (ANOVA); t-tests; and Pearson correlation. For these studies, we concluded that summary measures Simulated patients 1. Content validity of simulated cases: the level of corre- should capture a single underlying aspect of behaviour spondence between components of simulated cases and and measure that construct using a valid measurement actual clinical presentations of the condition in question. scale. The average relationship between the direct and proxy measures should be evaluated over the entire range 2. Face validity: judgments made by individuals other of the direct measure, and the variability about this aver- than the research team that the simulated case 'looks like' age relationship should also be reported. Hence, compar- a valid case representation of the clinical condition in isons of mean scores are inappropriate. ANOVA and t- question. tests are likewise inappropriate because they are essen- tially methods of testing whether the mean score is the 3. Training of simulated patients in the case protocol. same in both groups. Correlation is inappropriate because it cannot assess whether there is systematic bias in the 4. Assessment of cueing and reporting of detection of sim- proxy measure (i.e., whether the proxy measure consist- ulation. ently under- or overestimates performance by a certain amount). Furthermore, the strength of the estimated cor- relation depends on the range of scores of the proxy and Validity of the Proxy methods direct measures. Patient vignettes Content validity: Correspondence between the operation- alisation of the simulated case in the standardized patient Data extraction protocols and written vignettes. For each study, we extracted the: age and professional role of participants; behaviour assessed; quantitative data measuring the relationship between the direct and proxy Patient report and Clinician self-report Content validity: Correspondence between the content measures of behaviour; method of measuring behaviour and wording of items on the scoring checklist and the and psychometric properties of measure; and quality cri- items on the questionnaire or interview schedule. teria specified above. Appropriateness of the statistical methods used Evidence synthesis The studies included in the current review used a range of For studies that reported single binary (yes/no) items, we statistical methods to summarise and compare direct and extracted, if possible, the number of consultations for proxy measures of behaviour. To help us synthesise the which: both the direct and proxy measures recorded the data from included studies we conducted a companion item as performed (true positives); both the direct and the review to assess the appropriateness of the different statis- proxy measures recorded the item as not performed (true tical methods they used (Dickinson HO et al. Are there negatives); the direct measure recorded the item as per- valid proxy measures of clinical behaviour? Statistical con- formed but the proxy measure did not (false negatives); siderations, submitted). Our conclusions are summarized and the direct measure recorded the item as not per- below. formed but the proxy measure recorded it as performed (false positives). The included studies were based on recording whether a clinician performed one or more clinical actions that we We estimated the mean and 95% confidence intervals refer to as 'items'. Some studies compared direct and (CI) for the sensitivity, specificity, and positive predictive proxy measures 'item-by-item'; other studies combined value of the item and present these on forest plots. If stud- items into summary scores and then compared direct and ies did not report the above numbers but reported the sen- proxy summary scores. sitivity and/or specificity, these statistics were extracted. For all studies for which their mean values were available, Statistical methods used by studies that compared direct the sensitivity was plotted against the false positive rate and proxy measures item-by-item included: sensitivity (1-specificity) because studies which fall in the top left of Page 4 of 20 (page number not for citation purposes)
  5. Implementation Science 2009, 4:37 http://www.implementationscience.com/content/4/1/37 this plot are generally regarded as having better diagnostic ferent findings from the same study [5,6,10,12,14,18] accuracy (high sensitivity and high specificity); however, a present different data and, with the exception of two summary ROC curve was not fitted to plots due to the het- [10,18], used different methods of analysis, we have con- erogeneity between studies in behaviour measured and sidered them as 15 separate reports for the purpose of this methods of measurement. Where possible, we also calcu- review. lated the positive and negative predictive values for indi- vidual items. For the 15 reports, 771 clinicians were enrolled and proxy measures of the clinical behaviour of 717 (93%) clini- For studies that reported aggregated scores summarising cians were evaluated relative to a direct measure. A sum- several items, we extracted any statistics presented that mary of the characteristics of the 15 included reports is summarised the mean and variance of the direct measure presented in Table 2, with further detail presented in and/or proxy summary scores and the relationship Additional File 1. Ten reports originated in the United between the direct measure and proxy. States, two in the Netherlands and one each in the United Kingdom, Australia, and Canada. The aim of 12 of 15 reports was to validate or to assess the 'accuracy' of an Results indirect measure of clinician behaviour relative to a spe- Description of included studies The search strategy identified 5,260 references (Figure 1). cific direct measure. The aim of the remaining three The titles and abstracts of these references were screened reports was to assess the relative validity of different meas- independently by two reviewers. Ten papers were ures (both indirect and direct) to each other. retrieved for full text review and their reference lists screened for other potential papers. A further 102 papers Participants in 12 reports were primary care physicians [5- were identified from the reference lists of retrieved papers, 8,10,12-18]; in other reports participants were nurses their abstracts were again reviewed independently by two [19], community pharmacists [11], and paediatricians [9]. reviewers, and 41 of these were retrieved for full text review. Fifteen papers, based on comparisons from eleven Clinical behaviours separate source studies, fulfilled the inclusion criteria and Five reports considered a range of clinical behaviours (e.g., their data were abstracted [5-19]. As papers reporting dif- history taking, physical examination, ordering of labora- Potentially relevant references References excluded at identified by search and screened electronic screening stage n = 5,260 n = 5,250 References retrieved for more References excluded at abstract detailed evaluation screening stage n = 112 n = 32 (10 identified by original search, 102 identified from reference lists of retrieved papers) References retrieved for full paper References excluded following review full paper review n = 80 n = 65 Number of references identified by search meeting inclusion criteria n = 15 Figure 1 Identification of included references (QUORUM diagram) Identification of included references (QUORUM diagram). Page 5 of 20 (page number not for citation purposes)
  6. Implementation Science 2009, 4:37 http://www.implementationscience.com/content/4/1/37 tory tests, referral, diagnosis, treatment, patient education, [5,7,9,10,12,14,15,17]; patient report on self-completion and follow-up) in relation to the management of a variety questionnaire or by exit interview [5-8,12-14]; or eight of common out-patient conditions: urinary tract infection reports evaluated multiple proxy measures [5,7,9,12- (UTI) [16]; tension headache, acute diarrhoea, and pain 15,19]. in the shoulder [17]; coronary artery disease (CAD), low back pain, and chronic obstructive pulmonary disease Methodological quality of included studies (COPD) [10,14,18]; diabetes [10,17,18]. One report con- External validity sidered the behaviour of recommending non-prescription The target populations in nine reports were regional medication or physician visit for common cold and pain [5,6,8,11,12,14,16,17,19]; all other reports targeted local symptoms [11], and one report evaluated medication reg- populations, such as physicians in two general internal imens prescribed for patients with COPD [12]. Six reports primary care outpatients clinics [10,15,18], attending considered health promotion behaviours, e.g., giving physicians at a university medical centre [9,13], and gen- advice about: smoking cessation [5-8,13,15]; alcohol use, eral practitioners in ten general practices [7]. Six reports exercise, and diet [5-7]; preventive care in relation to CAD, approached all participants in their target population low back pain, and COPD [15]; and sun exposure, sub- [6,7,9,11,16,17], three randomly sampled a group of cli- stance use, seatbelt use, and sexual health [6]. One report nicians [10,15,18], and six used convenience sampling considered the provision of a wide range of outpatient [5,8,12-14,19]. The number of clinicians enrolled and services including counselling, screening, and physical analysed in each report ranged from three [9] to 138 [5,6] examination [5]; and one evaluated physician communi- (median 34). Ten reports retained and analysed 100% of cation in paediatric consultations [9]. One report consid- recruited clinicians [7-15,18]. The median number of con- ered hand hygiene [19]. sultations observed was 160, with a range from 27 [16] to 4,454 [5,6]. For further details see Additional File 2. With the exception of two studies [8,13], the clinical behaviours measured were 'necessary' or 'recommended' Internal validity clinical actions categorized as such according to either Validity of the checklists used national guidelines or expert consensus. Four studies also In six reports, the content of the checklist was based on included actions that were unnecessary or that should not national guidelines for the behaviour in question be performed (e.g., prescribing an antibiotic for a viral [5,6,10,15,18,19], and for a further six reports content infection) [10,11,16,18]. was derived by expert consensus [11-14,16,17]. Two reports asked simply whether or not a physician asked about a particular lifestyle behaviour (e.g., smoking), and Methods used for measuring clinical behaviour In all studies a checklist was used to record the perform- whether or not they offered counselling [7,8]. One report ance of clinical actions relevant to the clinical area stud- did not report the rationale for their choice of clinical ied. All clinical actions were discrete activities, that is, actions [9]. Inter-rater reliability for assignment of weights could be coded as 'yes' or 'no' (e.g., the recording of blood to individual checklist items was presented in one report pressure, asking about smoking habits). The number of [11] and was 0.73. possible clinical actions observed in each study ranged from one [19] to 168 [18]. An important criterion for validity is that a measure should be reliable. Inter-rater reliability of scores gener- A summary of the proxy and direct measures used by the ated from checklists using direct measures were reported 15 included reports is presented in Table 3, with further for eight of the 15 included reports detail presented in Additional File 2. The direct measure [5,7,8,11,14,16,17,19], and ranged from 0.39 [5] to 1.00 of clinical behaviour was based on either: post-encounter [5,16] (Table 2). Five additional reports evaluated the reli- reports from simulated patients, [10,11,15-18]; prospec- ability of scoring between raters – stating these to be tive reports made by trained observers during direct obser- 'good' – but did not present inter-rater reliability statistics vation of actual consultations[5,6,19]; or post-encounter [6,10,13,15,18]. Two reports presented intra-rater reliabil- reports from trained observers rating audio- or video- ities which were 0.78 to 0.96 [16] and 0.74 to 1.0 [8]. Two recordings of consultations [7-9,12-14]. reports did not discuss the reliability of the scoring proce- dure [9,12]. One report evaluated the reliability of the The proxy measure of clinical behaviour was based on proxy measures used [16]. either: clinician self-report of recent behaviour on self- completion questionnaire or by exit interview [5,12- Validity of the direct methods used 14,19]; clinician self-report of simulated behaviour in a Only one report presented assessment of the ability of the specified clinical situation using clinical vignettes direct measure to detect the behaviours of interest [14]. [11,15,16,18]; medical record review They found that videorecording captured a median of Page 6 of 20 (page number not for citation purposes)
  7. http://www.implementationscience.com/content/4/1/37 Page 7 of 20 (page number not for citation purposes) Table 2: Summary of included study characteristics and clinical behaviours measured Study Characteristics Behaviour measured 1. Type of participants Participants approached & analysed Consultations/sessions/indications 1. Clinical area/s No. of Summarised 2. Target population observed/vignettes completed & analysed 2. Behaviour/s checklist 3. Sampling strategy observed items (weighted) (No. of clinical actions scored) N n % N n % Stange [5] 1. Family practice physicians 138 128 93 4454 4432 99 1. Delivery of a range of 79 1998 2. Members of the Ohio (MR) (MR) outpatient medical Academy of FPs, practice 3283 74 services within 50 miles radius of (PR) (PR) 2. Counselling (29), Cleveland & Youngstown physical examination (16), 3. Convenience sample screening (5), Lab tests (10), immunisation (7), Referral (4) Flocke [6] 1. Family physicians 138 128 93 4454 2,670 60 1. Health promotion 10 2004 2. Primary care physicians in 2. Smoking (2), alcohol, North West Ohio exercise, diet, substance 3. All physicians approached use, sun exposure, seatbelt use, HIV & STD prevention Wilson [7] 1. General practitioners 16 16 100 3324 516 (MR) 16 (MR) 1. Health promotion 4 1994 (GPs) 335 (PR) 10 (PR) 2. Asked patient about 4 2. 10 general practices in health behaviours: Nottinghamshire smoking (1), alcohol (1), 3. Selection of GPs not diet & exercise (1); reported. Minimum of two measurement of blood non-random consultations pressure (1) were recorded Ward [8] 1. Post-graduate trainees 34 34 100 1500 1075 72 1. Smoking cessation 2 1996 2. Training general practices 2. Establish smoking status Implementation Science 2009, 4:37 in New South Wales & provide smoking 3. Trainees who were cessation counselling (2) having their first experience in supervised general practice Zuckerman [9] 1. Paediatricians 3 3 100 51 51 100 1. Paediatric consultation 15 1975 2. Physicians working in a 2. Diagnosis and university medical centre management (8), historical serving an inner-city items (7) population 3. All 3 staff physicians
  8. http://www.implementationscience.com/content/4/1/37 Page 8 of 20 (page number not for citation purposes) Table 2: Summary of included study characteristics and clinical behaviours measured (Continued) Ö(w) Luck [10] 1. Primary care physicians 20 20 100 160 160 100 1. Management of LBP, NR 2000 2. 2 general internal DM, COPD, CAD. medicine primary care 2. History, Physical exam, outpatient clinics Tests ordered, Diagnosis 3. Random sample of 10 & Treatment/management physicians at each site (21 for LBP) Ö(w) Page [11] 1. Community pharmacists 30 30 100 58 58 100 1. Management of: Cold, 103 1980 2. Participants on a Pain continuing education 2. Recommend either: course in British Columbia, non-prescription Canada medication (cold = 17, 3. All participants pain = 15) or see physician (cold = 17, pain = 18) Gerbert [12] 1. Primary care physicians 63 63 100 197 197 100 1. Medication regimens in 4 1988 2. Primary care physicians the management of COPD serving 6 counties in 2. Prescription of California theophyllines (1), 3. Convenience sample sympathomimetics (2), oral corticosteroids (1) Ö Pbert [13] 1. Primary care physicians 2. 12 12 100 154 108 70 1. Smoking cessation 15 1999 Attending physicians & their 2. Cessation counselling patients at University (15) medical centre in Massachusetts. 3. Convenience sample Ö Gerbert [14] 1. Primary care physicians 63 63 100 214 192 90 1. Management of COPD 75 1986 2. NR 2. Symptoms (8), signs (2), 3. Convenience sample Tests (3), Treatments (3), Patient education (4) Ö Dresselhaus 1. Primary care physicians 20 20 100 160 160 100 1. Management of low 7 [15] 2. 2 general internal back pain, diabetes 2000 medicine primary care mellitus, COPD, CAD. outpatient clinics 2. Preventive care: 3. Random sample of 10 tobacco screening (1), physicians at each site smoking cessation advice Implementation Science 2009, 4:37 (1), prevention measures (1), alcohol screening (1), diet evaluation (1), exercise assessment (1) & exercise advice (1) Ö Rethans [16] 1. GPs 55 25 46 27 25 93 1. Management of Urinary 24 1987 2. GPs working in Tract Infection Maastricht 2. History taking (8); 3. All participants Physical Examination (3); Instructions to patients (7); Treatment (2); Follow-up (4)
  9. http://www.implementationscience.com/content/4/1/37 Page 9 of 20 (page number not for citation purposes) Table 2: Summary of included study characteristics and clinical behaviours measured (Continued) Ö Rethans [17] 1. GPs 39 35 90 140 101 72 1. Management of tension 25–36 1994 2. Sampling strategy headache; acute diarrhoea; reported elsewhere. pain in the shoulder; 3. Sampling strategy check-up for non-insulin reported elsewhere dependent diabetes. 2. History, Physical exam, Lab exam, Advice, Medication & follow-up (range over 4 conditions: 25–36) Ö(w) Peabody [18] 1. Primary care physicians 20 20 100 160 160 100 1. Management of low 168 2000 2. 2 general internal back pain (LBP), diabetes medicine primary care mellitus (DM), Chronic outpatient clinics obstructive pulmonary 3. Random sample of 10 disease (COPD) oronary physicians at each site artery disease (CAD). 2. History taking (7), Physical examination (3), lab tests (5), Diagnosis(2), Management (6) (Averaged 21 actions per case) Ö O'Boyle [19] 1. Nurses 124 120 97 120 120 100 1. Adherence to hand 1 2001 2. ICU staff in 4 hygiene recommendations metropolitan teaching 2. Hand washing (for a hospitals in "Mid-West" maximum of 10 USA indications) Implementation Science 2009, 4:37 3. ICUs with comparable patient populations
  10. Page 10 of 20 http://www.implementationscience.com/content/4/1/37 (page number not for citation purposes) Table 3: Summary of the measures used by included studies, methods of analysis and results of comparisons Study Proxy measure Direct Measure (DM) Analysis Description Clinician Medical Patient Description SP Psychome Compared Compared Agreement between P 1. Method self report Record report 1. Method Training trics (IRR) Item by Summary measures: V = Clinical vignette (SR) Review (PR) SP = Simulated reported Item Scores Co-efficient r; kappa (k); (No. of case (MR) Patients Structural equation simulations) DO = Direct modelling (SEM); Sensitivity CI/Q = Clinician Observation (Sens) & Specificity (Spec) interview/ VR = Video Difference between questionnaire recording mean scores: MR = Medical Record AR = Audio ANOVA; T-test review recording PI/Q = Patient 2. Timing interview/ questionnaire 2. Timing Ö Ö Ö Stange [5] 1. MR; PQ DO 0.39 to 1.00 MR NR 1998 2. At end of (kappa) Sens = 8% (diet advice) – consultation 92% (Lab tests) Spec = 83% (social history) – 100% (counselling services, physical exam, lab tests) k = 0.12 to 0.92 (79 comparisons) PR Sens = 17% (mammogram) – 89% (Pap test) Spec = 85% (in-office Implementation Science 2009, 4:37 referral) – 99% (immunisation, physical exam, lab tests) k = 0.03 to 0.86 (53 comparisons) Ö Ö Flocke [6] 1. PQ DO NR Sens* = 11% (substance NA 2004 2. At end of use) – 76% consultation (24%) or (smoking cessation) postal return (76%)
  11. Page 11 of 20 http://www.implementationscience.com/content/4/1/37 (page number not for citation purposes) Table 3: Summary of the measures used by included studies, methods of analysis and results of comparisons (Continued) Ö Ö Ö Wilson [7] 1. MR; PQ AR 0.79 to 1.00 MR NA 1994 2. At end of Sens = 31%, Spec* = 99% consultation 28.6 (Alcohol) Sens = 29%, Spec* = 100% 83.3 (BP) Sens = 83%, Spec* = 93% % agreement between DM & MR: 45.5 (Smoking) PR Sens = 74%, Spec* = 94% 75.0 (Alcohol) Sens = 75%, Spec* = 94% 100 (BP) Sens = 100%, Spec* = 90% % agreement between DM & PR: 81.8 (Smoking) Ö Ö Ward [8] 1. PQ AR 0.74 to 0.94 Sens = 93% NA 1996 2. Questionnaire (kappa) (smoking status) mailed to patient Spec = 79% within 2 days of Sens = 92% consultation (cessation advice) Spec = 82% Ö Ö Zuckerman 1. MR AR NR Sens* = 0% (side effects) – NA [9] 2. At end of 100% (Diagnosis) 1975 consultation Spec* = 9% (Diagnosis) – 100% (side effects) Ö Ö Ö Ö Luck [10] 1. MR SP (27) each role- NR ANOVA
  12. Page 12 of 20 http://www.implementationscience.com/content/4/1/37 (page number not for citation purposes) Table 3: Summary of the measures used by included studies, methods of analysis and results of comparisons (Continued) Ö Ö Ö ÖR Ö Gerbert 1. CI; MR; PI 0.52 to 0.93 Median % agreement (All NA [14] 2. At end of (kappa) categories): 1986 consultation 0.84 (SR) 0.88 (MR) 0.86 (PR) Ö Ö Ö Ö Dresselhaus 1.V (8); MR SP (4) each role- NA ANOVA
  13. Implementation Science 2009, 4:37 http://www.implementationscience.com/content/4/1/37 48% of the content of the overall consultation observed, into categories: 'necessary' and 'unnecessary' actions [10]; but that the level of capture varied from 10% to 100% 'must do', 'should do', 'must not do' and 'should not do' depending on the clinical action. actions [11]; and 'essential' and 'intermediate' actions [17]. Luck et al. [10] then estimated the sensitivity and Of the six reports that used standardised patients as the specificity within each category, and it was possible to esti- direct measure, four assessed the content and face validity mate the sensitivity and specificity for each category spec- of the patient scripts using expert review [10,15,18]. All ified by Page et al. [11] from the raw data presented. reported that training was provided to standardised Rethans et al. [17] also calculated the sensitivity of each patients, but two reports did not provide detail about the item (referred to by the authors as 'content scores') but duration or nature of the training [16,17]. In three studies, reported only the mean and inter-quartile range of sensi- standardised patients were experienced actors, who were tivities within each clinical area. Hence, sensitivities were trained according to a published protocol which was available for seven studies and specificities for six studies. delivered by experienced university-based educators [10,15,18]. One report used graduate students who were Six reports comparing item-by-item used other statistical trained for four hours as standardised patients [11]. The methods to compare their data [12-17]. These studies experience of the trainer was not reported, but standard- assessed 'agreement' and/or 'disagreement' between meas- ised patients pilot tested one of their simulated roles with ures; five reported agreement as the percentage of recom- a community pharmacist, and their checklist ratings were mended behaviours performed as recorded on the direct compared across four videotaped standardised patient and proxy measures [7,12,13,15,16], one also reported encounters with pharmacists. Three reports reported disagreement as the proportion of behaviours not detection rates of the standardised patient (i.e., the clini- recorded by the proxy measure that were detected by the cian realised that standardised patients were not genuine direct measure [12]; and one study estimated the 'total patients), and these were low (3%) [10,15,18]. agreement' and 'total disagreement' between measures, reporting median 'convergent validity' for 20 individual items and five clinical categories [14]. Validity of the proxy methods used With the exception of one report [19], the proxy method was directly related to the study visit; for example, reports Studies comparing summary scores using medical record review as the proxy method Seven reports aggregated items into summary scores of cli- abstracted medical records pertaining only to the study nicians' behaviour [10,11,13,16-19]. Three studies used visit, or patients were asked about a specific consultation. ANOVA to compare summary scores [10,13,18]; one The proxy measure used by O'Boyle et al. [19] was col- study used paired t-tests [16]; and four studies reported lected two weeks to four months before the direct meas- Pearson correlation coefficients [11,13,17,19]. urement. Relationship between direct and proxy measures In four reports that compared performance on the direct behaviour measure with a written vignette [11,15,16,18], all but one Studies comparing items [11] reported these to be identical case matches. In the lat- Patient report ter report, two standardised patient case protocols differed Three reports comparing item-by-item and reporting sen- from the corresponding written vignette in the nature of sitivity and specificity [5,7,8], and one reporting sensitiv- the clinical complication presented by the standardised ity only [6], examined patient report as a proxy measure patient [11]. The correspondence of standardised patient of clinician performance. Measurement techniques used and vignette case protocols for two reports was not were either patient questionnaire or patient interview, reported [10,17]. which were compared with direct observation [5,6] and audio-recording [7,8] (Table 2). Appropriateness of statistical methods used to summarise Median sensitivities for clinical actions relating to the pro- and report the relationship between direct and proxy vision of general outpatient services [5] and for health measures advice on a range of patient behaviours [6] were 53% Studies comparing items Thirteen reports compared measures of behaviour item- (range 25 to 89) and 43% (range 11 to 76), respectively. by-item [5-17]. Four of these studies estimated the sensi- Sensitivities for: the provision of smoking cessation advice tivity of the proxy measure for each clinical action meas- were 74% [7], 93% [8], and 76% [6]; for asking about ured [5-8], two the specificity [5,8] and one [7] the false alcohol use they were 75% [7] and 29% [6], and 100% for positive rate from which we calculated specificity. It was measuring blood pressure [7] (Figure 2). Median specifi- possible to calculate the sensitivity and specificity for indi- city for patient report was 98% (range 83% to 99%) vidual clinical actions from the raw data presented in a [5,7,8] across a number of services, 79% [8] and 94% [7] further report [9]. Three studies grouped clinical actions Page 13 of 20 (page number not for citation purposes)
  14. Implementation Science 2009, 4:37 http://www.implementationscience.com/content/4/1/37 for smoking cessation counselling, and 90% for the meas- between measures for the performance of clinical actions urement of blood pressure [7] (Figure 2). relating to the management of COPD. Gerbert et al. [12] present a kappa coefficient of 0.50 for the level of con- Positive and negative predictive values could be calculated cordance between patient report and their direct measure from the raw data of two reports evaluating the provision of video-recording and a 'disagreement' between the of smoking and alcohol advice and the measurement of measures of 24%. Pbert et al. [13] made comparisons blood pressure [7,8]. The positive predictive values for across measures for the detection of individual items patient-report were: 0.49 [7], 0.42, and 0.55 [8] for smok- using Cochrane's Q tests. These comparisons suggested ing advice; 0.40 for alcohol advice [8]; and 0.70 for the that patients tended to over-report their clinician's behav- measurement of blood pressure [7,8] (Figure 3). The neg- iour compared to the direct measure of audio-recording. ative predictive values for patient-report of the same behaviours were high for both studies (>0.90) [7,8]. This The accuracy of patient-report would suggest that patients accurately reported not receiv- ROC curves were plotted for the three studies where both ing advice and not having their blood pressure measured, sensitivity and specificity were available [5,7,8](Figure but they are less accurate in reporting that clinicians did Figure 4). The accuracy of patient report varied according perform these behaviours. to the clinical action of interest. Performance of the behaviours located in the top-left quadrant of this plot Three further reports compared item-by-item but did not were reported most accurately by patients. These included report sensitivity or specificity for their data [12-14]. Ger- the provision of counselling for health behaviours such as bert et al. [14] report a median 'total agreement' of 86% smoking, alcohol use, seat belt use, and breast self-exami- Specificity Sensitivity Page Cold relief: recommend medication - must do - should do - should not do - must not do Cold relief: physician required - must do - should do - should not do - must not do Pain relief: recommend medication - must do - should do - could do - should not do - must not do Pain relief: physician required - must do - should do - should not do - must not do Luck Necessary care Unnecessary care Zuckermann Diagnosis Drug name Drug dosage Drug action Side effects Other therapy Appointments Diagnostic studies Wilson Smoking (proxy=notes) Alcohol use (proxy=notes) Blood pressure (proxy=notes) Smoking (proxy=patient questionnaire) Alcohol use (proxy=patient questionnaire) Blood pressure (proxy=patient questionnaire) Ward Ask about smoking Advise to stop smoking Flocke Smoking cessation Exercise Diet Smoking Alcohol use Substance use Sun exposure Seatbelt use HIV prevention STD prevention 10 0.5 1 0 0.5 Figure 2 Sensitivities and specificities for six studies Sensitivities and specificities for six studies. Page 14 of 20 (page number not for citation purposes)
  15. Implementation Science 2009, 4:37 http://www.implementationscience.com/content/4/1/37 nation, which were more accurately reported by patients counselling and the measurement of blood pressure sen- than the provision of counselling for accident prevention, sitivities were 31%, 29%, and 83%, respectively [7], and dental health, contraception, and exercise (behaviours for 'necessary' and 'unnecessary' actions sensitivities were located in the bottom-left quadrant). The accuracy of 70% and 65%, respectively [10] (Figure 2). Median specif- patient report for clinical actions relating to physical icity for medical record review across a number of services examination, laboratory tests, and screening services also was 90% (range 81% to 100%) [5], and 97% (range 9% varied with the type of examination, test, or service under- to 100%) [9]. Specificities for smoking counselling, alco- taken [5]. hol counselling, and the measurement of blood pressure were 99%, 100%, and 93%, respectively [7], and 64% and 81% for 'necessary' and 'unnecessary' actions, respectively Medical record review Four reports comparing item-by-item and reporting sensi- [10] (Figure 2). tivity and specificity compared medical record review with direct observation in one report [5], with audio-recording As the raw data were available for three reports evaluating in two reports [7,9], and standardised patient accounts in medical record review [7,9,10], it was possible to calculate one report [10], (Table 2). a range of positive and negative predictive values for this proxy method (Figure 3). The positive predictive ability of Median sensitivity for a range of clinical actions relating to medical record review ranged from 0.30 to 0.92 (Median the provision of general outpatient services was 60% = 0.86) across different clinical actions, and was highest (range 8% to 92) [5] and 83% (range 0 to 100%) [9] for for 'necessary' care items (PPV = 0.85) [10], recording of clinical actions undertaken during routine patient consul- drug dosage (PPV = 0.88), diagnostic behaviours (PPV = tations (Figure 2). For smoking cessation advice, alcohol 0.91) [9], and the measurement of blood pressure (PPV = Positive Predictive Value Negative Predictive Value Page Page Cold relief: recommend medication - must do - should do - should not do - must not do Cold relief: physician required - must do - should do - should not do - must not do Pain relief: recommend medication - must do - should do - could do - should not do - must not do Pain relief: physician required - must do - should do - should not do - must not do Luck Necessary care Unnecessary care Zuckermann Diagnosis Drug name Drug dosage Drug action Side effects Other therapy Appointments Diagnostic studies Wilson Smoking (proxy=notes) Alcohol use (proxy=notes) Blood pressure (proxy=notes) Smoking (proxy=patient questionnaire) Alcohol use (proxy=patient questionnaire) Blood pressure (proxy=patient questionnaire) Ward Ask about smoking Advise to stop smoking Flocke Smoking cessation Exercise Diet Smoking Alcohol use Substance use Sun exposure Seatbelt use HIV prevention STD prevention 10 0.5 1 0 0.5 Figure and Positive 3 Negative Predictive Values for six studies Positive and Negative Predictive Values for six studies. Page 15 of 20 (page number not for citation purposes)
  16. Implementation Science 2009, 4:37 http://www.implementationscience.com/content/4/1/37 Proxy = Patient report Proxy = Medical record 1 1 .4 .6 .8 .8 Sensitivity Sensitivity .6 .4 .2 .2 .2 0 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 1 - specificity 1 - specificity Proxy = Self-report 1 Luck Luck 2000 Stange 1998 Wilson 1994 Zuckermann 1975 .4 .6 .8 Sensitivity Ward 1996 Page 1980: must do should do .2 should not do must not do 0 0 .2 .4 .6 .8 1 1 - specificity Figure 4 ROC plots of sensitivities and specificities for three proxy measures ROC plots of sensitivities and specificities for three proxy measures. Behaviours/actions in the top left-hand quadrant have both high sensitivity and specificity. See Stange 1998 [5] for additional sensitivities and specificities for 78 items. 0.84) [7] (Figure 3). The negative predictive ability of between summary scores relating to the management of medical record review ranged from 0.39 to 1.00 (Median commonly presenting outpatient conditions (Table 2). = 0.73) across different clinical actions, and was lowest (
  17. Implementation Science 2009, 4:37 http://www.implementationscience.com/content/4/1/37 Sensitivities and specificities ranged from 0.47 to 0.95 and and 61.7%, respectively). One report [17] reported an 0.40 to 0.80, respectively, for 'must do' and 'should do' overall correlation coefficient of r = 0.54 between sum- behaviours, and from 0.20 to 0.70 and 0.45 to 0.90, mary scores relating to the management of commonly respectively, for 'must not do' and 'should not do' behav- presenting outpatient conditions (Table 2). iours (Figure 2). Positive (PPV) and negative (NPV) pre- dictive values were also calculated for this study [11]. Clinician self-report PPVs ranged from 0.17 (cold relief: physician required/ Six reports evaluating clinician self-report calculated sum- should not do) to 0.89 (cold relief: recommend medica- mary scores [11,13,15,16,18,19]. Different reports com- tion/should not do (Median = 0.42) (Figure 3). NPVs pared these self-reports to different direct measures. ranged from 0.50 (cold relief: physician required/should do) to 1.00 (cold relief: recommend medication/must not One report [16] presented scores for the mean number of do), median = 0.80 (Figure 3). clinical actions performed by a group of clinicians as measured by each method in relation to the management Item-by-item comparisons evaluating clinician self-report of urinary tract infection (mean (SD) self-report = 9.88 were made by three further reports that used methods (3.44), standardised patient report = 10.04 (3.37)). other than sensitivities and specificities [12-14]. Gerbert et Rethans et al. [16] also presented subgroup means that al. (1986) [14] report 84% total agreement between clini- suggest clinicians under-report their performance for cian self-report and a video-recording of the consultation. 'obligatory' actions and over-report for less essential Gerbert et al. (1988) [12] presented a kappa coefficient of 'Intermediate' and 'superfluous' actions (Table 2). Two 0.67 for the level of concordance between clinician self- reports calculated the proportions for actions correctly report during interview and video-recording, and a total performed; one in relation to the management of com- disagreement between these measures of 13%. Pbert et al. mon outpatient conditions (% (SD) self-report = 71.0 [13] made comparisons across measures for the detection (5.4), standardised patient report = 76.2 (7.2)) [18], and of individual items using Cochrane's Q tests. These com- one in relation to the provision of preventive care advice parisons suggest that clinicians tended to over-report their (% (SD) self-report = 48.3 (14.4), standardised patient behaviour on some items compared to audio-recording. report = 61.7 (12.9)) [15]. Page et al. [11] present an over- all total agreement of 66% between self-report and stand- ardized patient report. The accuracy of clinician self-report A ROC curve was plotted for the one study where both sensitivity and specificity could be calculated for several, Three reports [11,13,19] present correlation coefficients 'must do/not do' and 'should do/not do' clinical actions of: 0.26 to 0.68 [11] for the relationship between perform- [11] (Figure 4). Behaviours categorized as 'should not do' ance on clinical vignettes and standardized patient tended to group in the top left quadrant of the plot, tenta- reports; 0.21 for a global self-estimate of performance of tively suggesting that clinician's accurately report for such hand hygiene actions with direct observation [19]; and behaviours (e.g., should not recommend medication for 0.54 for clinician self-reported provision of smoking ces- cold relief). Accuracy was poorer for behaviours catego- sation counselling compared with audio-taped accounts rized as 'must not do' and 'should do' (which tended to of the consultation [13]. group in the bottom left quadrant of the plot) and behav- iours categorized as 'must do' (which tended to fall into Discussion the top right quadrant of the plot). Validity of the direct measures used A problem in assessing any proxy measure of clinician performance is the validity of the direct measure itself as a Studies combining items into summary scores true reflection of actual behaviour. Simulated patients Patient report One report that evaluated patient report and made item- (standardised patients) have been widely used in medical by-item comparisons also combined items into summary education, and there is an extensive literature to support scores [13]. Pbert et al. [13] calculated scores that repre- their validity as a 'gold standard' method for measuring sented the number of smoking advice intervention steps clinical behaviour [12,14,18]. Standardised patients taken by a clinician during a patient consultation. The cor- require careful and detailed training in the clinical case relation of these scores between patient report and audio- they are to represent [20], and for those studies reviewed recording was r = 0.67. here that provide information about the training of stand- ardised patients, this appears to have been adequate [20]. Three included studies assessed detection rates by clini- Medical record review Three reports evaluating medical record review [15,17,18] cians, and reported these to be low. The six studies presented summary percentage scores (65.6%, 54.0%, [10,11,15-18] that used simulated patients specify very and 45.8%, respectively) that were consistently lower than precisely the characteristics of the cases presented to the scores reported by a standardised patient (76.2%, 68.0%, clinicians. The other studies observed the clinicians' Page 17 of 20 (page number not for citation purposes)
  18. Implementation Science 2009, 4:37 http://www.implementationscience.com/content/4/1/37 behaviour with actual patients and therefore had less con- uating clinician self-report and patient-report also used trol over the clinical situation in which behaviour was different techniques to capture the measure of behaviour assessed, but are likely to be more generalisable to real-life (e.g., interview, self-completion questionnaire, patient clinical situations. vignettes). Direct observation using trained observers, audio- or Patient report video-recording are also methods that are commonly used Patient-report measures demonstrated greater accuracy as direct measures of clinical behaviour. However, one than the other two proxy measures for reporting clinician study [14] using video-recording of consultations found performance, particularly with respect to counselling that relevant clinical detail – for example, assessment of behaviours and routine procedures. A cautionary adjunct symptoms and signs – was more frequently reported as to this, however, is the finding of one study that the pre- having been done when measured by clinician self-report. dictive validity of patient-reported information deterio- Taken at face value, this may suggest over-reporting on rates markedly as the time between patient exposure to behalf of clinicians. However, it is feasible that some clinician behaviour and the timing of their recall of events aspects of the clinical assessment of symptoms and signs increases [8]. Also, patient recall was found by another are performed non-verbally. In another study, the meas- study to be significantly influenced by the duration of the urement of blood pressure was accurately recorded in the advice and factors relating to relevancy, i.e., advice pro- patient medical record but was not detected by the direct vided during well-care consultations and the presence of a measure used (audio-recording) [7]. It is also plausible health behaviour-relevant diagnosis during an illness visit that, while we can expect that standardized patients may [6]. observe a clinician making an entry in a medical record, they could not accurately comment on the content of the Medical record review entry. A further example of the limits of capture for direct Medical record review appeared to underestimate many measures can be seen in one of four reports that compared aspects of clinician behaviour, particularly in the domain the direct measure of audio-recording with the proxy of of patient counselling. Thus, our findings suggest that medical record review [9]. This report found that while medical record review, in the outpatient setting, lacks some clinical actions investigated (for example, the dis- validity as a general measure of clinician behaviour. How- cussion of a diagnosis or drug name during a consultation ever, there was evidence to suggest that the predictive abil- with a patient) were not detected during evaluation of the ity of medical record review improves substantially for, audiotape session a diagnosis and the name and dosage of but is restricted to, specific types of clinical action, for drugs prescribed had been recorded in medical records by example, physical examination, the recording of drug dos- the physician. As an aim of this report was to evaluate cli- ages, and the ordering of laboratory tests. Medical records nician communication with patients, the direct measure may therefore be a relatively low-cost and accessible proxy was valid as it gave an accurate account of what the physi- measure for these clinical behaviours. Medical records cian did, and did not, communicate to the patient. How- may also be advantageous in that they can be good 'his- ever, audio-recording would lack validity as a direct tory keepers' because they can store information from sev- measure for the making or documenting of a diagnosis eral consultations and a variety of conditions. and some related management decisions. Clinician self-report This suggests that there are very few gold standard, direct The accuracy of clinician self-report as a measure of actual methods for assessing clinical performance – possibly behaviour is harder to establish because different studies only standardised patient methodology and participant using different methods produced different outcomes. observation – that can validly cover an extensive range of Also, none of the studies evaluating clinician report used clinical actions, and that none can truly capture all aspects appropriate statistical methods to summarise and/or of behaviour. A direct measure can only be a valid gold report the relationship between the measures used. standard for any given behaviour of interest, if it can reli- ably capture that behaviour. Four reports that calculated summary scores of perform- ance on vignettes appear to suggest that clinician's self- reported estimates of their behaviour were, overall, close Validity of the proxy measures used The accuracy of three proxy measures was reviewed: to those generated by the direct measure. However, closer patient report, medical record review, and clinician self- examination of the individual behaviours contributing to report. These indirect measures were used by the included the overall summary scores by one of these studies [16] reports to estimate the performance of a wide range of revealed that clinicians were overestimating their per- clinical actions. The accuracy of each proxy measure var- formance of some clinical actions and underestimating ied across the clinical behaviours measured. Reports eval- their performance of others, an observation lost in the Page 18 of 20 (page number not for citation purposes)
  19. Implementation Science 2009, 4:37 http://www.implementationscience.com/content/4/1/37 summary score due to counterbalancing. Over- and because of the heterogeneity in the designs, proxy meas- underestimation was also tentatively suggested on the ures, and summary statistics used in the included studies, ROC plot for an additional study [11], albeit in a contrast- we could not use conventional methods of assessing pub- ing direction. lication bias [26]. Nevertheless, the included studies pre- sented various results – seven studies [5-7,9,11,14,17] Of these two studies demonstrating over- and underesti- presented a range of both positive and negative findings, mation of self-reported behaviour, one provided clini- six studies [8,10,12,13,15,18] presented positive findings cians with a closed-ended checklist of possible behaviours only and one [16] presented only negative or inconclusive [11]. The second study used an open-ended response findings – suggesting that there is no apparent systematic mode with responses coded later by an independent tendency towards publication bias in the current review. observer [16]. This may explain the conflicting outcomes of these two studies; because closed-ended checklists pro- Conclusion vide clinicians with an extensive list of possible actions, In validating a proxy measure of clinical behaviour it is they may produce a cueing effect for them to select addi- imperative that the direct measure for comparison is itself tional actions or act as a prompt to elicit knowledge about both reliable and valid. In some of the included reports what they could, or should not do [21-23]. Such variation the direct measure lacked validity. Only four studies were in the ability of vignettes to predict the occurrence of found that used appropriate statistical methods to com- important behaviours that clinicians should or should not pare measures. The validity of patient report and medical do undermines their validity. However, this may be a record review varied widely across a number of clinical problem that can be overcome by careful and rigorous actions but was high for some specific clinical actions. The development of vignette cases and the method of their evidence for the validity of clinician self-report is incon- presentation [21]. clusive. Measures that use vignettes require clinicians to report Two recent systematic reviews evaluated the efficacy of their behaviour in the context of what they would do in a social cognitive models of behaviour in explaining clinical given clinical scenario. The remaining studies evaluating performance [3,27]. Both reviews found that the relation- clinician self-report collected retrospective accounts of ship between clinicians' self-reported intention and their behaviour is not perfect (maximum R2 reported was 0.44 actual behaviour using either interview or questionnaire methods and report correlation coefficients and measures [27]), and that the strength of the relationship often var- of 'total agreement' that suggest good agreement between ied depending on the method used to measure their measures. However, correlation is a measure of associa- behaviour. The current review supports the notion that at tion, and a high correlation can effectively disguise impor- least some of the discrepancy between intentions and tant disagreement if there is a consistent bias in one behaviour can be explained by error originating from measure [24]. A similar problem exists with the interpre- unreliable measures of behaviour. tation of 'total' or 'observed' agreement in that a large pro- portion of the agreement may be for behaviours that were Valid measures of clinical behaviour are of fundamental reported by both measures as not performed, again dis- importance to accurately identify gaps in care delivery, to guising important deficits in a proxy measure to accurately continuous improvement of quality of care, and ulti- detect actual performance [25]. mately to improved patient care. However, the evidence base for three commonly used proxy measures of clini- cians' behaviour is very limited. Further research needs to Review limitations Many references reviewed were sourced from the reference establish the scope of capture for a range of both direct lists of retrieved articles. We did not find a common ter- and indirect measures of clinical behaviour and the minology for describing written case simulations or proxy potential for using a combination of proxy measures to methods, and it is therefore possible that our database obtain an all round picture of clinical behaviour. search was subsequently limited by this. A common ter- minology for measures would greatly facilitate research in Competing interests this area. The literature search only covered up to August The authors declare that they have no competing interests. 2004; an update of this review could provide further use- ful information. A further limitation of this review is that Authors' contributions we were not able to combine data due to the heterogeneity All authors contributed to the conception and design and of the included reports. We tried to minimise publication analysis of the study and approved the submitted draft. bias by searching not only the peer-reviewed literature but MPE, JJF, EK SH and HD reviewed the articles and also abstracts of conferences and unpublished theses. As abstracted the data. we were unable to conduct a formal meta-analysis Page 19 of 20 (page number not for citation purposes)
  20. Implementation Science 2009, 4:37 http://www.implementationscience.com/content/4/1/37 Additional material 16. Rethans JJ, van Boven CPA: Simulated patients in general prac- tice: a different look at the consultation. British Medical Journal 1987, 294:809-812. 17. Rethans JJ, Martin E, Metsemakers J: To what extent do clinical Additional file 1 notes by general practitioners reflect actual medical per- Characteristics of included studies. Detailed description of the charac- formance? A study using simulated patients. British Journal of General Practice 1994, 44(381):153-156. teristics of all studies included in the review. 18. Peabody JW, Luck J, Glassman P, Dresselhaus TR, Lee M: Compari- Click here for file son of vignettes, standardized patients, and chart abstrac- [http://www.biomedcentral.com/content/supplementary/1748- tion: a prospective validation study of 3 methods for 5908-4-37-S1.doc] measuring quality. JAMA 2000, 283(13):1715-1722. 19. O'Boyle C, Henly S, Larson E: Understanding adherence to hand hygiene recommendations: the theory of planned behavior. Additional file 2 Am J Infect Control 2001, 29:352-360. Results presented by studies included in the review. Detail of the sam- 20. Buellens J, Rethans JJ, Goedhuys J, Buntinx F: The use of standard- ples, analyses and outcomes presented by studies included in the review. ised patients in research in general practice. Family Practice Click here for file 1997, 14:58-62. 21. Peabody JW, Luck J, Glassman P, Jain S, Hansen J, Spell M: Measuring [http://www.biomedcentral.com/content/supplementary/1748- the quality of physician practice by using clinical vignettes: a 5908-4-37-S2.doc] prospective validation study. Annals of Internal Medicine 2004, 141:771-780. 22. Spies T, Mokkink H, De Vries Robbe P, Grol R: Which data source in clinical performance assessment? A pilot study comparing self-recording with patient records and observation. Interna- References tional Journal for Quality in Health Care 2004, 16(1):65-72. 1. The Information Centre: Quality and Outcomes Framework for 23. Jones TV, Gerrity MS, Earp J: Written case simulations: do they GP practices. [http://www.ic.nhs.uk/]. [cited 28.08.2008] predict physicians' behaviour? Journal of Clinical Epidemiology 2. Department of Health: New GMS Contract 2003. Investing in 1990, 43(8):805-815. general practice. NHS Confederation and the British Medical 24. Chia KS: Association or Agreement? Annals Academy of Medicine Association. London; 2003. Singapore 2000, 29:263-264. 3. Eccles MP, Hrisos S, Francis J, Kaner EF, Dickinson HO, Beyer F, John- 25. Hripcsak G, Heitjan DF: Measuring agreement in medical infor- ston M: Do self-reported intentions predict clinicians' behav- matics reliability studies. Journal of Biomedical Informatics 2002, iour: a systematic review. Implement Sci. 2006, 1:28. 35(2):99-110. 4. Streiner DL, Norman GR: Health Measurement Scales: a practical guide 26. Egger M, Davey Smith G, Altman DG, (Eds): Investigating and dealing to their development and use 3rd edition. Oxford: Oxford University with publication and other biases. Chapter 11 in Systematic reviews in Press; 2003. health care: meta-analysis in context 2nd edition. London: BMJ books; 5. Stange KC, Zyzanski SJ, Smith TF, Kelly R, Langa DM, Flocke SA, Jaen 2001. CR: How valid are medical records and patient question- 27. Godin G, Belanger-Gravel A, Eccles MP, Grimshaw J: Healthcare naires for physician profiling and health services research? A professionals' intentions and behaviours: A systematic comparison with direct observation of patients visits. Medical review of studies based on social cognitive theories. Imple- Care 1998, 36:851-867. mentation Science 2008, 3(36):. 6. Flocke SA, Stange KC: Direct observation and patient recall of health behavior advice. Prev Med 2004, 38:343-349. 7. Wilson A: Comparison of patient questionnaire, medical record, and audio tape in assessment of health promotion in general practice consultations. Source. BMJ 1994, 309:1483-1485. 8. Ward J, Sanson-Fisher R: Accuracy of patient recall of oppor- tunistic smoking cessation advice in general practice. Tobacco Control 1996, 5(2):110-113. 9. Zuckerman ZE, Starfield B, Hochreiter C, Kovasznay B: Validating the content of pediatric outpatient medical records by means of tape-recording doctor-patient encounters. Pediat- rics 1975, 56(3):407-411. 10. Luck J, Peabody JW, Dresselhaus TR, Lee M, Glassman P: How well does chart abstraction measure quality? A prospective com- parison of standardized patients with the medical record. American Journal of Medicine 2000, 108(8):642-649. 11. Page GG, Fielding DW: Performance on PMPs and perform- ance in practice: are they related? 1980, 55:529-537. 12. Gerbert B, Stone G, Stulbarg M, Gullion DS, Greenfield S: Agree- Publish with Bio Med Central and every ment among physician assessment methods. Searching for scientist can read your work free of charge the truth among fallible methods. Medical Care 1988, 26:519-535. "BioMed Central will be the most significant development for 13. Pbert L, Adams A, Quirk M, Herbert JR, Ockene JK, Luippold RS: The disseminating the results of biomedical researc h in our lifetime." patient exit interview as an assessment of physician-deliv- Sir Paul Nurse, Cancer Research UK ered smoking intervention: a validation study. Health Psychol 1999, 18:183-188. Your research papers will be: 14. Gerbert B, Hargreaves WA: Measuring physician behavior. Med- available free of charge to the entire biomedical community ical Care 1986, 24:838-847. 15. Dresselhaus TR, Peabody JW, Lee M, Wang MM, Luck J: Measuring peer reviewed and published immediately upon acceptance compliance with preventive care guidelines: standardized cited in PubMed and archived on PubMed Central patients, clinical vignettes, and the medical record. Journal of General Internal Medicine 2000, 15(11):782-788. yours — you keep the copyright BioMedcentral Submit your manuscript here: http://www.biomedcentral.com/info/publishing_adv.asp Page 20 of 20 (page number not for citation purposes)
ADSENSE

CÓ THỂ BẠN MUỐN DOWNLOAD

LV.15: Bộ Đồ Án Tốt Nghiệp Chuyên Ngành Cơ Khí
65 tài liệu
2431 lượt tải
THÔNG TIN
TRỢ GIÚP
HỖ TRỢ KHÁCH HÀNG
Theo dõi chúng tôi

Chịu trách nhiệm nội dung:

Nguyễn Công Hà - Giám đốc Công ty TNHH TÀI LIỆU TRỰC TUYẾN VI NA

LIÊN HỆ

Địa chỉ: P402, 54A Nơ Trang Long, Phường 14, Q.Bình Thạnh, TP.HCM

Hotline: 093 303 0098

Email: support@tailieu.vn

Giấy phép Mạng Xã Hội số: 670/GP-BTTTT cấp ngày 30/11/2015 Copyright © 2022-2032 TaiLieu.VN. All rights reserved.

 

Đồng bộ tài khoản
2=>2