Design of an instrument to measure the structural quality of faceted thesauri

Chia sẻ: Lê Trung Hiếu | Ngày: | Loại File: PDF | Số trang:284

Thêm vào BST

Báo xấu

55
lượt xem 5
download

Download Vui lòng tải xuống để xem tài liệu đầy đủ

This study has two main goals: to construct a measuring instrument with which to evaluate the structural quality of faceted thesauri; and to determine the validity and reliability of this measuring instrument. The measuring instrument consists of gods, objectives, and criteria against which to measure the structural quality of faceted thesauri.

Chủ đề:

Bình luận(0) Đăng nhập để gửi bình luận!

Lưu

Nội dung Text: Design of an instrument to measure the structural quality of faceted thesauri

DESIGN OF AN INSTRUMENT TO MEASURE THE STRUCTURAL QUALITY OF FACETED THESAURI Louise FClicie Spiteri A thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy Graduate Department of Infoxmation Studies University of Toronto O Copyright by Louise FClicie Spiteri 1996
National Library BibliothQue nationale du Canada Acquisitions and Acquisitions et Bibliographic Services sewices bibliographiques 395 Wellington Street 395, rue Wellington OttawaON KtAON4 Ottawa ON K I A ON4 Canada Canada The author has granted a non- L'auteur a accorde une licence non exclusive licence allowing the exclusive permettant a la National Library of Canada to Bibliotheque nationale du Canada de reproduce, loan, distribute or sell reproduire, preter, distribuer ou copies of this thesis in microform, vendre des copies de cette these sous paper or electronic formats. la fome de microfiche/film, de reproduction sur papier ou sur format electronique. The author retains ownersbp of the L'auteur conserve la propriete du copyright in this thesis. Neither the droit d'auteur qui protege cette these. thesis nor substantial extracts &om it Ni la these ni des extraits substantiels may be p ~ t e or otherwise d de celle-ci ne doivent Stre imprimes reproduced without the author's ou autrement reproduits sans son permission. autorisation.
Louise FOlicie Spiteri Doctor of Philosophy 1996 Graduate Department of Information Studies University of Toronto ABSTRACT DESIGN OF AN NSTR-T TO MEASURE THE STRUCWR4L QUALITY OF FACETED THESAURI This study has two main goals: (a) to construct a measuring instrument with which to evaluate the structural quality of faceted thesauri; and @) to determine the validity and reliability of this measuring instrument. The measuring instrument consists of gods, objectives, and criteria against which to measure the structural quality of faceted thesauri. These goals, objectives, and criteria were derived from the theories of facet analysis, as well as fiom manuals and guidelines for thesaurus construction. The PDCA (Plan, Do, Check, Act) model used to design the measuring instrument was derived f o evaluation theory and sohvare evaluation im theory. An Item-Objective Congruence Form was used to measure the content validity of the instrument. Three judges were asked to rank the degree to which the criteria in the instrument measure the objectives they were designed to measure. The results obtained f o this test im confirmed the content validity of the instrument. Fifteen students from the Faculty of Information Studies, University of Toronto, were asked to apply the instrument to the Building Services Thesaurus. The results of these fifteen applications were used to test the construct validity and reliability of the measuring instrument. Principal components factor analysis was used to test the instrument's construct validity, and Cronbach Alpha to test the instrument's reliability. The results obtained from these two tests confirmed that, for the most part, the measuring instnunent has both construct validity and reliability .
ACKNOWLEDGMENTS I am deeply indebted to my thesis committee members Professors Ethel Auster, Clare Beghtol and Nancy Williamson of the Faculty of Information Studies, University of Toronto, for their guidance, advice and patience. I am also gratehl to Professor Lynne Howarth, Dr. Mavis Cariou and Michele Hudon for helping to assess the content validity of my measuring instrument. Ruth Croxford and Jeny Brunner From the Statistical Consulting Service, University of Toronto, provided invaluable advice and words of encouragement. Finally, special thanks are extended to my parents, Doris and Lino Spiteri, for their constant support and belief in me.
TABLE OF CONTENTS ABSTRACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11. . ACKNOWLEDGMENTS ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . III CHAPTER 1: INTRODUCTION 1.1 STATEMENT OF THE RESEARCH PROBLEM ................................ 1 1.2 BACKGROUND OF THE PROBLEM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.3 RESEARCH QUESTIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.4 GOALS AND OBJECTIVES OF THE STUDY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.5 ASSUMPTIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.6 RATIONALE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7 1.7 IMPORTANCE OF THE STUDY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9 1.8 DEFINITIONS AND EXPLANATIONS OF TERMS USED IN THIS STUDY . . . . . . . .- 9 1.9 LIMITATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13 1.9.1 Value judgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.9.2 Variability in measuring the validity and reliability of the measuring instrument . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.9.3 Indexing vocabulary of faceted thesauri ................................ 15 1.9.4 Self-administration o f construct validity and reliability tests . . . . . . . . . . . . . . . . 15 1.9.5 Use of the Building Services Thesaurus to measure instrument validity and reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.11 OUTLINE FORTHE REMAINDEROF THE STUDY .......................... 16
v CHAPTER 2: REVIEW OF THE LITERATURE AND THEORETICAL FRAMEWORK 2.1 CLASSIFICATION AND CATEGORIZATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.1.1 Introduction ...................................................... 18 2.1.2 The classification of knowledge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.1.3 Aristotle's theories of classification and categorization ..................... 20 2.1.4 Problems with the Aristotelian model of classification ..................... 21 2.1.5 Summary ........................................................ 22 2.2 FACET ANALYSIS ....................................................... 22 2.2.1 Introduction ..................................................... - 2 2 2.2.2 Ranganathan and the Classification Research Group . . . . . . . . . . . . . . . . . . . . . - 2 3 2.2.3 Idea Plane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .24 2.2.4 Verbal Plane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 2.2.5 Notational Plane .................................................. - 2 9 2.2.6 Summary ........................................................ 30 2.3 INTERNATIONAL AND NATIONAL GUIDELINES FOR THESAURUS CONSTRUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - 3 1 2.3.2 The development of guidelines for thesaurus construction . . . . . . . . . . . . . . . . . - 3 1 2.3.3 Content analysis of current international and national guidelines for thesaurus construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 2.3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 2.4 MANUALS FOR THESAURUS CONSTRUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . .36 2.4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .36 2.4.2 Content analysis of manuals for thesaurus construction . . . . . . . . . . . . . . . . . . . . 36 2.4.3 Summary ........................................................ 38 2.5 EVALUATION STUDIES OF INFORMATION STORAGE AND RETFUEVAL LANGUAGES ........................................................... 39 2.5.1 Introduction ...................................................... 39 2.5.2 Types of evaluation of ISAR languages ................................ 40 2.5.3 Summary ........................................................ 46
2.6 EVALUATION THEORY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 2.6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 2.6.2 bowledge component . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - 4 7 2.6.3 Value component . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 2.6.4 Practice component . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .- 4 9 2.6.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 2.7 SOFTWARE EVALUATION THEORY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 2.7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 2.7.2 Knowledge component . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - 5 2 2.7.3 Value component . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 2.7.4 Practice component . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .- 5 3 2.7.5 Comparison between evaluation theory and software evaluation theory . . . . . . . . 54 2.7.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .54 2.8 EVALUATION MODELS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 2.8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - 5 5 2.8.2 Evaluation theory models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - 5 5 2.8.3 Software evaluation theory models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 2.8.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .60 2.9 VALIDITY AND RELIABILITY MEASUREMENT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 2.9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61 2-92 Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 2.9.3 Reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .67 2.9.4 Impartiality of validity and reliability testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 2.9.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 . 2.10 SUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .73 CHAPTER 3: METHODOLOGY 3.1 INTRODUCTION ................................ 3.2 MODIFIED PDCA MODEL ................................................ 76 3.3 PLAN COMPONENT .IMPLEMENTATION .................................. 79 3.4 DO COMPONENT .MPLEMENTATION .................................... 92 3.5 CHECK COMPONENT .IMPLEMENTATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .97
vii 3.6 ACT COMPONENT .IMPLEMENTATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 3.7 SUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . -104 CHAPTER 4: ANALYSIS AND EVALUATION 3.1 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 4.2 CONTENT VALIDITY OF THE MEASURING INSTRUMENT: FINDINGS ....... 105 4.2.1 Goal 1 Content Validity ............................................ 106 4.2.2 Goal 2 Content Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 4.2.3 Goal 3 Content Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 4.2.4 Goat 4 Content Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 4.2.5 Goal 5 Content Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 4.2.6 Goals 6-8 Content Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 4.2.7 Content validity - summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 4.3 CONSTRUCT VALIDITY OF THE M E A S U m G INSTRUMENT: FINDINGS . . . . . 115 4.3.1 Goal 1 Construct Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 4.3.2 Goal 2 Construct Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 4.3.3 Goal 3 Construct Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .127 4.3.4 Goal 4 Construct Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 4.3.5 Goal 5 Construct Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 4.3.6 Goal 6 Construct Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 4.3.7 Goal 7 Construct Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 4.3.8 Goal 8 Construct Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 4.3.9 Construct Validity - Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 4.4 RELIABILITY OF THE MEASURING INSTRUMENT: FINDINGS . . . . . . . . . . . . . . . 147 4.4.1 Goal 1 Reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 4.4.2 Goal 2 Reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 4.4.3 Goal 3 Reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 4.4.4 Goal 4 Reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 4.4.5 Goal 5 Reliability ................................................. 155 4.4.6 Goal 6 Reliability ................................................. 156 4.4.7 Goal 7 Reliability ................................................. 158 4.4.8 Goal 8 Reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 4.4.9 Reliability - Summary .............................................. 160 4.5 SUMMARY ............................................................ 161
CHAPTER 5: DISCUSSION AND RECOMMENDATIONS 5.1 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 5.2 RESEARCH QUESTIONS: DISCUSSION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 5.2.1 Research Question #1 . Content Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 5.2.2 Research Question #2 . Construct Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 5.2.3 Research Question #3 - Reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . -172 5.3 METHODOLOGY: DISCUSSION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 5.3.1PDCAModel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 5.3.2 Instrument construction procedures ................................... 176 5.3.3 Measurement and scoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 5.3.4 Item-Objective Congruence Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .179 5.3.5 Application o f the instrument . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 5.3.6 Principal components factor analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182 5.3.7CronbachAlpha . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184 5.4 USE AND VALIDITY OF THE MEASURING INSTRUMENT . . . . . . . . . . . . . . . . . . . 186 5.5 RECOMMENDATIONS FOR FUTURE RESEARCH . . . . . . . . . . . . . . . . . . . . . . . . . . .188 5.5.1 Evaluation theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 5.5.2 Principles of facet analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .190 5.5.3 Guidelines and manuals for thesaurus construction . . . . . . . . . . . . . . . . . . . . . . . 192 5.5.4 Input evaluation of existing faceted thesauri . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 5.5.5 Output evaluation of faceted thesauri . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 5.6 GENERAL CONCLUSIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .195 APPENDIX 1: POPULATION OF FACETED THESAURI . . . . . . . . . . . . . . . . . . . . . . 198 APPENDIX 2: MEASURING INSTRUMENT BY WHICH TO EVALUATE THE STRUCTURAL QUALITY OF FACETED THESAURI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .200 APPENDIX 3: ITEM-OBJECTIVE CONGRUENCE RATING FORM ................ 227 APPENDIX 4: TECHNICAL R E m W FORM .................................231 APPENDIX 5: LETTER TO POTENTIAL EVALUATORS . . . . . . . . . . . . . . . . . . . . . . 233
APPENDIX 6: REVISED MEASURING INSTRUMENT BY WHICH TO EVALUATE THE STRUCTURAL QUALITY OF FACETED THESAURI ................ 234 APPENDIX 7: EXCERPTS FROM THE BUILDING SERVICES THESAURUS . . . . . . 260 REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
CHAPTER 1 INTRODUCTION 1.1 STATEMENT OF THE RESEARCH PROBLEM Much has been written in the literature of Library and Information Science (LIS) about the principles of facet analysis and their benefits to information retrieval (IR). Facet analysis has been applied to information storage and retrieval (ISAR) languages such as thesauri, but there has been little attempt to evaluate the quality of this application. Such an evaluation requires the design of an instnunent that contains a set of criteria with which to measure the quality of faceted thesauri. The purpose of this study is thus to design a measuring instnunent with which to evaluate the structure of faceted thesauri and hence their use of facet analysis. Evaluation studies of ISAR languages have focused traditionally upon the 'outputs' of the languages, i.e., their retrieval effectiveness. What has been largely absent from such studies is an assessment of the inputs, or structural quality, of the ISAR languages. Evaluation theorists suggest that the outputs of a system depend upon the quality of its inputs (Rossi & Freeman 1993; Scriven 1972; Suchrnan 1967; Tyler 1950). In other words, it may be difficult to assess properly the retrieval effectiveness of faceted thesauri without first understanding their structural quality. At this point an important distinction should be made between an evaluation study of the inputs of faceted thesauri and a measuring instrument for these inputs. An evaluation study entails designing the measuring instrument, applying it to a faceted thesaurus or to a sample of faceted thesauri, and drawing conclusions about the quality of the thesaurus/thesauri. The focus
2 of this study is upon designing the instrument, and not upon applying it to evaluate the quality of faceted thesauri. Since, however, the design of the instrument is an integral part of evaluation, it is necessary to base its design upon an understanding of evaluation theory. There are three main reasons for designing a measuring instrument for the inputs of faceted thesauri: (a) to provide a set of quality criteria with which to evaluate the quality of existing faceted thesauri and to act as a guideline for the construction of new faceted thesauri; (b) to facilitate future in-depth evaluation studies of how facet analysis is applied to IR thesauri by their designers; and ( c ) to contribute to future evaluation studies of the effectiveness of facet analysis as a retrieval device (the latter purpose may allow evaluators to determine the degree to which the application of facet analysis affects the retrieval effectiveness of faceted thesauri). 1.2 BACKGROUND OF THE PROBLEM Facet analysis is based upon two processes: (a) Analysis, whereby a subject field is divided into fundamental categories, each of which represents an essential characteristic of division of the subject field; and (b) Synthesis, whereby individual concepts from these categories may be combined to express compound subjects (Rangmathan 1962; 1967). ISAR languages such as thesauri may stand to benefit from the application of facet analysis. Facet analysis provides indexers and searchers with a conceptual map of a subject field, indicating precisely how indexing terms are related to one another and to the subject field. The conceptual structure of the subject field is demonstrated by sorting indexing terms into homogenous, mutually exclusive groups (or 'facets') that represent specific characteristics of division of the subject field. The facet structure also provides the context for the meanings of t
specific indexing terms. The term COMPUTER, for example, is an 'instrument' [instrument facet] for the 'processing' [processfacet] of 'data' [materialsfacet] (Biswas & Smith 1989; Fugmann 1990). The facet structure can also be used to distinguish among homographs and synonyms without the need of scope notes or, perhaps, parenthetical qualifiers as, for example: (Broadcastingfacet) (Transportationfacet) Station Station Facet analysis may also serve to reduce the size of the indexing vocabulary of a thesaurus. Faceted displays contain indexing terms that express only single concepts at a time. The principle of synthesis obviates the need to enumerate compound concepts. Instead of containing three separate terms such as ADULTS; EDUCATION; ADULT EDUCATION, a faceted thesaurus need contain only the first two terns, because the third concept may be combined at the search stage. A thesaurus that contains precoordinated compound concepts must be updated constantly if it is to keep up with the growth of compound subjects as reflected in the literature of its subject field. Since a faceted thesaurus allows for the free expression of virtually unlimited compound subjects within the subject field, it may need far less frequent revision (Foskett, D. J. 1980; Fugmann 1983). Over the past thirty years, a number of IR thesauri have been constructed based upon the principles of facet analysis (e.g., Thesaurofacet; DHSS-DATA Thesaurus; BSI Root Thesaurus). The growth in the number of faceted thesauri has not generally, however, been accompanied by any comprehensive attempts to evaluate their structure and use of facet analysis. Van Dijk and Sager have conducted the only two published evaluation studies of the structure of IR thesauri - that use a measuring instrument designed for this purpose (Sager et al. 1981; Van Djk 1976). i
4 Both skdies conclude that the faceted thesauri they examined did not use the principles of facet analysis in consistency with accepted practice. The validity of these conclusions may be questionable for two reasons: (a) the measuring instruments used in the studies contain criteria that are too general to reflect the specific structural quality of faceted thesauri; and (b) neither study defines clearly what is the accepted practice of facet analysis as it pertains to thesauri. In other words, neither study uses a measuring instrument that defines clearly the criteria against which the structural quality of faceted thesauri can be measured. Evaluation theorists generally agree that the evaluation of any system should consider both its structure (inputs) and its effects (outputs). Michael Scriven (1972) identifies three key activities in evaluation that may determine the structure of the measuring instruments used in evaluation studies: (a) criteria determination, to identify the dimensions on which the system must do well to be considered good; (b) setting standards, which tells how well the system must do on each dimension to be good; and (c) measuring performance, which requires measuring the system and comparing the results to the standards of merit. A measuring instrument for the inputs of faceted thesauri should therefore outline clearly the criteria against which structural quality is to be measured. The design of a measuring instrument for the inputs of faceted thesauri is impeded by the fact that there is no one source from which to compile a set of quality criteria for these thesauri. The literatures of facet analysis and categorization theory explain the principles of facet analysis, but emphasis is placed upon the application of these principles to bibliographic classification systems, and not to thesauri. International and nztional guidelines for thesaurus construction provide useful quality criteria for alphabetical displays, but few for faceted displays. The 1979 British Standards Institution (BSI) guidelines were the first to mention the use of facet analysis,
but provided no specific guidance pertaining to its actual application (BSI 1979). The 1981 Unesco guidelines were the first to provide a section devoted exclusively to the use of facet analysis in thesauri (Austin 1981). This section does not explain clearly, however, what it means by facet analysis and its use in thesauri. Despite its lack of clarity, the 1981 Unesco section on facet analysis has been adopted, unchanged, by the most recent BSI and International Organization for Standardization (ISO) guidelines (BSI 1987; IS0 1986). American guidelines for thesaurus construction make no provision for the use of facet analysis (NISO 1994). There is thus no consensus among the designers of IR thesauri regarding what constitute the essential characteristics of faceted thesauri and the standards of quality for these characteristics. 1.3 RESEARCH QUESTIONS This study seeks to answer the underlying question: to what extent is it possible to design a measuring instrument for the inputs of faceted thesauri that is both valid and reliable? This question may be subdivided as follows: (a) Are the contents of the instrument valid? In other words, to what extent do the contents of the measuring instrument actually measure the structural quality of faceted thesauri? (b) Are the constructs of the instrument valid? In other words, what are the factors that underlie the consistency or variance of the scores obtained from the measuring instrument? (c) is the instrument stable and accurate? In other words, will applications of the instrument by different people to the same faceted thesaurus produce similar scores?
The goals of this study are as follows: (a) to construct a measuring instrument with which to evaluate the structural quality of faceted thesauri; and @) to determine the validity and reliability of this measuring instrument. The objectives of this study are as follows: (a) to design an appropriate evaluation framework and model upon which to structure the measuring instrument; @) to determine the criteria against which to measure the stnlcturd quality of faceted thesauri; (c) to design the measuring instrument; (d) to determine the measurement processes and data collection and analysis methods to be used in the instrument; (e) to test the content validity of the measuring instrument; (f) to test the construct validity of the measuring instrument; (g) to test the reliability of the measuring instrument; (h) to analyse the strengths and weaknesses of the measuring instrument; and (i) to draw conclusions and make recommendations about the quality and applicability of the measuring instrument. 1.5 ASSUMPTIONS The assumptions of this study are as follows: (a) Evaluation studies entail making value judgments about the quality and worth of the item being evaluated (the "evaluand"). In this study, quality is meant to assume adherence to stated goals, objectives, and criteria for faceted thesauri.
(b) ~ i k e e n students from the Faculty of Information Studies (FIS), University of Toronto, were involved in testing the construct validity and reliability of the measuring instrument. These 15 students (referred to henceforth as "evaluators") all completed the course LIS 2171 Major Subject Heading and Classification Systems at FIS. Based upon the content of this course, it was assumed that the students are familiar with the following areas: principles of classification, subject analysis, indexing, and retrieval; the structure of ISAR languages, including thesauri; and major classification and subject heading systems. It was also assumed that these evaluators would be capable of evaluating the structural quality of faceted thesauri. The 15 evaluators were each asked to apply the measuring instrument to the Buildine Services Thesaurus (1993). The scores derived from these applications were used to test the instrument's construct validity and reliability, as is explained further in Chapter 3. (c) The measuring instrument does not distinguish between monolingual and multilingual faceted thesauri. These two types of thesauri should not differ in their underlying structures and use of facet analysis. IS0 and BSI guidelines for both types of thesauri contain identical sections pertaining to the use of facet analysis (BSI 1985, 1987;I S 0 1985, 1986). (d) The emphasis of the measuring instrument is upon the structure of the thesauri, and not upon the quality and currency of their indexing vocabularies. The indexing vocabulary of the Building Services Thesaurus was therefore assumed to be current at the time of its publication. 1.6 RATIONALE Since its introduction to classification theory through the works of Rangmathan and the Classification Research Group (CRG), facet analysis has grown in popularity over the years, manifesting itself in numerous bibliographic classification systems, thesauri, and indexing systems. More recently, facet analysis is being used in the design of knowledge-based ISAR languages (Burton 1986; Gibb & Fleming 1991; Revie & Smart 199 1; Stiles 1985; Travis 1989, 1990). Given the growth in its popularity and the extent of its use, it is perhaps time to re-examine our understanding of what constitutes the use of facet analysis in IR thesauri. As is demonstrated in Chapter 2, Van Dijk (1 976) and Sager (1981) are not alone in observing that
faceted thesauri do not always follow accepted norms of facet analysis. Herein lies the crux of the problem: are there clearly established norms of facet analysis that pertain to thesauri? Do thesaurus designers share a common understanding of what constitutes facet analysis and its use in thesauri? Classification theorists have traditionally focused their attention upon applying facet anatysis to bibliographic classification systems. These systems rely upon the use of notation and citation order. What is not clear, however, is whether the principles of facet analysis as formulated by Ranganathan and the CRG are appropriate for the structure and functions of information retrieval (IR) thesauri. The selection of the specific measures with which to evaluate the structural quality of faceted thesauri may serve to point out those facet principles that need to be modified to suit the needs of IR thesauri. Evaluation studies of ISAR languages indicate that emphasis has been placed mostly upon impact evaluation studies, with little attention given to input evaluation studies. Because of the scarcity of input evaluation studies of ISAR languages, it is necessary to formulate a new theoretical framework and evaluation model upon which to design a measuring instrument for the inputs of faceted thesauri. Evaluation theorists have proposed different frameworks that may be used for input evaluation studies and consequently, for the design of measuring instruments. Michael Quinn Patton and Peter Rossi suggest that, since no two evaluation studies are alike, it is best to adapt the elements from various frameworks to the special needs of the individual study, rather than try to fit the study into a particular framework (Patton 1982; Rossi & Freeman 1993). The theoretical framework used to design a measuring instrument for the inputs of faceted thesauri followed this eclectic approach. This framework consists of specific evaluation methods and
9 procedures, as well as the goals, objectives, and criteria with which to measure the thesauri. An evaluation model was used to design the measuring instrument for faceted thesauri. This model is a combination of different approaches used in evaluation theory and software evaluation theory and was adapted to accommodate the chosen theoretical framework. 1.7 IMPORTANCE OF T m STUDY It is hoped that this study will make contributions to the following areas: (a) the establishment of acceptable goals, objectives, and criteria for faceted thesauri that may serve as guidelines to future designers of such thesauri. An added benefit may be an increased consistency in the structure of faceted thesauri. This consistency may prove useful to both indexers and searchers as they move tiom one faceted thesaurus to another; (b) the design of a theoretical framework and evaluation model to serve as the basis for future input evaluation studies of IR thesauri; (c) the creation of a measuring instrument that may be used to determine the quality of the inputs of faceted thesauri; and (d) the establishment of a foundation upon which to conduct future impact evaluation studies of faceted thesauri as retrieval tools. 1.8 DEFINITIONS AND EXPLANATIONS OF TERMS USED I THIS STUDY N Enumerative Classification "An enumerative scheme assumes a universe of knowledge and divides it successively into narrower classes. It tries to enumerate specifically all topics of a scheme, whether expressed in simple terms or compound terms and phrases" (Aitchison 1982: 209). Examples of -
10 enumerative classification systems include the Dewey Decimal Classification and the Library of Cong-ress Classification. Facet A facet is a group of terms that represents one, and only one, characteristic of division of a subject field (e.g., STUDENTS, CURRICULUM, TEACHING METHODS are facets of the subject field "Education7'). Facets must be homogeneous and mutually exclusive (Ranganathan 1962, 1967). In this context, "mutually exclusive" means that each facet should represent a characteristic of division that is not found in any other facet. In other, words, no two facets may contain terms that could represent the same concepts. For example, all terms related exclusiveIy to the facet TEACHING METHODS should not appear in any other facet. Facet Analysis "The essence o i facet analysis is the sorting of terms in a given field of knowledge into homogenous, mutually exclusive facets, each derived from the parent universe by a single characteristic of division ... every distinctive logical category should be isolated, every new characteristic of division should be clearly formulated ... the facets so distinguished are not locked into rigid, enumerative schedules, but are left to combine with each other in the fbllest freedom, so that every type of relation between terms and between subjects may be expressed" (Vickery 1960: 12-13).