Campbell Collaboration Information Retrieval Methods Group
Conveners
Hannah R. Rothstein
Department of Management
Baruch College
1 Bernard Baruch Way
New York, NY 10010
Telephone (646) 312-3635
Fax (646) 312-3621
Email: hannah_rothstein@baruch.cuny.edu
Darcy L. Strouse
Health Programs Research
Aspen Systems Corporation
Mail Stop 9X
2277 Research Blvd.
Rockville, MD 20850
Telephone (301) 519-5284
Fax (301) 519-5585
Email: dstrouse@aspensys.com
Julia Lavenberg
Research Assistant
University of Pennsylvania, Graduate School of Education
3700 Walnut Street, Rm. 306
Philadelphia, PA 19104-6216
Telephone (215) 573-5688
Email: jlavenbe@dolphin.upenn.edu
Steering Committee Group Members
Alan Gomersall
Julia Lavenberg
Hannah R. Rothstein
Darcy L. Strouse
Herb Turner
This description of the Campbell Collaboration Information Retrieval Methods Group has four sections. In the first, the reasons for starting this group are introduced. This is followed by a description of the activities through which the Information Retrieval Methods Group supports the goals of the Campbell Collaboration. Then, we offer several short-term objectives for the group. Finally, biographical sketches of the conveners are provided.
Introduction
The Campbell Collaboration was established to facilitate the production of systematic reviews of the effectiveness of social programs and public policies. A comprehensive, unbiased compilation of potentially relevant studies is one of the key characteristics of a systematic review. In fact, if the literature located by the reviewer is unrepresentative of the population of studies that have been conducted, the rest of the review process will be compromised. Given the nature of the problems addressed by the Campbell Collaboration, potentially relevant studies are likely to be widely distributed and unreliably categorized. While retrieval of information from the literature is a critical concern for any systematic reviewer, retrieval of information about complex socio-behavioral interventions is likely to be particularly challenging.
The Information Retrieval Methods Group furthers the aims of the Campbell Collaboration by providing assistance to reviewers in the areas of information retrieval and assessment of publication bias.
Activities of the Information Retrieval Methods Group (IRMG)
1. Provide Advice on Information Retrieval and Assessment of Publication Bias
IRMG provides advice to review groups and individual researchers on retrieval of studies and assessment of publication bias. This includes advice on developing and documenting a systematic search strategy, keeping track of studies, and assessment of the likelihood of bias in the results of the review. It also includes assistance with the development of study registers, and the development of a comprehensive list of searchable databases and alternative sources that are likely to contain relevant studies.
Although certain aspects of searching are specific to specific review groups and/or research questions (such as generating a thesaurus of key words), we believe that it may be useful to develop a search strategy checklist that can be adopted Collaboration-wide. We also might work with the statistics methods group on methods for assessment of publication bias.
2. Provide Training and Support
IRMG provides training and support to review groups and individual researchers. This includes workshops associated with Campbell Collaboration meetings on such topics as hand searching, the use of electronic databases, reference-tracking strategies, and the assessment of publication and reporting biases. We also produce other written support materials in collaboration with other review groups.
In addition to formal mechanisms of training and support, IRMG is also available to provide assistance to review groups at the time they are planning reviews and/or developing protocols for reviews.
3. Conduct Research on Information Retrieval and Publication Bias
IRMG plans to develop a handsearch manual, and also intends to develop bibliographies on locating and selecting relevant studies, and on publication bias and other availability biases. The starting point will be bibliographic material produced by the Cochrane Collaboration, but will also include material from social and behavioral science sources not included in the Cochrane databases.
In addition, IRMG conducts research on information retrieval strategies in the social and behavioral sciences, such as studies on the sensitivity, specificity and reliability of various search strategies, and studies of the likely magnitude of biases produced by incomplete information retrieval.
4. Help with Monitoring the Quality of Campbell Reviews
IRMG helps monitor the quality of the information retrieval and study selection components of systematic reviews carried out under the auspices of the Campbell Collaboration.:
We encourage and are available to review groups to consult in the development of protocols. We believe that explicit consideration of information retrieval issues during the development of review group protocols and search strategies contributes substantially to the quality of the reviews produced.
5. A Forum for Discussion of Issues Related to Information Retrieval and Publication Bias
IRMG serves as a forum for discussion of issues related to the comprehensive and unbiased retrieval and inclusion of information in systematic reviews, particularly those produced by the Campbell Collaboration.
Short Term Objectives 2002-2003:
- Develop Campbell Collaboration Hand Search Manual (project coordinator Julie Lavenberg).
- Develop Bibliography of: Information Retrieval Resources (project coordinator Darcy Strouse)
- Develop annotated bibliography of Publication Bias Research (project coordinator Hannah Rothstein)
- Develop training materials and/or training workshop. (Co-coordinators Julie Lavenberg and Darcy Strouse, to be coordinated with C2 Training Group).
- Survey other Campbell entities about their information retrieval needs and concerns.
The IRMG welcomes new members. Please contact Hannah Rothstein (hrothstein@mindspring.com) if you are interested in becoming a member.
Biographies of Conveners
Hannah Rothstein is Professor of Management at Baruch College of the City University of New York, where she serves as coordinator of the doctoral specialization in Organizational Behavior and Human Resource Management. She received her Ph.D. in Industrial/Organizational Psychology from the University of Maryland in 1980.
Professor Rothstein has been an active researcher in the area of meta-analysis for 15 years. She has published numerous articles concerning methodological issues in meta-analysis, beginning with the 1986 publication of "Forty Questions about Validity Generalization and Meta-analysis," in Personnel Psychology. Rothstein has been first author of four meta-analyses of employment selection methods published in Journal of Applied Psychology and Personnel Psychology, and has co-authored several other published meta-analyses. Together with Michael Borenstein and others, she has developed computer software for meta-analysis (Comprehensive Meta-Analysis), and for power and confidence interval analysis (Power and Precision). Her most recent publications include “Meta-Analysis: A Review of Quantitative Cumulation Methods” in Measuring and Analyzing Data in Organizations (F. Drasgow and N. Schmitt, Eds. Jossey-Bass, 2002) and “Contributions of Validity Generalization and Meta-Analysis to the Development and Communication of Knowledge in I/O Psychology” in Validity Generalization: A Critical Review, (K. R. Murphy, Ed., Erlbaum, 2002). Rothstein is on the editorial board of the International Journal of Selection and Assessment, and is a reviewer for Psychological Methods, Journal of Applied Psychology, Psychological Bulletin, Personnel Psychology, the Academy of Management Journal, and Human Performance. She is a member of the Reporting Bias Methods Group of the Cochrane Collaboration.
Darcy Strouse is Director of Research for Aspen System’s Corporation Health Programs Division, where she leads a research support unit providing scientific expertise, including research synthesis support, to applied health and social science professionals. She received her Ph.D. in Developmental Psychology from the Catholic University of America in 1996. Strouse has been actively involved in large-scale research synthesis projects, serving as the Project Director of several federally funded research synthesis projects, including the Centers for Disease Control’s Prevention Research Synthesis (PRS) project. She is currently directing a research synthesis project for the World Health Organization (WHO). Strouse has been actively involved in the Cochrane Collaboration, serving as Trials Coordinator for the Cochrane HIV/AIDS Review Group. She has presented on information retrieval methods at the 6th and 7th International Cochrane Colloquia, the 12th and 13th World AIDS Conferences and the 8th International Congress on Medical Librarianship. Strouse co-authored a series of publications, including a meta-analysis of HIV behavioral and social prevention interventions for youth, appearing in a July 2002 special issue of JAIDS.
Julia Lavenberg, MSEd. RN, is currently a doctoral student entering her final year of coursework at the University of Pennsylvania Graduate School of Education. She has been active in writing hospital policies and procedures, nursing documentation protocols, and patient education materials in her role as a registered nurse in hospitals in Los Angeles, Boston, and Philadelphia. Expanding these interests into the social research setting, she is interested in exploring how researchers within social welfare, education, and criminology discuss and document their research methodology. In addition, she is interested in teaching future social science researchers and educating policymakers about health and social issues, using high quality systematic reviews.