Statistics Methods Group Application
Conveners
Larry V. Hedges (Primary Contact)
Department of Education
University of Chicago
5835 S. Kimbark Avenue
Chicago, IL 60637
Telephone: (773) 702-1589
FAX: (773) 702-0248
E-mail: hedge@cicero.src.uchicago.edu
William R. Shadish
Department of Psychology
202 Psychology Building
The University of Memphis
Memphis, TN 38152-3230
Telephone: (901) 678‑4687
FAX: (901) 678‑2579
E-mail SHADISH@MAIL.PSYC.MEMPHIS.EDU
Background
One essential aspect of systematic reviews is that they contain some representation of study results and the summary of those representations across studies. Statistical methods are most often used to develop summary indicators of study results (in the form of effect size statistics) and to combine them across studies. Therefore statistical issues are often of major concern in systematic reviews. The proposed statistics methods group will further the aims of the Campbell Collaboration in five ways.
Advice on Statistical Methods and Statistical Policy
The statistics methods group will provide advice to review groups and to individual researchers on statistical aspects of methodology. This will include advice on methods for combining effect size measures across studies and characterizing variation of effects across studies. However it will also include advice on the choice of metric for representing study results (choice of effect size measure), procedures for calculating effect size estimates in the metric chosen, and interpretation of effects. The statistics group can also provide advice to other Campbell Collaboration groups (such as the empirical studies group) conducting empirical research involving statistical methods. The statistics group will also provide advice to the Campbell Collaboration on methods policy. For example, the statistics group can provide advice on statistical software for systematic reviews. We believe that it would be desirable to develop a set of statistical requirements for any software that Campbell Collaboration might develop to ensure that it will include the analyses that are likely to be used.
We anticipate that some of the advice provided may be reactive, in the form of responses to queries from the steering committee, methods coordinating group, review groups, or individual researchers. However we also anticipate that it will be desirable to develop a set of statistical standards for Campbell Collaboration reviews for incorporation into the reviewers handbook or other Campbell Collaboration documentation. Specifically, the steering committee or its designee may wish to ask us to develop and/or revise a set of statistical standards for Campbell Collaboration reviews.
Training and Support
The statistics methods group will also provide training and support to review groups and individual researchers as part of a program of methodological training provided by the methods groups. It is anticipated that this training will include formal training courses (short courses) associated with Campbell Collaboration meetings. If there is sufficient demand, such courses might be organized in other venues such as in conjunction with other professional meetings (e.g., the American Educational Research Association, the American Evaluation Association, or the American Psychological Association), as training sessions associated with other series (such as those sponsored by the Inter-University Consortium for Political Science Research [ICPSR] at the University of Michigan), or even as free standing training sessions.
We note that several of those invited for membership in this group (e.g., Becker, Hedges, Olkin, Shadish, and Vevea) have directed such short courses on statistical methods in meta-analysis for various professional societies, so there is considerable experience in the training function. However we see the training function as having an element of evolutionary operation. That is, we anticipate that the issues that arise in actual Campbell Collaboration reviews will be incorporated into the training and support materials to make them more comprehensive and relevant to the Campbell Collaboration. Examples of such support materials would be a compendium of effect size calculations or computer software to carry out computation of effect sizes and their sampling variances in various situations that arise frequently.
In addition to formal training, the statistics group will provide support to review groups involved in systematic reviews. We anticipate that this support will usually be provided by a member of the statistics group who is also a member of the review group. Conversely, membership in the review groups will help inform the statistics group about the problems faced in Campbell Collaboration reviews, leading to improvements in training and support materials.
Research on Statistical Methods
The statistics group will develop a bibliography on statistical aspects of systematic reviewing that are particularly pertinent to reviews in the social sciences. The starting point will be the Cochrane Collaboration bibliography on statistics. However the bibliography will also contain references from social and behavioral science sources not included in the Cochrane database. These reference searches will be coordinated with and contribute to the Campbell register of methodological studies.
The statistics group will also conduct research on statistical aspects of systematic reviewing. This will include individual and collaborative efforts in analytic research (e.g., deducing sampling theory for new effect size measures), simulation studies to address problems that are intractable analytically (e.g., studies of robustness to violations of assumptions), and empirical comparative research on reviews (e.g., evaluating the consequences of statistical decisions by comparing the mean effect sizes generated from studies using different analytic designs).
Quality Monitoring
The statistics group will help monitor the quality of statistical aspects of systematic reviews being carried out by the Campbell Collaboration. This monitoring will be carried out in two ways.
First, members of the statistics group will serve as prepublication reviewers for systematic reviews conducted by the review groups in Campbell Collaboration. We believe that it would be a wise policy to require that at least one member of the statistics group serve as a reviewer for every systematic review conducted by the Campbell Collaboration.
Second, members of the statistics group will also serve as members of Campbell Collaboration review groups. We believe that it would be a wise policy to encourage that at least one member of the statistics group to serve as a member of every systematic review group in the Campbell Collaboration, including those that chose not to use statistical methods in their reviews.
A Forum For Discussion of Statistical Issues
The statistics group will serve as a forum for discussion of statistical aspects of representing and combining evidence in general and in systematic reviews being carried out by the Campbell Collaboration, in particular.
Scope
The goals of the statistics group are to contribute to the Campbell Collaboration in the five ways elaborated in the previous section:
- providing advice on methodology and methods policy
- providing training and support
- conducting empirical research
- helping to monitor the quality of systematic reviews prepared within the Collaboration
- serving as a forum for discussion.
To achieve these goals, we have developed five short term objectives for the group. These objectives are detailed below.
Objective 1: Organizational Meeting in Philadelphia
We will organize a first meeting among the initial members who have expressed interest. Of the 47 individuals asked to participate, 36 have agreed to participate in the statistics methods group, including 8 from outside the United States. Twenty-four of these 36 have indicated that they intend to attend the Philadelphia meeting of the Campbell Collaboration, 3 have indicated that they do not intend to attend and the rest have not indicated whether or not they will attend. At this meeting we will approve a set of rules for operation of the statistics group, agree on short term objectives, and obtain commitments from individuals to work on specific projects that will help us reach our objectives.
Because the other objectives specified here have not been formally approved by other members of the statistics group, they should be regarded as provisional.
Objective 2: Statistical Requirements for Review Software
Should we be asked to do so, the statistics group will prepare a set of draft statistical requirements for statistical software that might be developed for Campbell Collaboration reviews. The purpose is to ensure that review software that might be developed for Campbell Collaboration will have the capacity for statistical analyses that are likely to be needed in Campbell Collaboration reviews. A draft will be developed within three months to be circulated within the methods groups and review groups.
Objective 3: Statistics Bibliography
The first objective should be to develop a bibliography of statistical methods for systematic reviews. The starting point will be the Cochrane Collaboration bibliography on statistics. Because the Cochrane Collaboration bibliographies do not include all social and behavioral science sources, a separate (but overlapping) bibliography will be developed. The starting point for augmenting the bibliography might be the bibliographies of the books on statistical methods for meta-analysis by Hedges and Olkin (2001), Lipsey (2000).
The goal would be to have a working bibliography by the end of September 2001, and this will be shared with the Campbell Center for Systematic Review Methods for inclusion in the Campbell register of methodological studies.
Objective 4: Training Sessions
Two short courses on statistical methods for systematic reviews will be developed. These short courses will be planned in conjunction with training in other aspects of systematic reviews that the Campbell Collaboration develops and coordinated via the Campbell Center for Systematic Review Methods. Therefore the proposed courses are subject to change in order to better fit the overall training agenda of the methods groups.
Half Day Short Course
One short course will be a half day introduction to the statistical methods for systematic reviews. The emphasis of this course is conceptual, answering the question “What are statistical methods for meta-analysis and why do we need them?” The course will begin with an explanation of the vote-counting fallacy and why special statistical methods are needed for meta-analysis. It will provide an introduction to the effect size idea and several common effect size measures, the distinction between effect size parameters and their estimates, the notion of sampling error variance, homogeneity and heterogeneity of effect sizes. The distinction between conditional and unconditional inferences and between fixed and random effects statistical combination procedures will also be addressed. A brief introduction to the nature and interpretation of meta-analytic summaries will be given, with examples. Finally there will be an introduction to the problem of publication and reporting bias and methods of detecting such biases.
One-Day Short Course
The one day course will provide a more comprehensive introduction to statistical methods for meta-analysis. The emphasis is on providing concrete tools for use in systematic reviews. Material will include an introduction to computing effect size estimates from real studies. Material on combination will include fixed- and random-effects statistical procedures for estimating a common effect size, methods for assessing heterogeneity, fixed- and mixed-effects statistical procedures for modeling between-study variation in effect sizes, and special methods for combining small studies. Methods for detecting and estimating the impact of publication bias will also be discussed.
The goal would be to have a half-day short course available as soon as the next meeting of the Campbell Collaboration in Philadelphia in February 2001, and the one-day short course available by the second meeting of the Campbell Collaboration in 2002.
Objective 5: Statistical Standards for Campbell Collaboration Reviews
Should we be asked to do so, the statistics group will prepare a set of draft standards for statistical aspects of Campbell Collaboration reviews, with a draft to be developed within six months.
Coordination with the Cochrane Collaboration
At the statistics methods group meeting of the Cochrane Collaboration in Capetown, Fred Wolf was formally asked to be the liaison between the Cochrane Collaboration statistics group and the Campbell Collaboration statistics group. Michael Borenstein also volunteered to help as a liaison between the two groups.
Biographical Sketches of Convenors
Larry V. Hedges
Larry V. Hedges is the Stella M. Rowley Professor of Education, Psychology, Public Policy Studies, and Sociology at the University of Chicago. He earned his BA form the University of California, San Diego and his MA and PhD from Stanford University. His primary research interests are the application of statistical methods to problems in education, the social sciences, and policy studies, particularly the combination of results of replicated empirical research studies (meta-analysis). He has served as chairman of the Department of Education at the University of Chicago. He is Editor of the Journal of Educational and Behavioral Statistics and was Quantitative Methods Editor of Psychological Bulletin, and currently serves on the editorial board of Psychological Methods, the American Journal of Sociology, and the Review of Educational Research. He is a member of the National Academy of Education, a Fellow of the American Statistical Association and the American Psychological Association, an elected member of the Society for Multivariate Experimental Psychology, and was a visiting fellow of the Russell Sage Foundation. He has served on numerous professional boards and panels including several National Research Council committees, and is currently a member of the technical advisory committees of both the International Education Association and the OECD’s Programme of Indicators of Student Achievement. His recent books include Statistical Methods for Meta-analysis (with Ingram Olkin) and the Handbook of Research Synthesis (with Harris Cooper).
William Shadish
William R. Shadish is Dunavant University Professor of Psychology at the University of Memphis, where he is Director of the Research Design and Statistics doctoral training program. He received his bachelor’s degree in sociology from Santa Clara University in 1972, and his M.S. (1975) and Ph.D. (1978) degrees from Purdue University in clinical psychology. He completed a postdoctoral fellowship in methodology and program evaluation at Northwestern University from 1978-1981. His current research interests include experimental and quasi-experimental design, the empirical study of methodological issues, the methodology and practice of meta-analysis, and evaluation theory. He is author (with T.D. Cook & L.C. Leviton) of Foundations of Program Evaluation, (with L. Robinson & C. Lu) of ES: A Computer Program and Manual for Effect Size Calculation, co-editor of five other volumes, and the author of numerous articles and chapters. He was 1997 President of the American Evaluation Association, winner of the 1994 Paul F. Lazarsfeld Award for Evaluation Theory from the American Evaluation Association, winner of the 2000 Robert Ingle Award for service to the American Evaluation Association, winner of the 1994 and 1996 Outstanding Research Publication Awards from the American Association for Marriage and Family Therapy, and a past editor of New Directions for Program Evaluation.