By Ciara Keenan
This article was first posted on the Meta-evidence website on 12 September 2018.
We are ecstatic to welcome Dr Joshua R. Polanin to the meta-evidence blog this week. Dr Polanin is a Principal Researcher at American Institutes for Research and a methodologist with a passion for improving systematic review and meta-analysis techniques.
Dr Polanin specialises primarily in the areas of Education and Social Sciences, tackling issues such as problem behaviours and school violence, although like many other methodologists, his skills are often used across disciplines.
Dr Polanin is committed to improving scientific research and has contributed enormously to this pursuit through his continued work as Methods Editor in the Campbell Collaboration. Through his work on updating the WWC’s standards and procedures; updating the statcheck R programme which automatically checks the primary authors reported statistical significance tests; and is working on establishing data-sharing agreements between primary authors and meta-analysts to improve transparency and reporting.
In this interview, Ciara and Dr Polanin chat about the capabilities of machine learning; issues associated with missing data; and various other challenges and solutions in the pursuit of improving meta-analysis.
The statcheck program checks primary study statistical significance tests.
You are updating statcheck, tell us more about this project.
The statcheck project is exciting because of what it means for the future of statistical significance reporting in both primary studies and meta-analyses. My colleague Michèle Nuijten really deserves the vast majority of the credit. She and her colleagues developed statcheck as a means to check primary study statistical significance tests. The application behind statcheck does something relatively simple, but in a really efficient way. After converting the pdf to readable text (also no small feat), the program searches for APA-reported null hypothesis significance tests, recalculates the p-value based on the test statistic and degrees of freedom, then checks the calculated p-value against the author reported p-value. It’s quite ingenious! My contribution to this enterprise was to help Michèle think through what significance tests needed to be added for meta-analyses. It turns out, Michèle and her team had programmed most the of the tests, but a few outstanding tests remained. We ran the program on over 400 meta-analyses, and not surprisingly, meta-analysts make errors and sometimes errors that suggested the authors deliberately mis-reported a p-value. Of course the theory is that the deliberate misreporting is in favor of significant results so that the paper is more easily publishable. Our hope is in the future, all published papers will be put through the program before publication.
Do you hold any concerns about the capabilities of machine learning in meta-analysis?
I honestly don’t hold many concerns about machine learning (ML) for meta-analysis. The possibilities for ML application in systematic review and meta-analysis are boundless. Each stage – searching, screening, coding, analysis, and really even summary – can and will eventually be done through ML. Researchers are already working on many of these applications. Applications are readily available for abstract screening; I use Abstrackr, but many other programs are available that do about the same thing. After a pilot round where researchers screen 30-50 of the same citations, the program will use those decisions to analyze the remaining citations and then make predictions about whether the citation is eligible or not. It then rank orders the citations based on the probability of their inclusion. Let me tell you, for large-scale projects, it works really well!
Because you asked the question, though, I do have one concern: lack of content knowledge driving the synthesis decision-making. The more I work on reviews the more I recognize how important content knowledge is for the review process. Without sufficient input from experts, a review is rendered useless. To the extent that we can ensure experts are included in the process, both now and in the future of ML reviews, we will produce higher quality and impactful products.
If you were interested in exploring age as a moderator to the effectiveness of an intervention, but only 20% of the primary research reported this demographic variable, what would be your next steps?
The first step is to try to exhaust all possible opportunities to increase the percentage of studies where age can be inferred, either directly or indirectly. One of the things I do with my team, after we have a sufficient number of studies coded, is to go back through each study and see what is missing. The question of exploring age is a good one because age is a variable that can be inferred through various means. For example, you might be interested in the average age of the sample. But only the grade levels are reported or maybe only the school level (elementary, middle, etc.). While it is not a direct report from the authors, that information will provide you with an estimate of age. If that’s not available, then I look for other reports of the same study. Maybe an initial report of the same sample is available? Maybe the dataset has been published publicly? Maybe the authors wrote a conference abstract that included age? A 5-minute Google search will often provide these answers.
Failing these options, I next turn to the authors. It’s a tedious process, but especially for missing effect size information or important moderators, an author query can illicit increased reporting. A conservative response rate is usually around 20%. Let’s say you have 40 studies in your example, and 20% report age (8 studies). Of the 32 remaining, if 20% respond with an answer, you’ve almost doubled the number of reports with age!
So what to do with the remaining missing studies? Well this is where things breakdown and require a lot more assumptions be made. A couple options are available. The first is simply to remove those studies without age reported from any moderator analyses, but that’s not a great option (for the same reasons listwise deletion isn’t a great option in primary research). The second is to assign all of the studies with missing age to the same moderator category, maybe a “missing” group. You have to make the assumption that all of these studies are about the same though. Finally, the third option is to impute the missing age variable. This is an option I use most often. It’s not unlike imputing missing covariates in primary research – and really the same assumptions and sensitivity analyses should be conducted. The advantage, of course, is the ability to use all available information. And so long as the sensitivity analyses don’t indicate any suspicious results, I’m pretty confident in the analyses.
When you peer review meta-analysis, what are some of the quality indicators you look out for?
So many great tools are available to help peer reviewers, and meta-analysts, read and review the reporting of meta-analyses. PRISMA and MARS are two of the most frequently used tools, but many others exist. These tools should really be used by anyone peer-reviewing a meta-analysis for the first time, or the first time in a long time.
Perhaps not surprisingly, I’ve read and reviewed a lot of meta-analyses. So for me, at this point, I review studies looking for more and more advanced techniques. I’ll start by looking for the basic information: what databases were searched, how were studies screened, what type of effect sizes were coded, what type of model was used, what types of moderator analyses were conducted, how were the results reported?
Probably about half the studies I read or review fail some aspects of these baseline characteristics. But assuming they don’t, I turn to more complex aspects: how was the search string developed and search terms derived, were multiple screeners and coders (or researchers checking each decision) used, how did the authors address effect size dependency, was a meta-regression model with multiple covariates applied, did the authors explain what the results mean in an easy to understand way? The vast majority of reviews don’t use all of the techniques described.
One last thing I’ve started looking for is how the author report the results extracted from each primary study. More and more I believe that meta-analysts have a responsibility to report the results of each study included. Meta-analyses simply cannot be considered static, one-off reports any more. The results from these studies will need to be updated, or used in overviews, or dissected for methodological reviews – and each and every result from each primary study simply needs to be accurately reported! In 5-10 years after the results of a meta-analysis are first reported, an older you or a different set of meta-analysts entirely will need to use the data within that review. Having each study reported clearly will ensure all available data can be used.
How should meta-analysts deal with statistically dependent effect sizes? Are there programmes to help with this analysis?
I really, truly believe we will look back at the 2010’s as the decade where meta-analysts figured out how to use and model dependent effect sizes. And I’ve gone on record, and will do so here again, to say: modeling dependent effect sizes is one of the single most important advances to meta-analysis.
Before I explain what meta-analysts should be doing, let’s briefly discuss what meta-analysts used to do. (Tangent: “used to do” is not quite accurate. Meta-analysts are still doing the things I’m about to say! But thankfully the trend is decreasing each year.) In the past, meta-analysts had several choices available when dependent effect sizes arose. A meta-analyst could: (1) ignore effect sizes (terrible option), (2) create decision rules for selecting only one effect size per study (also a bad option), (3) average effect sizes within each study (a better option, but still not great), or (4) use multivariate meta-analysis IF the study authors reported the correlation between the dependent effects (not likely). Without new techniques, meta-analysts spend paragraphs defending which of the bad choices they chose! Future meta-analysts are going to think this was the dark ages of research synthesis!
Then Larry Hedges, Beth Tipton, and Matthew Johnson, in 2010, wrote the seminal paper on modeling dependent effect sizes using robust variance estimation (RVE). The technique allows the meta-analyst to synthesize all available effect sizes within each study, without knowing the correlation between effects. Now, meta-analysts don’t have to choose between lots of bad choices! Every single available effect size can be calculated and incorporated into a model. Even better, now a meta-analyst can analyze variance at the study-level OR the effect size level, too! This has led to greater and greater increases in the size of meta-analyses and will continue to push the usefulness of the technique.
Since the seminal paper on RVE, Beth and some of her colleagues, most notably James Pustejovsky, have continued to advance the technique. Beth and James are advancing the field again by incorporating the underlying technique into existing multilevel modeling frameworks. These are implemented easily in the R environment using the metafor and clubsandwich packages. Because this is a developing field, however, the techniques continue to be updated and fewer resources are available to the applied meta-analyst. As such, in terms of software, I’d suggest using the robumeta package that Beth originally developed with her student Zach Fisher. Beth, Emily Tanner-Smith, and I wrote a paper that explains the technique and how to use it in the R and Stata platforms, too (Tanner-Smith, Tipton, Polanin, 2016).
Finally, while I’m plugging my own papers, I’ll mention that I wrote a paper that reviewed all available meta-analytic R packages. My colleagues, Emily Hennessy and Emily Tanner-Smith, and I (Polanin, Hennessy, & Tanner-Smith, 2017), at the time of publication, found 63 packages that will implement some aspect of meta-analysis. Some of them handle dependent effect sizes (see page 211). There are many other things the packages will do as well!
As an experienced systematic reviewer, how easy have you found it to get primary authors to share their data? What challenges have you have faced?
It’s a challenge! And historically, primary authors have been unwilling to share their data for meta-analyses. Two recent analyses investigated the response rates from primary authors when meta-analysts ask for individual participant data, and both indicated that primary authors are still pretty reluctant (Nevitt et al., 2017; Polanin, 2018).
Meta-analysts must overcome a number of obstacles to retrieve data from primary authors. Logistics like finding email addresses, keeping track of who to send author queries (AQ) to, and knowing what to say in the query slow the process down. If the authors respond, the next step is to convince them to help. This usually means reassuring the authors that the data will only be used for the meta-analysis, stored properly, and won’t be given away for other projects. Even when authors agree, these types of requests tend to be at the end of their lists, so simply getting the authors to respond with the analyses is a challenge!
One additional challenge we’ve been facing recently results from the type of effect sizes we’ve been calculating. For a project funded by the National Institute of Justice, we are synthesizing regression coefficients (transforming them into semi-partial correlations) from longitudinal, multiple regression models where school violence predicts later internalizing behavior and attitude problems (among other outcomes). In a few cases, we have found datasets that fit our eligibility criteria. While this is an excellent addition to our meta-analysis, we did not anticipate the need to make decisions about how to analyze the datasets! For example, a dataset might contain various school violence variables at time 1, along with a host of covariates, as well as several outcomes. The challenge is to determine what covariates are added, and in what order, so as to be as consistent as possible across the other studies. This isn’t a problem for these types of effect sizes either: this same problem can and does arise when analyzing treatment effects! The meta-analyst essentially has to decide, preferably a priori, what variables are important and how they will be applied across all available datasets.
Regardless of the challenges, I’m encouraged by the response. My sense is that this will continue to become the norm. Eventually these types of requests will become commonplace – and the usefulness and robustness of meta-analytic results will follow.