C2 Implementation Process Methods Group

Implementation Process Methods Group

Protocol April 2001

1. Introduction to the group:

The Campbell Collaboration Steering Group agreed to establish the Implementation Process Methods Group in February 2001. The Group will undertake methodological studies relevant to the work of the collaboration and provide advice, support and training to groups and individuals undertaking Campbell reviews. In this protocol we describe the background to our work including the ways in which studies of the processes of implementation can contribute to systematic reviews of evidence; we highlight some of the methodological issues that arise from extending systematic reviews in this way; and outline our objectives for 2001-2003. The membership of the group are currently agreeing how the work will be taken forward and further details will be posted on the web-site as soon as they are available. We welcome enquiries from people interested in contributing to the work of the group, which is described more fully below. If you are interested in getting involved or would like more information about the group – as a researcher, policy maker, service provider or consumer - please contact one of the convenors whose details are provided below.

2. Contacting the convenors:

This Methods Group has two convenors combining methodological and policy/practice expertise. The policy/practice convenor signals the importance given to need for Campbell systematic reviews to be of maximum relevance to anticipated users of systematic reviews including consumers of services. The convenors locations – in the UK and Australia - also reflects the importance we attach to developing international networks within the Campbell Collaboration. We are currently seeking a third convenor, with methodological expertise relevant to the work of the group, preferably from outside the UK and Australia. Contact details for the convenors are provided here. Brief biographical information about them can be found in the final section of this protocol.

Primary Contact: (Methods convenor)
Professor Jennie Popay (Primary Contact)
Nuffield Institute for Health
University of Leeds
Fairburn House
71 – 76 Clarendon Road,
Leeds LS2 9PL, UK
Telephone +44 113 233 6983/6901
Facsimile +44 113 246 0899
E-mail J.M.Popay@leeds.ac.uk

Policy/practice convenor
Roger Dunston
(Convenor, AASW National Special Interest Group in Evidence Based Practice in Social Work),
Deputy Head, Division of Allied Health Services
C/O Royal North Shore Hospital
St Leonards,
NSW 2065
Australia
Telephone 61 (02) 9926 7580
Email: Roger.dunston@pgrad.arts.usyd.edu.au

3. Background to the work of the group

3.1 The contribution of evaluative studies of implementation processes to systematic reviews

The Campbell Collaboration has been established to prepare, maintain and disseminate systematic reviews of the effects of social and behavioural interventions in major areas of public policy. These include: education; delinquency and criminal justice; mental health and substance abuse; and other areas of social policy including social services social welfare social work, housing, employment and training. The findings of high quality randomised controlled trials are central to this endeavour. However, the Campbell Collaboration recognises that findings from studies designed to evaluate the process of implementing interventions and to study the experience of all those involved in providing and receiving interventions have an important role to play in informing policy and practice decision-making in these and other fields.

Social and behavioural interventions are typically complex and multi-faceted. In this context, the processes involved in implementation can powerfully shape the impact of interventions. In the past a lack of interest in studying the processes involved in implementation may have impeded the successful development of such interventions. Today, many more evaluative studies include a focus on the processes involved in implementing interventions often, albeit not always, embedded within experimental studies of outcomes.

Process evaluations focus on the way in which interventions are implemented. The concern is with the internal dynamics and actual operations of a program/intervention, in order to understand its strengths and weaknesses. As Patton (1990) has suggested:

'if program implementation is characterised by a process of adaptation to local conditions, needs and interests, then the methods used to study implementation must be open-ended, discovery-oriented, and capable of describing developmental processes and program changes'[1].
Frequently, these studies also involve iterative feedback as the evaluation proceeds, to those involved in implementation in order to contribute to the further development of an intervention as it evolves.

To manage this dynamism, process evaluation studies are characterised by a flexible approach to data collection and the use of a plurality of methods generating a range of different types of data. Significantly, they frequently encompass both quantitative and qualitative methods. These include: the collation of routine data, surveys generating quantitative data, in-depth qualitative interviews producing narrative data, participant and non-participant observation, focus groups and ethnographic fieldwork. Inevitably, given this range of data types, there are also diverse approaches to analysis in process evaluations. These include the use of statistical packages such as SPSS as well as various approaches to the analysis of qualitative data including grounded theory approaches and software packages such as Nudist and Ethnograph. The identification and refinement of thematic categories is a major aspect of qualitative analysis.

Different methods generate different kinds of findings. Overall, however, process evaluations can help us to understand the way in which an intervention is experienced by all of those involved in developing, delivering and receiving interventions, what aspects of the intervention they value, or not, and why. These types of evaluations can provide insight into the factors, including aspects of professional, managerial and/or consumer behaviour and external factors such as other policy developments, which facilitate or hinder successful implementation of a programme or service and how it might need to be adapted for large scale roll out. Process evaluations may also generate qualitative data on the outcomes of interventions. The findings from process evaluation studies can therefore help to answer questions about the impact, appropriateness and acceptability of interventions and thus enhance the scope, relevance and utility of effectiveness reviews.[i]

Findings from evaluative studies of the process of implementation can therefore make a number of specific contributions to systematic reviews of the effectiveness of particular interventions. These include:

influencing the criteria by which effectiveness is to be judged and hence the process and outcome data to be extracted from trials;
providing a research-based context for interpreting and explaining trial results including insights into how change was achieved, whether it could be achieved more cost-effectively and the factors that constrained/facilitated change;
contributing to understanding of heterogeneity in outcomes;
providing evidence on the subjective experience of all those involved in developing, delivering and receiving an intervention;
revealing the extent to which the criteria adopted to judge effectiveness are actually adopted in policies and practice;
contributing to the methodological development of systematic reviews by expanding the existing concern with publication bias to include other forms of bias in the production of systematic reviews.

3.2 Methodological Issues for Systematic Reviews

The inclusion of evaluative studies of the processes of implementation into systematic reviews of evidence raises a number of important methodological issues. At a general level, methodological work is required to determine what type of process evaluations are best included in systematic reviews. The extent to which evaluative research has considered the process aspects of interventions varies between substantive policy areas and across research traditions in different countries. So, for example, education research has a long tradition of process evaluation studies embedded within experimental outcome oriented evaluative studies, particularly in the USA. In contrast, in the social welfare field experimental outcome oriented research designs less frequently include evaluation of implementation processes. In other fields of social/public policy evaluative studies involving experimental designs are particularly rare. For example, a recent review of the research literature on the links between regeneration/urban renewal programmes and population health has highlighted the diversity in evaluative approaches in this field and the dearth of studies using experimental designs. The methodological issues involved in the systematic review of studies of implementation processes may therefore vary across policy fields. Given this diversity, however, a key question is whether it is ‘reasonable’ to apply findings from independent process evaluation studies to the results of a systematic review of RCTs or whether only process evaluations embedded within relevant RCTs should be included.

A number of methodological issues arise from the systematic review process itself. Specifically, there is a need to develop methods for identifying, recording, appraising and extracting data from process evaluation studies whether these are embedded within experimental or quasi-experimental studies of effect or stand alone process evaluations. In terms of identifying relevant studies, there are implications for the development of registers of studies and for search strategies. Most registers of RCTs, for example, do not routinely record the availability of data from process evaluations collected during RCTs and/or as part of linked studies. Whilst the ideal may be for process evaluation to be bedded within experimental designs evaluating effect, the reality is that many relevant process evaluations are conducted independently of experimental studies. The range of literature relevant to a systematic review of the process of implementing an intervention may be very wide and cross disciplines. Conventional search techniques may not pick up relevant material. Search strings to identify evaluative studies of processes of implementation need to be further developed.

Once identified there is a need to assess the quality of evaluative studies of processes of implementation. Much of the data produced is qualitative. Critical appraisal tools are being developed which seek to deal with the interpretative nature of the qualitative research process but there is a need to review, test and disseminate the different models becoming available. Most frequently, however, process evaluations produce both quantitative and qualitative data and work is needed on the development of appraisal tools for studies using mixed methods. Finally, there is a need for methodological work on the processes by which findings from different studies (generating a range of types of data) can be synthesised and/or summarised and combined with the findings on effect size without compromising the need to minimise bias. Here there is a more general issue that needs to be addressed. Given the importance of ‘context’ to understanding processes of implementation, there is a need for methodological research to clarify the basis on which findings can be generalised across studies undertaken in different socio-cultural conditions.

The Campbell Implementation Processes Methods Group has been established to develop and support methodological work on the issues associated with the inclusion in systematic reviews of findings from studies of implementation processes. This Methods Group is working closely with the Cochrane Qualitative Methods Network also co-ordinated by Professor Jennie Popay. The Network is currently developing a proposal to establish a Cochrane Qualitative Methods Group, which should be submitted to the Cochrane Collaboration Steering Group for registration in spring 2001. There is substantial commonality in the methodological issues facing these two methods group and overlap in the people registering an interest in being involved in the work of the groups. The expectation is that the Cochrane Methods Group will focus particularly on methodological matters arising from the inclusion of findings from qualitative studies into systematic reviews. Such studies may form part of process evaluation but this is not necessarily the case. For example, research into the experience of people receiving interventions may use qualitative methods but would not be process evaluations. The Campbell Methods Group will focus particularly on methodological issues associated with process evaluations. As already noted these may include qualitative methods but typically involve multiple methods. The convenors will ensure that work is not duplicated across the two Methods Groups and it is anticipated that the outputs from the two groups will be available to people involved in both Collaborations.

4. Scope of the group

The goals for the Campbell Implementation Process Methods Group are to contribute to the Campbell Collaboration in the five ways set out below.

4.1 Undertake research on, and disseminate, methodological standards for:

deciding on which process evaluation studies can be included in systematic reviews;
searching for studies of implementation processes relevant to Campbell reviews;
critically appraising evaluative studies of implementation processes;
generalising and synthesising data from different process evaluation studies;
combining findings from process evaluations with studies of effect;
drawing conclusions and making recommendations from syntheses including process evaluations.

4.2 Provide advice for the Campbell Steering Group and individual Review Groups on methods and methods policy in relation to the evaluation of implementation processes. This advice would be provided through a network of people with expertise and experience of evaluative research on implementation processes. Where appropriate, and when asked, members of the methods group would serve as members of Campbell Collaboration Review Groups.

4.3 Provide a mechanism for quality assurance of review protocols and completed reviews that incorporate findings from studies of implementation processes.

4.4 Provide training and other resources (e.g. methodological guidance) for members of
Campbell Review Groups.

4.5 Provide a forum for discussion and debate about (i) the contribution evaluative studies of implementation processes can make to the systematic review process and (ii) the development of rigorous and systematic methods to enhance this contribution.

4.6 To work with other Campbell Methods Groups and Methods Groups within the Cochrane Collaboration, particularly the proposed Qualitative Methods Group to:

encourage transparency of, and learning about, method developments;
encourage and facilitate liaison and sharing with other methods groups on topics of common interest.

5. Objectives for 2001 – 2003

In the context of the above goals we have developed short terms objectives for the Methods Group to achieve during its first 24 months.

Objective 1 In-augural meeting – February 2001

Since spring 2000, an international network of people has been involved in discussions about the proposed Campbell Implementation Process Methods Group through a workshop held in London in May 2000 and electronic communication. Additionally, at the first annual Campbell Colloquium held in Philadelphia in February 2001 a small group of people also meet to discuss the future work of the group and the proposed objectives. The Group’s objectives for the next two years are set out below and the membership is currently discussing how these are to be taken forward. The currently membership of the group is listed at the end of this protocol.

Objective 2: To begin work on a bibliography of methodological work on systematic reviews of process evaluations.

During 2001-2 we expect to begin to develop a bibliography of methodological work relevant to the inclusion of process evaluations into systematic reviews. However, there are significant questions to be resolved concerning the feasible scope for such a resource. Ideally, depending on the range and quality of the literature identified, this could be incorporated into the Campbell register of methodological studies. Discussions will also take place regarding the inclusion of process evaluation studies within the Campbell Register of non-experimental studies.

Objective 3: Developing frameworks for quality appraisal of process evaluations

An early objective for the Group is to identify existing frameworks for quality appraisal relevant to the mixed methods that characterise evaluative studies of implementation processes. These will be disseminated to Campbell Review Groups via the methods groups website. In the longer term – in the second year and beyond - these will be subjected to critical appraisal and review and recommendations would be made for use in systematic reviews. The results of this work will be incorporated into any review handbook/guidance produced by the Campbell Collaboration. Draft guidance on the inclusion of findings from qualitative research into systematic reviews has already been produced by one of the convenors. This will be circulated to Campbell Review Groups during 2001-2002 as well as forming the basis for work extending the guidance to mixed method studies.

Objective 4. Develop proposals for funding for methodological research

Some of the methodological work the Group would wish to develop will require substantial external funding. Proposals for work in two areas will be developed during 2002-3. The first of these will be concerned to explore the relatively advantages and disadvantages of using findings from process evaluations embedded within RCTs compared with those undertaken independently. The second will be concerned to explore the potential for generalising from process evaluation studies and to develop approaches to synthesising findings from different process evaluations. Although methods are available that are relevant (e.g. triangulation), the application of these methods to systematic reviews remains to be developed. During 2001 preparatory work will begin involving identification of existing relevant methodological literature. This will be linked to the development of the methodological bibliography. In the second year, this literature will be reviewed and proposals for new empirical methodological work will be developed.

Objective 5. Training

During the first year the Group will place less emphasis on providing training opportunities than on the methodological work. There is, however, an urgent need for people to share their experiences of attempting to incorporate findings from process implementation studies into systematic reviews of effects and to clarify the contribution these studies can make to systematic reviews. The group will therefore seek funding to organise an international seminar early in 2002. This would involve people with direct experience of attempting to review process implementation studies in the context of systematic reviews of effect.

In terms of formal training opportunities, an early objective for the group will be to map the potential areas in which training could be developed and to explore with Campbell Review Groups what their /needs preferences for training in this sphere would be. It is anticipated that the first formal training session will be provided at the 2002 Campbell Collaboration meeting. It is likely that this will be a general introduction to methodological issues in the systematic review of process evaluations. As the methods for critical appraisal and synthesis of process implementation studies develop and more people have experience of combining these studies into systematic reviews, the capacity for providing formal training initiatives on more detailed topics will grow.

Given the range of methods used within studies of the processes of implementation, a diversity of methodological matters could be the focus of such training. The critical appraisal of studies using qualitative methods, for example, is an important aspect of the systematic review of process evaluation studies. A number of group members have experience of running training sessions in this area. Members also have experience of the appraisal and synthesis of findings from mixed methods studies that could form the basis of training initiatives. Additionally, although at an early stage, approaches to the incorporation of findings from process implementation studies into systematic reviews of effects are emerging and training on these could be developed in the longer term.

6. Biographical Details of Methods Group Convenors

Jennie Popay is Professor of Sociology and Health Policy at the Nuffield Institute for Health at the University of Leeds in the UK. She has researched and published extensively on aspects of public policy in the fields of social welfare, employment, public health, health and social care and gender and family relationships. Her primary substantive research interests are into the causes of social inequalities in health, social relationships of care; professional practice and ideology; and the sociology of knowledge. She has a particular interest in exploring the distinct theoretical contribution lay experiential knowledge, in narrative form, can make to understanding of the meaning and impact of social inequalities alongside codified academic knowledge. Her work has involved the development of new conceptual models of the relationship between research, policy and practice and practical applications of these models in the health field. She has experience of a wide range of research methods, including the design and conduct of national evaluations of the process of implementing major health care reforms. She has made a particular contribution to thinking about the role of findings from qualitative research in evidence-based policy/practice. She currently serves on the editorial board of the European Journal of Epidemiology and Community Health and the Journal of Health Education and has been on the editorial board of the Sociology of Health and Illness. She has been a member of many professional and/or academic committees and groups including the national committees of the Medical Sociology Group of the British Medical Association and the Society of Social Medicine. Her current academic related work includes membership of the Medical Research Council’s Advisory Group, the Public Health Statistics Advisory Group of the Office of National Statistics; the Advisory Group for the ESRC funded Future of Welfare Research Group and several research commissioning panels at national and local levels. She established and currently convenes the Cochrane Qualitative Methods Network and has written guidance on the inclusion of findings from mixed methods studies into systematic reviews of effect in the Systematic Review handbook published by the Centre for Reviews and Dissemination at York University in the UK. Recent publications include a jointly edited book: Williams, F, Popay, J. Oakley, A. 2000, Welfare Research: A Critical Review, University College Press.

Roger Dunston is Director of the social work department at Sydney’s Royal North Shore Hospital. He is a social worker with many years’ experience as a practitioner and senior manager. He has a long-standing interest in linking research-based evidence to developments in social welfare practice and policy and is currently completing his doctorate at Sydney University. In recent year he has been pivotal in developing a national special interest group in evidence based practice in social work. This group is currently in the process of establishing an Australian Centre for Evidence Based Practice in Social Work. Whilst he is not an expert in research methods, Roger is strongly research literate and brings to the methods group a strong commitment to the development of evidence-based approaches in the social welfare field and extensive academic and professional links within Australasia.

[1] Patton, M.Q. (1990) Qualitative Evaluation and Research Methods, 2nd Edition, Sage, London.