Better evidence for a better world

Campbell evidence and gap maps

Coming soon – Campbell EGMs are a new evidence synthesis product. Plain language summaries of our EGMs will be published on this website. The interactive EGMs and full EGM reports will be available in our journal on the Wiley Online Library platform: click here.



Learn more about Campbell EGMs

Campbell-partnered EGMs

Campbell has produced maps on other topics, sometimes in partnership with other organisations.



See the Campbell-partnered EGMs
Better evidence for a better world
Better evidence for a better world

Better evidence for a better world (177)

Additional Info

  • Authors Anthony A. Braga, Brandon Turchan, Andrew V. Papachristos, David M. Hureau
  • Published date 2019-09-08
  • Coordinating group(s) Crime and Justice
  • Type of document Review Plain language summary
  • Title Hot spots policing of small geographic areas effects on crime
  • Library Image Library Image
  • See the full review https://onlinelibrary.wiley.com/doi/full/10.1002/cl2.1046
  • English

    PLAIN LANGUAGE SUMMARY

    Hot spots policing is associated with reductions in crime

    Hot spots policing is associated with small but meaningful reductions in crime at locations where criminal activities are most concentrated. Focusing police efforts at high activity crime places is more likely to produce a diffusion of crime prevention benefits into areas adjacent to targeted hot spots than crime displacement.

    What is this review about?

    Crime is concentrated in small places, or "hot spots," that generate half of all criminal events. Hot spots policing focuses police resources and attention on these high crime places. For the purpose of this review, hot spots programs must have consisted of police-led crime prevention efforts that targeted high-activity crime "places" rather than larger areas such as neighborhoods.

    This review considers both randomized controlled experimental and quasi-experimental evaluations of the effects of hot spots policing interventions on crime where the control group in each study received routine levels of traditional police enforcement tactics.

    What is the aim of this review?

    This Campbell systematic review assesses the preventive effects of focusing police efforts on crime "hot spots" as compared to traditional police crime control strategies.  The review summarises evidence from 65 studies containing 78 tests of hot spots policing interventions, including 27 randomized controlled trials and 38 quasi-experimental evaluations.

    What studies are included?

    A total of 65 studies containing 78 tests of hot spots policing interventions were identified. However, standardized effects sizes were only calculated for 73 main effects tests due to reporting deficiencies in three included studies.

    All studies were published from 1989 to 2017: 51 studies were conducted in the USA, four in the UK, four in Sweden, and six in other countries.

    What are the main findings of this review?

    Does focusing crime prevention efforts on crime hot spots reduce crime?

    Yes. Hot spots policing generates statistically-significant small reductions in overall crime and disorder in areas where the strategy is implemented.

    These crime control gains were evident across specific categories of crime outcomes including drug offenses, disorder offenses, property crimes, and violent crimes.

    Does policing crime hot spots inevitably produce crime displacement effects?

    No. Overall, it is more likely that hot spots policing generates crime control benefits that diffuse into the areas immediately surrounding the targeted locations than displacing crime into nearby locations.

    What do the findings of the review mean?

    Findings from this review support hot spots policing as a proactive crime reduction strategy. Police departments should incorporate focusing resources at high-activity crime places as part of their broader approach to crime prevention.

    The majority of studies included in the updated review have been published since the previous iteration of the review and utilized rigorous research designs.

    Despite the drastic increase in eligible studies, only one study conducted a formal cost-benefit assessment of the hot spot policing intervention. The growth of hot spots policing warrants further empirical attention on the efficiency of hot spots policing for reducing crime.

    How up-to-date is this review?

    The review authors searched for studies up to February 2017.

Additional Info

  • Authors Hugh Waddington, Ada Sonnenfeld, Juliette Finetti, Marie Gaarder, Denny John, Jennifer Stevenson
  • Published date 2019-08-02
  • Coordinating group(s) International Development
  • Type of document Review Plain language summary
  • Title Citizen engagement in public services in low‐ and middle‐income countries: A mixed‐methods systematic review of participation, inclusion, transparency and accountability initiatives
  • Library Image Library Image
  • See the full review https://onlinelibrary.wiley.com/doi/full/10.1002/cl2.1025
  • English

    PLAIN LANGUAGE SUMMARY

    Citizen engagement improves access to public services in low- and middle-income countries, but evidence on development outcomes is limited

    Interventions promoting citizen engagement in public service management involve participation, inclusion, transparency and accountability (PITA) mechanisms. In low- and middle-income countries (LMICs), these interventions are effective in improving active citizenship and service delivery, and may improve the responsiveness of service provider staff for services provided directly by public servants (for example, in health).

    In contrast, interventions providing information to stimulate pressure on politicians are not usually effective in improving provider response or service delivery. There is insufficient evidence to conclude whether these interventions are effective in improving wellbeing or the relationship between citizens and the state.

    What is this review about?

    Failures in governance lead to the exclusion of large portions of society from public services and to waste, fraud and corruption. This review assesses evidence for interventions promoting better governance of public services: participation (participatory planning), inclusion (involvement of marginalised groups), transparency (information about citizen rights or performance of public officials), and accountability (citizen feedback) mechanisms, known collectively as PITA mechanisms.

    What is the aim of this review?

    This Campbell systematic review examines the effects of interventions to promote citizen engagement in public service management. The review synthesises evidence from 35 impact evaluations and 36 related studies of interventions promoting participation, inclusion, transparency and accountability (PITA) mechanisms.

    What studies are included?

    The review includes impact evaluations relating to 35 PITA programmes from 20 LMICs. In addition, 36 qualitative and programmatic documents were included to strengthen understanding of implementation context and programme mechanisms.

    What are the main findings of this review?

    Citizen engagement interventions (i) are usually effective in improving intermediate user engagement outcomes, for example, meeting attendance and contributions to community funds; (ii) improve access to and quality of services but not service use outcomes; (iii) can lead to improvements in some wellbeing outcomes such as health and productive outcomes; (iv) may improve tax collection; but (v) do not usually lead to changes in provider action outcomes such as public spending, staff motivation and corruption. There may be an exception where there is direct interaction between citizens and service providers in the regular delivery of services. Interventions providing performance information do not generally improve access or lead to improvements in service quality.

    Only interventions focused on services delivered by front-line staff (e.g., in health) achieve positive outcomes. Those delivered without public interaction (e.g., roads) do not. However, engagement with civil society organisations and interest groups may lead to better outcomes for services accessed independently of providers. Inclusive citizen engagement programmes have at least as big an effect on user engagement and access to services as less inclusive approaches.

    Many interventions experienced challenges stemming from a lack of positive engagement with supply-side actors, whose power the interventions often sought to diminish. Interventions implemented with the strong support of the targeted service providers were better able to realise positive impacts.

    Approaches to citizen-service provider engagement appear to work more effectively when implemented through phased, facilitated collaborative processes rather than one-off accountability meetings that are seen as confrontational.

    Only four studies present any data on intervention costs. This limited the potential for any analysis of comparisons across programmes and settings.

    In interpreting the findings, it must be noted that each individual outcome is reported in only a few studies and that included studies have important methodological weaknesses with risks of bias arising from weak design, analysis and reporting.

    What do the findings of this review mean?

    For policy and programme managers: A collaborative rather than confrontational approach with the service providers whose services are under scrutiny is more likely to be effective. Engaging communities may require using civil society organisations to facilitate the community’s participation. Programme design should ensure positive engagement with supply-side actors within the intervention setting.

    For researchers: More high-quality studies are needed, comparing different approaches to improving service delivery, paying attention to complete description of the different approaches being compared. Since implementation is a crucial factor, mixed methods studies should be the norm, and will help focus on equity considerations which have been neglected. Finally, there should be standardisation of indicators in PITA studies.

    How up-to-date is this review?

    The review authors searched for studies up to March 2018.

Additional Info

  • Authors Takayuki Harada, Hiroshi Tsutomi, Rintaro Mori, David Wilson
  • Published date 2019-07-25
  • Coordinating group(s) Crime and Justice
  • Type of document Review
  • Title Cognitive-behavioural treatment for amphetamine-type stimulants (ATS)-use disorders
  • See the full review https://onlinelibrary.wiley.com/doi/10.1002/cl2.1026
  • English

    PLAIN LANGUAGE SUMMARY

    Cognitive‐behavioural treatment for amphetamine‐type stimulants‐use disorders

    What was the aim of this review?

    The aim of this Cochrane review was to find out whether cognitive‐behavioural treatment (CBT) is effective to treat people with amphetamine‐type stimulants (ATS)‐use disorders. Researchers in the Drugs and Alcohol Group of Cochrane collected and analysed all relevant studies to answer this question and found two studies.

    Key messages

    The current evidence was inadequate to draw any firm evidence‐based treatment recommendations for the client population.

    What was studied in the review?

    ATS are a group of synthetic stimulants and their use has been widespread globally. These types of drugs are highly addictive and prolonged use may result in a series of mental and physical symptoms including anxiety, confusion, insomnia (difficulty sleeping), mood disturbances, cognitive impairments (difficulty thinking and understanding), paranoia (irrational feeling that people are 'out to get you'), hallucinations (where someone experiences something that does not exist outside their own mind) and delusion (a mistaken belief).

    Currently there is no widely accepted treatment for ATS‐use disorder. However, CBT is often the first choice of treatment. It is a psychological treatment (talking therapy) approach to modify distorted thoughts and beliefs, and maladaptive behaviours (things that people do to stop them from adjusting to situations). The effectiveness of CBT for other substance‐use disorders (e.g. alcohol‐, opioid‐ and cocaine‐use disorders) has been well documented and as such this basic treatment approach has been applied to the ATS‐use disorder. These types of therapies are expected to prevent relapse and decrease drug use.

    What are the main results of the review?

    The review authors found two eligible studies. Both studies were conducted in Australia. One study compared a single session of brief CBT to a waiting‐list control where participants received no treatment during the study period. One study compared web‐based CBT to a waiting‐list control. Both studies were funded by the Australian Government of Health and Ageing.

    The review showed that when participants received CBT, compared to waiting‐list control, there was no difference. There was insufficient evidence to conclude that CBT was effective or ineffective at treating ATS‐use disorders.

    How up‐to‐date is this review

    The review authors searched for studies that had been published up to July 2018.

Additional Info

  • Authors Robert M. Bernard, Eugene Borokhovski, Richard F. Schmid, David I. Waddington, David Pickup
  • Published date 2019-07-19
  • Coordinating group(s) Education
  • Type of document Review Plain language summary
  • Title 21st century adaptive teaching and individualized learning operationalized as specific blends of student-centered instructional events: A systematic review and meta‐analysis
  • Library Image Library Image
  • See the full review https://onlinelibrary.wiley.com/doi/10.1002/cl2.1017
  • English

    PLAIN LANGUAGE SUMMARY

    Adaptive teaching and individualization for K‐12 students improve academic achievement

    Teaching methods that individualize and adapt instructional conditions to K‐12 learners’ needs, abilities, and interests help improve learning achievement. The most important variables are the teacher's role in the classroom as a guide and mentor and the adaptability of learning activities and materials.

    What is the aim of this review?

    This Campbell systematic review assesses the overall impact on student achievement of processes and methods that are more student‐centered versus less student‐centered. It also considers the strength of student‐centered practices in four teaching domains.

    • Flexibility: Degree to which students can contribute to course design, selecting study materials, and stating learning objectives.
    • Pacing of instruction: Students can decide how fast to progress through course content and whether this progression is linear or iterative.
    • Teacher's role: Ranging from authority figure and sole source of information, to teacher as equal partner in the learning process.
    • Adaptability: Degrees of manipulating learning environments, materials, and activities to make them more student‐centered.

    What is this review about?

    Teaching in K‐12 classrooms involves many decisions about the appropriateness of methods and materials that both provide content and encourage learning.

    This review assesses the overall impact on student achievement of processes and methods that are more student‐centered versus less student‐centered (and thus more teacher‐centered, i.e., more under the direct control of a teacher). It also considers in which instructional dimensions the application of more of these student‐centered practices is most appropriate, and the strength of student‐centered practices in each of four teaching domains.

    What studies are included?

    This review presents evidence from 299 studies (covering 43,175 students in a formal school setting) yielding 365 estimates of the impact of teaching practices. The studies spanned the period 2000–2017 and were mostly carried out in the United States, Europe, and Australia.

    What is the overall average effect of more versus less student‐centered instruction on achievement outcomes? Which demographic variables moderate the overall results?

    More student‐centered instructional conditions have a moderate positive effect on student achievement compared to less student‐centered.

    Which dimensions of instruction are most important in promoting better achievement through the application of more versus less student‐centered instruction? Do these dimensions interact?

    The teacher's role has a significantly positive impact on student achievement; more student‐centered instruction produces better achievement. Pacing of instruction/learning—where learners have more choice over setting the pace and content navigation of learning activities—has a significant effect in the opposite direction; i.e., a significantly negative relationship. There is no relationship between adaptability and flexibility and student achievement.

    There are interactive effects. The teacher's role combined with adaptability produces stronger effects, whereas flexibility (greater involvement of students in course design and selection of learning materials and objectives) has the opposite effect; it reduces the effectiveness of teacher's role on learning outcomes.

    Special education students perform significantly better in achievement compared to the general population.

    Three other factors—grade level; Science, technology, engineering, and mathematics (STEM) versus non‐STEM subjects; individual subjects—do not have any effect on the impact of the intervention.

    What do the findings of this review mean?

    This review confirms previous research on the effectiveness of student‐centered and active learning. It goes further in suggesting the teacher's role promotes effective student‐centered learning, and excessive student control over pacing appears to inhibit it.

    An important element of these findings relates to the significant combination of teacher's role and adaptability, in that it suggests the domain in which the teacher's role should focus.

    Since adaptability relates to increasing the involvement of students in more student‐centered activities, the evidence suggests that instruction that involves activity‐based learning, either individually or in groups, increases learning beyond the overall effect found for more student‐centered versus less student‐centered activities.

    Various student‐centered approaches, such as cooperative learning and peer‐tutoring, have been found to accomplish this goal.

    How up‐to‐date is this review?

    This meta‐analysis contains studies that date from 2000–2017.

Additional Info

  • Authors Brandy R. Maynard, Anne Farina, Nathaniel A. Dell, Michael S. Kelly
  • Published date 2019-07-17
  • Coordinating group(s) Education
  • Type of document Review
  • Title Effects of trauma-informed approaches in schools
  • See the full review https://onlinelibrary.wiley.com/doi/10.1002/cl2.1018
  • English

    PLAIN LANGUAGE SUMMARY

    The review in brief

    Despite growing support and increased rate of which trauma‐informed approaches are being promoted and implemented in schools, evidence to support this approach is lacking.

    What is this review about?

    Exposure to different types of trauma have been associated with varying types and complexity of adverse outcomes, including adverse effects on cognitive functioning, attention, memory, academic performance, and school‐related behaviors. Given the growing research on trauma and increased knowledge about the prevalence, consequences and costs associated with trauma, there have been increased efforts at the local, state and federal levels to make systems “trauma‐informed” (Lang, Campbell, & Vanerploeg, 2015). While the intent of creating trauma‐informed approaches in schools is a noble one, relatively little is known about the benefits, costs, and how trauma‐informed approaches are being defined and evaluated (Berliner & Kolko, 2016). Adopting a trauma‐informed approach in a complex system such as a school building or district is a time consuming and potentially costly endeavor and thus it is important to assess the effects of this approach to inform policy and practice.

    This aim of this review was to assess trauma‐informed approaches in schools on trauma symptoms/mental health, academic performance, behavior, and socioemotional functioning. Trauma‐informed approaches include programs, organizations, or systems that realize the impact of trauma, recognize the symptoms of trauma, respond by integrating knowledge about trauma policies and practices, and seeks to reduce retraumatization. At least two of the three key elements of a trauma‐informed approach must have been present: Workforce development, trauma‐focused services, and organizational environment and practices, which differ from trauma‐specific interventions designed to treat or otherwise address the impact/symptoms of trauma and facilitate healing.

    What is the aim of this review?

    This Campbell systematic review sought to examine the effects trauma‐informed schools on trauma symptoms/mental health, academic performance, behavior, and socioemotional functioning. Although we conducted a comprehensive search to find studies testing trauma‐informed approaches in schools, no studies met the inclusion criteria.

    What are the main findings of this review?

    No studies met criteria for this review, indicating that there is a lack of evidence of trauma‐informed approaches in schools.

    What do the findings of this review mean?

    Despite widespread support and growing adoption of trauma‐informed approaches in schools across the globe, we found no studies to provide good evidence to suggest that this approach is effective in achieving the stated goals. Given the degree to which trauma‐informed approaches are being adopted in schools across the US and other countries, it is important that the effects of these programs be assessed.

    How up‐to‐date is this review?

    The review authors searched for studies June through September, 2017.

Additional Info

  • Published date 2019-06-01
  • Title Campbell partnership evidence and gap maps (EGMs)

Additional Info

  • Authors Padraic Fleming, Sinead McGilloway, Marian Hernon, Mairead Furlong, Siobhain O’Doherty, Fiona Keogh, Tim Stainton
  • Published date 2019-01-25
  • Coordinating group(s) Disability
  • Type of document Review Plain language summary
  • Title Individualized funding interventions to improve health and social care outcomes for people with a disability
  • Library Image Library Image
  • See the full review https://onlinelibrary.wiley.com/doi/10.4073/csr.2019.3
  • Records available in English, Spanish
  • English

    PLAIN LANGUAGE SUMMARY

    Individualised funding has positive effects on health and social care outcomes

    Individualised funding provides personal budgets for people with disabilities, to increase independence and quality of life. The approach has consistently positive effects on overall satisfaction, with some evidence also of improvements in quality of life and sense of security. There may also be fewer adverse effects. Despite implementation challenges, recipients generally prefer this intervention to traditional supports.

    What is this review about?

    Individualised funding is an umbrella term for disability supports funded on an individual basis. It aims to facilitate self-direction, empowerment, independence and self-determination. This review examines the effects and experiences of individualised funding.

    What is the aim of this review?

    This Campbell systematic review examines the effects of individualised funding on a range of health and social care outcomes. It also presents evidence on the experiences of people with a disability, their paid and unpaid supports and implementation successes and challenges from the perspective of both funding and support organisations.

    What are the main findings of this review?

    This study is a review of 73 studies of individualised funding for people with disabilities. These include four quantitative studies, 66 qualitative and three based on a mixed-methods design. The data refer to a 24-year period from 1992 to 2016, with data for 14,000 people. Studies were carried out in Europe, the US, Canada and Australia.

    What studies are included?

    Overall, the evidence suggests positive effects of individualised funding with respect to quality of life, client satisfaction and safety. There may also be fewer adverse effects. There is less evidence of impact for physical functioning, unmet need and cost effectiveness. The review finds no differences between approaches for the Adult Social Care Outcomes Toolkit (ASCOT), self-perceived health and community participation.

    Recipients particularly value: flexibility, improved self-image and self-belief; more value for money; community integration; freedom to choose ‘who supports you; ‘social opportunities’; and needs-led support. Many people chose individualised funding due to previous negative experiences of traditional, segregated, group-orientated supports.

    Successful implementation is supported by strong, trusting and collaborative relationships in their support network with both paid and unpaid individuals. This facilitates processes such as information sourcing, staff recruitment, network building and support with administrative and management tasks. These relationships are strengthened by financial recognition for family and friends, appropriate rates of pay, a shift in power from agencies to the individual or avoidance of paternalistic behaviour.

    Challenges include long delays in accessing and receiving funds, which are compounded by overly complex and bureaucratic processes. There can be a general lack of clarity (e.g. allowable budget use) and inconsistent approaches to delivery as well as unmet information needs. Hidden costs or administrative charges can be a source of considerable concern and stress.

    Staff mention involvement of local support organisations, availability of a support network for the person with a disability and timely relevant training as factors supporting implementation. Staff also highlight logistical challenges in support needs in an individualised way including, for example, responding to individual expectations, and socio-demographic differences.

    What do the findings of this review mean?

    This review provides an up-to-date and in-depth synthesis of the available evidence over 25 years. It shows that there are benefits of the individualised funding model. This finding suggests that practitioners and funders should consider moving away from scepticism, towards opportunity and enthusiasm. Policy makers need to be aware of the set-up and transitionary costs involved. Investment in education and training will facilitate deeper understanding of individualised funding and the mechanisms for successful implementation.

    Future studies should incorporate longer follow-ups at multiple points over a longer period. The authors of the review encourage mixed-methods approaches in further systematic reviews in the field of health and social care, to provide a more holistic assessment of the effectiveness and impact of complex ‘real-world’ interventions.

    How up-to-date is this review?

    The review authors searched for studies up to the end of 2016. This Campbell systematic review was published in January 2019.

  • Spanish

    RESUMEN EN LENGUAJE SENCILLO

    El financiamiento individualizado tiene efectos positivos en los resultados de salud y asistencia social

    El financiamiento individualizado provee un presupuesto personal a las personas con discapacidades para aumentar su independencia y calidad de vida. Este enfoque tiene efectos positivos en la satisfacción general, y parece mejorar la calidad de vida y la sensación de seguridad. Puede haber incluso menos efectos adversos. A pesar de los desafíos de implementación, los receptores generalmente prefieren este tipo de intervención sobre los apoyos tradicionales.

    ¿Qué estudió la revisión?

    El financiamiento individualizado es un término genérico para los métodos de apoyo a la discapacidad que son financiados de forma individual. Su objetivo es facilitar la autodirección, el empoderamiento, la independencia y la autodeterminación. Esta revisión examina los efectos y experiencias del financiamiento individualizado.

    ¿Cuál es el objetivo de esta revisión?

    Esta revisión sistemática Campbell examina los efectos del financiamiento individualizado en la salud y los resultados de asistencia social. También presenta evidencia con base en las experiencias de personas con discapacidades, los mecanismos de apoyo remunerados y no remunerados, y los aprendizajes y desafíos de la implementación desde la perspectiva de las organizaciones, tanto de financiamiento como de apoyo.

    ¿Qué estudios se incluyen?

    Este estudio es una revisión de 73 investigaciones de financiamiento individualizado para personas con discapacidades. Esto incluye, cuatro estudios cuantitativos, 66 cualitativos y tres basados en un diseño de métodos combinados. La información engloba un período de 24 años, desde 1992 a 2016, con información de 14,000 personas. Los estudios se llevaron a cabo en Europa, Estados Unidos, Canadá y Australia.

    ¿Cuáles son los principales hallazgos de esta revisión?

    En general, la evidencia sugiere que existen efectos positivos del financiamiento individualizado con respecto a la calidad de vida, la satisfacción del cliente y la seguridad. Puede haber también efectos adversos menores.

    Existe menos evidencia acerca del impacto por funcionamiento físico, las necesidades insatisfechas y la eficiencia de costos. La revisión no encuentra diferencias entre los enfoques por Adult Social Care Outcomes Toolkit (ASCOT), la autopercepción de la salud y la participación de la comunidad.

    Los receptores valoran particularmente: la flexibilidad, la mejora en su percepción de sí mismos y su autoconfianza; una mayor rentabilidad de su dinero; integración en la comunidad; la libertad para elegir ‘quién te apoya’; ‘oportunidades sociales’; y el apoyo orientado hacia las necesidades.

    Muchas personas escogieron el financiamiento individualizado debido a experiencias negativas anteriores con métodos de apoyo tradicionales, segregados y orientados a grupos.

    La implementación exitosa está respaldada por relaciones sólidas, confiables y colaborativas dentro sus redes de apoyo, tanto con individuos que reciben remuneración, como aquellos que no. Esto facilita procesos tales como la búsqueda de información, la contratación de personal, la generación de redes y el apoyo con tareas administrativas y de gestión. Estas relaciones se fortalecen con el reconocimiento financiero para familias y amigos, las tarifas de pago apropiadas, la transferencia del poder desde las agencias hacia los individuos o al evitar comportamientos paternalistas.

    Los desafíos incluyen largas esperas para acceder o recibir los fondos, lo cual se suma a un proceso muy complejo y burocrático. Existe poca claridad en general (ej.: usos adecuados del presupuesto) y métodos inconsistentes para la entrega, al igual que necesidades de información insatisfechas. Los costos adicionales o cargos administrativos pueden ser una fuente considerable de estrés y preocupación.

    Entre los factores que respaldan la implementación, el personal señala la participación de organizaciones locales, la disponibilidad de una red de apoyo para la persona con discapacidad, y la formación relevante y oportuna. El equipo también destaca los desafíos logísticos en las necesidades de apoyo de una manera individualizada que incluye, por ejemplo, responder a las expectativas individuales y las diferencias sociodemográficas.

    ¿Qué significan los resultados de esta revisión?

    Esta revisión proporciona una síntesis actualizada y a profundidad de la evidencia disponible a lo largo de 25 años. Esto demuestra que existen beneficios del modelo de financiamiento individualizado.

    Este hallazgo sugiere que los participantes y patrocinadores deben considerar alejarse del escepticismo y virar hacia las oportunidades y el entusiasmo. Los responsables de la formulación de políticas deben tener presentes los costos de instalación y transición involucrados. La inversión en educación y entrenamiento facilitará un entendimiento más profundo del financiamiento individualizado y de los mecanismos para su implementación exitosa.

    Los estudios futuros deberían incorporar seguimientos actualizados en múltiples áreas durante un período más extensivo. Los autores de la revisión promueven el desarrollo de un enfoque de método mixto en las próximas revisiones sistemáticas en las áreas de la salud y asistencia social, con el fin de generar una evaluación más holística de la efectividad y el impacto de intervenciones complejas en el mundo real.

    ¿Cuán actualizada es esta revisión?

    Los autores de la revisión buscaron estudios hasta finales de 2016. Esta revisión sistemática Campbell fue publicada en enero de 2019.

Additional Info

  • Authors Maren Duvendack, Philip Mader
  • Published date 2019-01-07
  • Coordinating group(s) International Development, Methods
  • Type of document Review Plain language summary
  • Title Impact of financial inclusion in low‐ and middle‐income countries: A systematic review of reviews
  • Library Image Library Image
  • See the full review https://onlinelibrary.wiley.com/doi/10.4073/csr.2019.2
  • Records available in English, Hindi, Spanish
  • English

    PLAIN LANGUAGE SUMMARY

    Financial inclusion interventions have very small and inconsistent impacts

    A wide range of financial inclusion programmes seek to increase poor people’s access to financial services to enhance the welfare of poor and low-income households in low- and middle-income countries. The impacts of financial inclusion interventions are small and variable. Although some services have some positive effects for some people, overall financial inclusion may be no better than comparable alternatives, such as graduation or livelihoods interventions.

    What is this review about?

    Financial inclusion programmes seek to increase access to financial services such as credit, savings, insurance and money transfers and so allow poor and low-income households in low- and middle-income countries to enhance their welfare, grasp opportunities, mitigate shocks, and ultimately escape poverty.  This systematic review of reviews assesses the evidence on economic, social, behavioural and gender-related outcomes from financial inclusion.

    What is the aim of this review?

    This systematic review of reviews systematically collects and appraises all of the existing meta-studies – that is systematic reviews and meta-analyses – of the impact of financial inclusion. The authors first analyse the strength of the methods used in those meta-studies, then synthesise the findings from those that are of a sufficient quality, and finally, report the implications for policy, programming, practice and further research arising from the evidence. Eleven studies are included in the analysis.

    What are the main findings of this review?

    What studies are included?

    This review includes studies that synthesise the findings of other studies (meta-studies) regarding the impacts of a range of financial inclusion interventions on economic, social, gender and behavioural outcomes. A total of 32 such meta-studies were identified, of which 11 were of sufficient methodological quality to be included in the final analysis. The review examined meta-studies from 2010 onwards that spanned the globe in terms of geographical coverage.

    Impacts are more likely to be positive than negative, but the effects vary, are often mixed, and appear not to be transformative in scope or scale, as they largely occur in the early stages of the causal chain of effects. Overall, the effects of financial services on core economic poverty indicators such as incomes, assets or spending, and on health status and other social outcomes, are small and inconsistent. Moreover, there is no evidence for meaningful behaviour-change outcomes leading to further positive effects.

    The effects of financial services on women’s empowerment appear to be generally positive, but they depend upon programme features which are often only peripheral or unrelated to the financial service itself (such as education about rights), cultural and geographical context, and what aspects of empowerment are considered.

    Accessing savings opportunities appears to have small but much more consistently positive effects for poor people, and bears fewer downside risks for clients than credit. A large number of the meta-studies included in the final analysis voiced concerns about the low quality of the primary evidence base that formed the basis of their syntheses. This raises concerns about the reliability of the overall findings of meta-studies.

    What do the findings of this review mean?

    This systematic review of reviews draws on the largest-ever evidence base on financial inclusion impacts.  The weak effects found warn against unrealistic hype for financial inclusion, as previously happened for microcredit. There are substantial evidence gaps, notably studies of sufficient duration to measure higher-level impacts which take time to materialize, and for specific outcomes such as debt levels or indebtedness patterns and the link to macroeconomic development.

    This study is the first review of reviews published by the Campbell Collaboration. Some important limitations were encountered working at this level of systematisation.  It is recommended that authors of primary studies and meta-studies engage more critically with study quality and ensure better, more detailed reporting of their concepts, data and methods. More methods guidance and clearer reporting standards for the social science and international development context would be helpful.

    How up-to-date is this review?

    The review authors searched for studies in November 2017, updating elements of the searches in January 2018. This Campbell Systematic Review was published in January 2019.

  • Spanish

    RESUMEN EN LENGUAJE SENCILLO

    Las intervenciones de inclusión financiera tienen impactos menores e inconsistentes

    Una amplia gama de programas de inclusión financiera busca aumentar el acceso de las personas en situación de pobreza, en los países de ingresos medios y bajos, a los servicios financieros para mejorar su bienestar. Los impactos de las intervenciones de inclusión financiera son menores y variables. Aunque algunos servicios demuestran ciertos efectos positivos para algunas personas, puede que la inclusión financiera en general no sea mejor que otras alternativas similares, tales como las intervenciones de graduación o de medios de subsistencia.

    ¿De qué trata esta revisión?

    Los programas de inclusión financiera buscan aumentar el acceso a servicios financieros tales como el crédito, ahorros, seguros y transferencias monetarias. De esta forma, permiten que los hogares pobres y de bajos ingresos en países de ingresos medios y bajos mejoren su bienestar, aprovechen las oportunidades, mitiguen los impactos y, en definitiva, eviten la pobreza. Esta revisión sistemática de revisiones evalúa la evidencia de los resultados económicos, sociales, de género y del comportamiento de la inclusión financiera.

    ¿Cuál es el objetivo de esta revisión?

    Esta revisión sistemática de revisiones recopila y evalúa sistemáticamente todos los meta estudios existentes (es decir, revisiones sistemáticas y metaanálisis) acerca del impacto de la inclusión financiera. Los autores analizaron la solidez de los métodos utilizados en esos meta estudios, luego sintetizaron los hallazgos de aquellos que poseían una calidad suficiente y, finalmente, informaron sus implicancias para las políticas, la programación, la práctica y la investigación adicional derivada de la evidencia. El análisis incluye 11 estudios.

    ¿Qué estudios se incluyen?

    Esta revisión incluye estudios que sintetizan las conclusiones de otros estudios (meta estudios) acerca de los impactos que han tenido una gama de intervenciones de inclusión financiera en los resultados económicos, sociales, de género y del comportamiento. Se identificaron un total de 32 de estos meta estudios, de los cuales 11 demostraron tener una calidad metodológica aceptable que les permitiera ser incluidos en el análisis final. La revisión examinó meta estudios realizados en todo el mundo a partir de 2010, en términos de cobertura geográfica.

    ¿Cuáles son los principales hallazgos de esta revisión?

    Es más probable que los impactos sean positivos que negativos, sin embargo, los efectos varían. En algunos estudios, los resultados son mixtos y no parecen generar transformaciones en términos de alcance o escala, debido a que ocurren en gran medida durante las etapas iniciales de la cadena causal. En general, los efectos de los servicios financieros sobre indicadores de pobreza económica básicos tales como ingresos, patrimonio o gastos, y sobre la situación sanitaria y otros resultados sociales, son menores e inconsistentes. Además, no existe evidencia sobre cambios en el comportamiento significativos que conduzcan a mayores efectos positivos.

    Los efectos de los servicios financieros en cuanto al empoderamiento femenino parecen ser generalmente positivos, aunque dependen de características programáticas que a menudo son únicamente periféricas o no están relacionadas directamente con el servicio financiero (como la educación acerca de sus derechos), el contexto cultural y geográfico, ni con los aspectos del empoderamiento que son pertinentes.

    El acceso a las oportunidades de ahorro parece tener efectos positivos menores, pero mucho más consistentes para las personas en situación de pobreza, y conlleva menos riesgos de pérdidas para los clientes que el crédito. Un gran número de los meta estudios incluidos en el análisis final expresaron preocupación acerca de la baja calidad de la base empírica primaria que fundamentó sus síntesis. Esto plantea inquietudes respecto de la fiabilidad de las conclusiones globales de los meta estudios.

    ¿Qué significan los resultados de esta revisión?

    Esta revisión sistemática de revisiones se realizó siguiendo la base empírica más grande jamás recopilada relacionada con los impactos de la inclusión financiera. Su débil impacto advierte acerca de las expectativas exageradas y poco realistas de la inclusión financiera, tal y como sucedió anteriormente con el microcrédito. Existen brechas de evidencia considerables, en particular una falta de estudios que hayan durado lo suficiente como para medir impactos de mayor nivel y que requieren tiempo para materializarse, al igual que resultados específicos, tales como los niveles de deuda o los patrones de endeudamiento, y cómo se relacionan con el desarrollo macroeconómico.

    Este estudio es la primera revisión de revisiones publicada por la Colaboración Campbell. Se encontraron algunas limitaciones importantes al trabajar a este nivel de sistematización. Se recomienda que los autores de los estudios primarios y los meta estudios sean más críticos en cuanto a la calidad de los estudios, y que busquen informar mejor y más detalladamente acerca de sus conceptos, datos y métodos. Sería útil contar con más métodos de orientación y normas de presentación de información claras para el ámbito de las ciencias sociales y el desarrollo internacional.

    ¿Cuán actualizada es esta revisión?

    Los revisores buscaron estudios en noviembre de 2017, actualizando los elementos de las búsquedas en enero de 2018. Esta revisión sistemática Campbell se publicó en enero de 2019.

  • Hindi

    Click on 'Download PDF' on the right to view the plain language summary in Hindi.

Additional Info

  • Authors Heather Hensman Kettrey, Robert A. Marx, Emily E. Tanner-Smith
  • Published date 2019-01-04
  • Coordinating group(s) Crime and Justice
  • Type of document Review Plain language summary
  • Title Effects of bystander programs on the prevention of sexual assault among adolescents and college students
  • Library Image Library Image
  • See the full review https://onlinelibrary.wiley.com/doi/10.4073/csr.2019.1
  • Records available in English, Spanish
  • English

    PLAIN LANGUAGE SUMMARY

    Bystander programs increase bystander intervention but no effect on perpetrating sexual assault

    Bystander sexual assault prevention programs have beneficial effects on bystander intervention but there is no evidence of effects on sexual assault perpetration. Effects on knowledge and attitudes are inconsistent across outcomes.

    What is this review about?

    Sexual assault is a significant problem among adolescents and college students across the world. One promising strategy for preventing these assaults is the implementation of bystander sexual assault prevention programs, which encourage young people to intervene when witnessing incidents or warning signs of sexual assault. This review examines the effects bystander programs have on knowledge and attitudes concerning sexual assault and bystander behavior, bystander intervention when witnessing sexual assault or its warning signs, and participants’ rates of perpetration of sexual assault.

    What is the aim of this review?

    This Campbell systematic review examines the effects of bystander programs on knowledge and attitudes concerning sexual assault and bystander intervention, bystander intervention when witnessing sexual assault or its warning signs, and the perpetration of sexual assault. The review summarizes evidence from 27 high‐quality studies, including 21 randomized controlled trials.

    What studies are included?

    This review includes studies that evaluate the effects of bystander programs for young people on (a) knowledge and attitudes concerning sexual assault and bystander intervention, (b) bystander intervention behavior when witnessing sexual assault or its warning signs, and (c) perpetration of sexual assault. Twenty‐seven studies met the inclusion criteria. These included studies span the period from 1997 to 2017 and were primarily conducted in the USA (one study was conducted in Canada and one in India). Twenty‐one studies were randomized controlled trials and six were high quality quasi‐experimental studies.

    What are the main findings of this review?

    Do bystander programs have an effect on knowledge/attitudes, on bystander intervention, or on sexual assault perpetration?

    Bystander programs have an effect on knowledge and attitudes for some outcomes. The most pronounced beneficial effects are on rape myth acceptance and bystander efficacy outcomes. There are also delayed effects (i.e., 1 to 4 months after the intervention) on taking responsibility for intervening/acting, knowing strategies for intervening, and intentions to intervene outcomes. There is little or no evidence of effects on gender attitudes, victim empathy, date rape attitudes, and on noticing sexual assault outcomes.

    Bystander programs have a beneficial effect on bystander intervention. There is no evidence that bystander programs have an effect on participants’ rates of sexual assault perpetration.

    What do the findings of this review mean?

    The United States 2013 Campus Sexual Violence Elimination (SaVE) Act requires postsecondary educational institutions participating in Title IX financial aid programs to provide incoming college students with sexual violence prevention programming that includes a component on bystander intervention.

    Bystander programs have a significant effect on bystander intervention. But there is no evidence that these programs have an effect on rates of sexual assault perpetration. This suggests that bystander programs may be appropriate for targeting the behavior of potential bystanders but may not be appropriate for targeting the behavior of potential perpetrators.

    Beneficial effects of bystander programs on bystander intervention were diminished by 6 months post‐intervention. Thus, booster sessions may be needed to yield any sustained effects.

    There are still important questions worth further exploration. Namely, more research is needed to investigate the underlying causal mechanisms of program effects on bystander behavior (e.g., to model relationships between specific knowledge/attitude effects and bystander intervention effects), and to identify the most effective types of bystander programs (e.g., using randomized controlled trials to compare the effects of two alternate program models). Additionally, more research is needed in contexts outside of the USA so that researchers can better understand the role of bystander programs across the world.

    How up‐to‐date is this review?

    The review authors searched for studies up to June 2017.

  • Spanish

    RESUMEN EN LENGUAJE SENCILLO

    Los programas contra la omisión de socorro para transeúntes aumentan su intervención, pero no afectan la incidencia de las agresiones sexuales

    Los programas de prevención de agresiones sexuales para crear conciencia en los transeúntes acerca de la omisión de socorro tienen efectos positivos en cuanto a la intervención de los mismos, aunque no existe evidencia que muestre impacto alguno en la incidencia sobre las agresiones sexuales. Los efectos en el conocimiento y las actitudes son contradictorias en todos los resultados.

    ¿Qué estudió la revisión?

    Las agresiones sexuales son un gran problema entre adolescentes y estudiantes universitarios en todo el mundo. Una estrategia prometedora para prevenir estas agresiones es la implementación de programas de prevención de agresiones sexuales para educar a transeúntes, que fomenten a los jóvenes a intervenir cuando sean testigos de este tipo de incidentes o adviertan señales de que existe algún tipo de agresión sexual. Esta revisión examina los efectos que los programas contra la omisión de socorro para transeúntes tienen en cuanto al conocimiento y las actitudes relacionadas con las agresiones sexuales, el comportamiento de los transeúntes, su intervención al momento de presenciar una agresión sexual o sus señales de advertencia, y las tasas de participación en la comisión de agresiones sexuales.

    ¿Cuál es el objetivo de esta revisión?

    Esta revisión sistemática Campbell examina los efectos de los programas de transeúntes en cuanto al conocimiento y las actitudes relacionadas con las agresiones sexuales y la intervención de transeúntes, su intervención cuando presencian una agresión sexual o sus señales de advertencia, y la comisión de agresiones sexuales. Esta revisión resume evidencia de 27 estudios de alta calidad, incluyendo 21 ensayos de control aleatorizados.

    ¿Qué estudios se incluyen?

    Esta revisión incluye estudios que evalúan los efectos de los programas de transeúntes para jóvenes en cuanto a (1) su conocimiento y las actitudes relacionadas con las agresiones sexuales, y la capacidad de intervención del transeúnte, (2) el ímpetu de los transeúntes a intervenir cuando son testigos de una agresión sexual o advierten señales de ella, y (3) la incidencia de agresiones sexuales.

    Veintisiete estudios cumplieron los criterios de inclusión. Estos estudios abarcaron el período entre 1997 y 2017 y fueron llevados a cabo principalmente en Los Estados Unidos (un estudio fue hecho en Canadá y otro en la India). Veintiún estudios fueron ensayos controlados aleatorizados y seis fueron estudios cuasi experimentales de alta calidad.

    ¿Los programas de transeúntes tienen un efecto en el conocimiento/actitudes en la intervención de peatones, o en la perpetración de un asalto sexual?

    Los programas para concientizar a los transeúntes muestran efectos en el conocimiento y las actitudes en algunos resultados.

    Los efectos más beneficiosos son que se acepte la historia contada por la víctima de la violación y los resultados en cuanto a la eficiencia de la intervención del transeúnte. También hay consecuencias diferidas (ej.: de uno a cuatro meses después de la intervención) en cuanto a asumir la responsabilidad por mediar/actuar, estar en conocimiento de estrategias de intervención y los resultados de la intención de intervenir. Existe poca o insuficiente evidencia acerca de los efectos sobre las actitudes de género, la empatía con la víctima, las actitudes de violación en citas, y en notar los efectos de las agresiones sexuales.

    Los programas de concientización de transeúntes tienen un efecto positivo en su intervención. No hay evidencia que demuestre que los programas de transeúntes afecten los índices de participación en la comisión de agresiones sexuales.

    ¿Qué significan los resultados de esta revisión?

    La ley de Eliminación de Violencia Sexual en los Campus de Estados Unidos 2013 (SaVE) procura que las instituciones educativas postsecundarias participen en el Título IX de los programas de asistencia financiera, entregando a los nuevos alumnos universitarios programas de prevención de violencia sexual, la cual incluye un componente sobre la intervención de los transeúntes.

    Los programas de concientización de transeúntes tienen un efecto significativo en su intervención. No obstante, no hay evidencia que estos programas afecten las tasas de incidencia de agresiones sexuales. Esto sugiere que los programas de transeúntes pueden ser apropiados en cuanto al comportamiento de posibles transeúntes que observen un hecho de agresión, aunque no tanto en cuanto al comportamiento de los potenciales autores del crimen.

    Los efectos positivos de los programas de transeúntes en cuanto a la intervención peatonal disminuyeron seis meses después de la intervención. De este modo, sería necesario contemplar sesiones de refuerzo para mantener los efectos de forma sostenida.

    Aún quedan preguntas importantes que requieren una exploración más profunda. Es decir, es necesario hacer más estudios que investiguen los mecanismos causales subyacentes de los efectos del programa en el comportamiento de los transeúntes (ej.: para modelar las relaciones entre los efectos de conocimiento/actitud específica y los efectos de la intervención de los peatones), e identificar los tipos de programas de transeúntes más efectivos (ej.: usar ensayos controlados aleatorizados para comparar los efectos de dos modelos de programas alternativos). Además, se necesita mayor investigación en contextos fuera de Estados Unidos para que los investigadores puedan comprender mejor el rol de los programas de transeúntes en todo el mundo.

    ¿Cuán actualizada es esta revisión?

    Los autores de la revisión buscaron estudios hasta junio de 2017. Esta revisión sistemática Campbell fue enviada en octubre de 2017, revisada en octubre de 2018 y publicada en enero de 2019.

Additional Info

  • Authors Angela Higginson, Kathryn Benier, Yulia Shenderovich, Laura Bedford, Lorraine Mazerolle, Joseph Murray
  • Published date 2018-11-29
  • Coordinating group(s) Crime and Justice, International Development
  • Type of document Review Plain language summary
  • Title Factors associated with youth gang membership in low- and middle-income countries
  • Library Image Library Image
  • See the full review https://onlinelibrary.wiley.com/doi/10.4073/csr.2018.11
  • Records available in English, Spanish
  • English

    PLAIN LANGUAGE SUMMARY

    Evidence shows which factors predict gang membership in low- and middle-income countries, but more studies needed

    Youth gang membership is associated with delinquency, violent crime and trafficking. A range of individual, peer, family, school and community factors can predict the likelihood of youths getting involved with gangs. Knowledge of these factors can be helpful for reducing gang membership.

    What did the review study?

    Youth gang membership is associated with delinquency, violent crime and trafficking – and gang members are themselves frequently the victims of these offences. Yet youth gangs can also provide a form of social capital, a sense of belonging and purpose to disenfranchised youth.

    This review identifies the factors associated with young people joining gangs, and the differences between gang-involved and non- gang-involved youth. Understanding these associations is essential to reduce the levels of gang membership and the incidence of related violence.

    What is the aim of this review?

    This Campbell systematic review examines the predictors of youth gang membership in low- and middle-income countries. The review summarises findings from eight reports from five countries and the Caribbean region.

    What studies are included?

    Studies of youth gangs in in low- and middle-income countries were included, with participants aged 10-29 years. The studies had to assess an individual predictor or correlate of youth gang membership, where the predictor or correlate is a single characteristic, not a conglomeration of multiple constructs. Included studies had designs including data on both gang- involved and non-gang-involved youth, recruited with strategies that were eligible.

    Nine studies met the eligibility criteria and were included in the review. One of these studies did not report all the required data and so was not included in the analyses. The studies were conducted in Turkey, Trinidad and Tobago, the Caribbean, El Salvador, China and Brazil.

    What are the main results in this review?

    The eight studies analysed in the review address the associations between life events and circumstances, and the likelihood of being a youth gang member across five domains: individual, peers, family, school and community. Significant associations were found with factors in each domain.

    What do the findings of this review mean? The lack of available evidence limits the extent to which clear conclusions can be drawn about the factors associated with youth gang membership. The review is based on a very small number of studies, and has significant limitations in coverage. The limited evidence of the correlates of youth gang membership suggests factors that may drive gang membership and suggests areas where interventions may prove promising in the family, school, and community domains, as well as provide a starting point for future studies.

    How up-to-date is this review?

    The review authors searched for studies published up to September 2013.

  • Spanish

    RESUMEN EN LENGUAJE SENCILLO

    La evidencia resalta los factores que explican la pertenencia a pandillas en los países de ingresos medios y bajos, pero se requieren más estudios

    La pertenencia a pandillas juveniles está asociada con la delincuencia, los delitos violentos y el tráfico ilícito. Una variedad de factores individuales, de pares, familiares, escolares y comunitarios pueden predecir la probabilidad de que los jóvenes se involucren con pandillas. Conocer estos factores puede resultar útil para disminuir el ingreso de jóvenes a pandillas.

    ¿Qué estudió la revisión?

    La pertenencia a pandillas juveniles está asociada con la delincuencia, los delitos violentos y el tráfico ilícito, y los pandilleros mismos son a menudo víctimas de estos delitos. No obstante, las pandillas juveniles también pueden proveer capital social, sentido de pertenencia y un propósito para los jóvenes marginados. Esta revisión identifica los factores asociados con la afiliación de jóvenes a pandillas y las diferencias entre los jóvenes afiliados a pandillas y los no afiliados. Comprender estas relaciones es esencial para reducir los niveles de pertenencia a pandillas y la incidencia de la violencia que se relaciona con ellas.

    ¿Cuál es el objetivo de esta revisión?

    Esta revisión sistemática Campbell examina los factores predictivos de la pertenencia a pandillas juveniles en países de ingresos bajos y medios. La revisión resume las conclusiones de ocho informes de cinco países y la región del caribe.

    ¿Qué estudios se incluyen?

    Se incluyeron estudios de pandillas juveniles en países de ingresos medios y bajos, con participantes de 10 a 29 años. Los estudios debían analizar un factor predictivo o correlacionado a nivel individual con la pertenencia a pandillas juveniles, donde el factor predictivo o correlacionado fuera una característica individual y no grupos de múltiples constructos. Los estudios incluidos tenían diseños que comprendían datos acerca de jóvenes afiliados y no afiliados a pandillas, reclutados con estrategias elegibles para la inclusión.

    Nueve estudios cumplieron los criterios de elegibilidad y se incluyeron en la revisión. Uno de estos estudios no reportó todos los datos requeridos, por lo que no se incluyó en los análisis. Los estudios se realizaron en Turquía, Trinidad y Tobago, el Caribe, El Salvador, China y Brasil.

    ¿Cuáles son los principales hallazgos de esta revisión?

    Los ocho estudios analizados en la revisión abordan la relación entre las experiencias de vida y las circunstancias, con la probabilidad de pertenecer a una pandilla juvenil a lo largo de cinco esferas sociales: individual, de pares, familiar, escolar y comunitario. Se hallaron relaciones significativas con determinados factores para cada una de ellas.

    ¿Qué significan los resultados de esta revisión?

    La falta de evidencia disponible limita la medida en que se pueden extraer conclusiones claras acerca de los factores asociados con la pertenencia a pandillas juveniles. La revisión se basa en un número muy pequeño de estudios y tiene limitaciones de cobertura importantes. La evidencia sugiere que existen factores que pueden impulsar la pertenencia a pandillas, además resalta áreas donde las intervenciones pueden resultar prometedoras dentro de cada uno de los dominios: familiar, escolar y comunitario; por lo que proporciona un punto de partida para estudios futuros.

    ¿Cuán actualizada es esta revisión?

    Los revisores buscaron estudios publicados hasta septiembre de 2013. Esta revisión sistemática Campbell se publicó en diciembre de 2018.

Page 2 of 18

Contact us