This week’s blog was co-written by the Development Impact Evaluation (DIME) team at the World Bank. Here they share their experience at a workshop in Lagos sharing how Impact Evaluation (IE) can be used to assess how intervention projects are able to affect development outcomes. IE has become an important tool in evidence based policy-making, enabling development agencies and institutions to accountably evaluate development programme outcomes and assess their impact on people’s lives.
“How can we achieve better health outcomes in Nigeria?”
The simplicity of the question belies the complexity of the reality. From May 2-5 2017, the World Bank’s DIME team hosted a workshop in Lagos to present new ways of thinking and practice to better address this question.
The focus of the workshop was on impact evaluations (IEs). This approach uses research methods which allow evaluators to separate the causal impact of a specific project or intervention from changes in outcomes due to other factors. IEs are a powerful tool not only to measure the true impact of policies, but also to understand how and why they work and, perhaps even more importantly, how to make them work better. When done well and in partnership with policymakers and program implementers, IEs become a tool to generate impact.
In a 1st phase, for several years, DIME has been working with the Government of Nigeria and the Bill & Melinda Gates Foundation to conduct IEs, in areas such as maternal and child health, primary healthcare, and malaria and HIV/AIDS control (more details here). This is research with real-world implications – it has the potential to save and improve tens of thousands of lives. For example, through IEs testing different policy design alternatives we have learned that:
- expanding the availability of human resources by deploying midwives to underserved areas and providing effective incentives to retain them could save 5,500 lives per year, reducing maternal mortality in Nigeria by 17% (baseline: 33,000);
- scaling up a program that diagnoses health facility weaknesses, making a plan for improvement, and providing regular coaching nationwide could save at least 3,000 lives per year just through increased hand washing to prevent infections – not even counting other channels of clinical improvement;
- interventions that pay for themselves do exist: the 3rd season of the MTV series Shuga needed to reach only 2% of its 500 million viewers to fully recover the costs of production, in terms of better health outcomes for viewers, because of healthy behaviours they adopted as a result of watching the show.
While we continue to learn from these and other completed health IEs in Nigeria, there is now the opportunity to build on this by incorporating IE into the next generation of health policy. The Lagos workshop moved us towards this by bringing together policymakers and program implementers to develop initial IE concepts to address current priority questions. Nine program teams, strategically selected for their potential to improve health outcomes for vulnerable populations, participated in the four-day workshop and developed preliminary IE concepts. These will be refined and further developed in the coming weeks and months.
In addition to working with each team to develop a tailored IE concept to improve the delivery of their program, the workshop presented and discussed recent evidence relevant to current health policy and operational priorities and featured several lively discussions on the role of evidence, data, and non-traditional interventions (such as Nollywood productions) in achieving better health for Nigerians. Core themes which emerged from these discussions include:
- Through testing different policy design and delivery alternatives, IEs help us identify the most effective alternatives and can therefore be used to allocate scarce resources towards their most effective use.
- Investing in quality and quantity of data is crucial to building evidence to inform policy at the federal, state, and local levels. Monitoring systems should not be an afterthought, but need to be seen as core to program and policy implementation.
- Moving policies and programs from the design to the implementation phase requires careful planning and organization. It is often useful to do a pilot as this can inform adjustments to the design (if needed) to design prior to scale-up. This can increase efficiency and allocation of resources.
- Edutainment (a combination marriage of education and entertainment) and the use of arts and media can be a powerful tool to promote healthy behavior. This provides a cost-effective means of reaching literally millions of consumers of entertainment. But quality of the narrative is key – viewers must be engaged and entertained, and messaging must be carefully woven into the storyline.
As we continue to work with our partners in government and in the private sector to develop the initial IE concepts, highlighted below, into full operational research and implementation plans, we hope to continue engaging with stakeholders in the Nigerian health sector to discuss the emerging evidence on how to achieve better health outcomes.
We’d also love your feedback – which of these ideas resonate and do you have other ideas that you think need to be tested at scale? Please leave a comment on this blog. We look forward to the conversation.
Initial IE concepts developed by participating teams:
- BHCPF: The Basic Health Care Provision Fund Pilot Project (BHCPF) proposes an impact evaluation to test whether health services provision can be improved in Abia, Niger, and Osun states through different combinations of (i) provision of a free basic minimum package of health services, (ii) “service-ready” primary health care facility investments, and (iii) supportive supervision.
- MSS: The Midwife Services Scheme (MSS) proposes an impact evaluation to test maternal health care outcomes can be increased by different combinations of (i) providing professional advice to midwives, (ii) disseminating quality measures in the community, (iii) efficient payment system to ensure regular delivery of allowances, and (iv) small monetary incentives.
- NSHIP: The Nigeria State Health Investment Project (NSHIP) proposes an impact evaluation to test whether health care services can be improved through performance based financing based on (i) quality and (ii) increased weight on services targeting Fragility, Conflict, and Violence in Northeast Nigeria.
- NPHCDA/Nutrition/Nollywood: The National Primary Healthcare Development Agency (NPHCDA) proposes an impact evaluation to test whether immunization uptake and nutrition intake of children under 5 can be increased through edutainment programs delivered via radio or television, with or without partnerships with community leaders.
- NSCIP: The Nigeria Supply Chain Integration Project (NSCIP) proposes an impact evaluation to test whether drug distribution, quality and stock-out rates can be improved by different improved reporting mechanisms, training, and/or equipment supply and renovation.
- PHN/SOML: The Private Sector Health Alliance of Nigeria (PHN), as manager of the Innovation Fund of the Saving One Million Lives project, proposes an impact evaluation to test whether access and utilization of health services can be increased and maternal and child mortality and morbidity can be reduced through mobile health service delivery in Internally Displaced Camps in Northeast Nigeria.
- Oloibiri: The Oloibiri Health Project in Bayelsa State proposes an impact evaluation to test whether skilled birth attendance and antenatal care utilization can be increased through engaging unskilled birth attendants (including traditional birth attendants) to refer women to health facilities for delivery.
- LASHMA: The Lagos State Health Insurance Scheme Management Agency (LASHMA) proposes an impact evaluation to test whether health insurance enrolment by the informal sector can be increased by targeting individuals, trade groups and unions, community development associations, and/or community leaders.
- NIPEP: The Nigeria Partnership for Education Project (NIPEP) proposes an impact evaluation to test whether enrolment, retention and learning outcomes of girls aged 6-12 can be improved either through edutainment programs delivered through radio and television, or through reading camps with or without aspirational talks for parents.
I love this summarised brief that is direct to the point in the face of current penchant for information overload insufficient available time.
How does an IE proposal get to your organization?
I like the direction of this organisation.
Dear Dr. Oluoha,
many thanks for your comment. Feel free to contact us with details of what you would like to propose and we can then advise on next steps. Here is my email: fdunsch “at” worldbank.org
World Bank IE Coordinator
Thanks for the summary.
I have a question though. To carry out an impact evaluation, do you have to always do the actual project in order to see the outcome or you can use questionnaires in some cases to have an idea of what the outcome can be like.
Dear Mr Oni,
many thanks for your question. We mainly work on measuring impact of actual projects, principally projects run by our government partners.
By evaluating actual policies and projects we make sure that we evaluate the effectiveness of projects as they are implemented in practice and not just hypotheticals. This helps governments and others on how to improve and scale up policies going forward.
This World Bank publication is an excellent resource for all questions concerning IEs:
Thank you so much, I would say that this precise is very apt given the state of our healthcare presently and the future as predicted by these present trends ; which you will agree with me needs aggressive and dedeicated measures to influence positive outcomes. The IE has potentials to do just that.
It appears from the projects listed, that the proposed studies speak to predictions based on implementations , thus my question borders on the role of IEs as performance measurements, does IE not first of all show “how well, so far” before “next steps”?
Dear Healthwatch Admin,
Please is there a way to contact Miss Eze-Ajoku or Mr Dunsch for collaboration in Impact Evaluation research projects? My group at UCSF is presently designing one to target cancer screening programs and I greatly desire total into their technical expertise.