Consultancy - Mid-Term External Evaluation of the Pioneer Project: A Special Initiative Working Towa
Malaria Consortium is an international organisation working primarily in Africa and Asia on communicable disease control. Working closely with Ministries of Health (MoH), academic institutions and a range of partners, Malaria Consortium is involved in all aspects of malaria control from policy and strategy development to implementation, monitoring and evaluation and operational research. Malaria Consortium also works on the control of other communicable diseases, including tuberculosis, diarrhoea and neglected tropical diseases.
Malaria Consortium International¡¦s office is in London, UK and the Africa Regional Office is in Kampala, Uganda. The Malaria Consortium¡¦s Uganda office, also located in Kampala, has been active since 2003 and supports a large and diverse country programme.
Project Overview
The Pioneer project is a four year Special Initiative grant supported by Comic Relief UK and implemented by Malaria Consortium Uganda from May 2009 ¡V April 2013. The project aims to reduce malaria morbidity and mortality through improving systemic malaria control, by increasing both supply and demand for malaria control tools in mid-Western Uganda. The ideal outcome of the project is that families are empowered to take positive action for the health of their own family: they know about the best tools to protect and treat their families, act on this knowledge and are able to access these tools.
The project will be completing its second year of implementation in April 2011. In the first two years of the project, coverage of life-saving interventions that have a dramatic impact on malaria has been increased via campaign distributions of 600,000 LLINs, the introduction of malaria RDTs in low level health facilities, and provision of rectal artesunate as pre-referral treatment for severe malaria at community level. Various behaviour change communication (BCC) activities have also been carried out to increase families¡¦ knowledge about these interventions and encourage them to act on this knowledge. Other areas of work within the project to support its goal are: health systems strengthening and monitoring and evaluation components to support the assessment of impact and document learning to inform the wider scale up of interventions as well as advocacy.
2. Purpose, scope and objectives
Purpose: The purpose of the mid-term evaluation is to inform overall project learning in relation to the project¡¦s Theory of Change and progress against the workplan; to guide the project team and stakeholders in the identification of successes and challenges to date; and to suggest recommendations towards achieving the desired changes and impact by the end of project (as outlined in the project¡¦s Theory of Change and Monitoring and Evaluation plan).
The evaluation is intended to provide a space for all stakeholders to reflect on the work to date and to facilitate a process of critical analysis that allows people to celebrate success, challenge assumptions and ways of working, and explore areas that have been more difficult.
Scope and objectives:
The consultant will conduct an independent assessment of the appropriateness, timeliness, effectiveness and efficiency of the Pioneer project¡¦s implementation to date. The specific objectives and key questions for assessment are:
1. To assess the appropriateness and relevance of the project design and approaches used to achieve the desired change. This should include a review of the project¡¦s Theory of Change and its implementation, including strategies developed, interventions applied and methods designed to test those approaches.
- Were the original objectives valid? Are these achievable and did they meet the needs and priorities of the target groups? If not, has the project been able to adapt them in an appropriate way?
2. To assess the effectiveness of the project in achieving the desired change and intermediate results, including the nature and extent of impact on the target communities and populations and any barriers to the longer-term achievement of the desired change.
- Have project activities generated the planned outputs and have they been delivered on time? If not, what have been the reasons for this? Were the reasons valid?
- What significant achievements has the project brought about thus far? Where has it performed particularly well and where has its performance been weakest? For what reasons have there been fewer achievements, if any, in some areas?
3. To assess the efficiency of the project in achieving the desired change, including project responsiveness to changes in the implementing context.
- How have work plans been adapted during the life of the project? Were these adaptations appropriate? Have the project activities represented good value for money?
4. To determine the level of ownership and satisfaction felt by key stakeholders (including the District Health Teams) and target communities.
- How well has the project identified and engaged key partners and other relevant stakeholders? To what extent have local communities, district health teams and other agents of change been involved and empowered throughout the implementation of the project?
- How has the project supported local organisations, including its project partner, the Malaria and Childhood Illness NGO Secretariat (MACIS)?
- Were the identified target populations (families, children under five, pregnant women, Village Health Team members and district Ministry of Health personnel) the most in need? Have those in greatest need been reached? To what extent have people¡¦s basic needs as well as their basic rights been addressed?
5. To assess the potential sustainability of achieved changes and project plans for ensuring a continuation of benefits generated under the project.
- Does the project embody a long term commitment to the target population?
- How well does the project design and selected approaches support the achievement of lasting changes in the lives of those directly benefiting, as well as those benefiting from changes in policy, legislation and practice?
- How has the project engaged the Ministry of Health and other decision makers and developed relationships for influence towards achieving longer term, systemic change?
6. To assess the value of monitoring and evaluation approaches used and the effectiveness of learning documentation and dissemination.
- How well has the programme learnt from experience and improved practice?
- How well do the project¡¦s ¡§Learning Questions¡¨ contribute to gaining an in-depth understanding of the assumptions underpinning the project¡¦s Theory of Change and specific approaches?
- How effectively have lessons learnt been disseminated to all stakeholders, key target audiences and policymakers?
7. To assess the quality of relationships and added value of the implementing organisation and donors.
- How well have the relationships between the implementing organisation, local organisations and donors (including Comic Relief) worked?
3. Analytical plan for the evaluation
As a first step, the consultant will develop an analytical plan for the evaluation, in coordination with the Malaria Consortium Pioneer project team, and based on the project logic or Theory of Change (ToC). The project¡¦s Theory of Change envisions that by simultaneously ensuring the best available tools are accessible to the people who need them the most and increasing their understanding of the importance and effectiveness of these tools, families will be empowered to take positive action for the health of their own households and malaria related morbidity and mortality will be dramatically reduced within the region.
The analytic plan should provide a structure for reflecting on the Theory of Change, including its strengths and weaknesses and potential implications for implementation. This plan will also serve to guide the final external evaluation at the end of the project, and may be useful for informing future evaluations of other Comic Relief Special Initiative grants.
4. Process and Roles
The independent lead consultant/evaluation team leader will lead the evaluation undertaken, working with the project team as required throughout implementation. The lead consultant will be responsible for assembling an evaluation team, drawing on both external and internal resources. The consultant will be responsible for ensuring that all project stakeholders, including representatives of local partners and Comic Relief grants staff, have ample opportunity to input to the evaluation team and are actively involved in the evaluation process. The lead consultant should also ensure timely write up and dissemination of the evaluation report as agreed, and facilitate subsequent feedback and discussion with both Comic Relief and the Malaria Consortium Pioneer team.
All evaluation activities will be coordinated with and/or supported by Malaria Consortium Uganda¡¦s Pioneer project team. The Pioneer project team will assist with identifying local research assistants or translators as required to complete the evaluation team. Pioneer project staff will ensure the collation and availability of all background documents and resources prior to the evaluation as well as the organization of logistical arrangements (arranging interviews, etc). The Pioneer project team should also be actively engaged in the design of the evaluation plan and as active participants/key informants throughout the in country fieldwork.
Comic Relief UK and the Malaria Consortium Uganda Pioneer project team will both review the detailed evaluation plan, analytical plan and tools, as well as the preliminary findings and final report. Both Comic Relief UK and Malaria Consortium Uganda will sign off on the final report.
5. Methodology
The consultant will be expected to design the evaluation plan, selecting the most appropriate methodologies, to be discussed and agreed with the Pioneer project team. The selected methodologies should be participatory, reflect the project logic (Theory of Change) and be designed to effectively assess the process of change. It is expected that some primary data collection will be conducted using a combination of primarily qualitative methods and tools which may include:
„X Document review: Review of program documents (proposal, M&E framework, activity reports, annual project reports, workplan, budget and financial reports, survey reports, formative KAPB study report, and protocols).
„X Key informant interviews and/or group review meetings with internal informants (Comic Relief grants staff; Pioneer project staff in Kampala and Hoima offices) and external informants (partners, stakeholders and target groups at central, district, health facility, and community level).
„X Focus group discussions with partners, stakeholders and target groups at district, health facility, and community level.
„X Most Significant Change (MSC): It is desired that the Most Significant Change technique be incorporated in the evaluation methodology. This participatory method would be a valuable opportunity for involving project stakeholders in deciding what types of changes are important to be recorded under the project.
„X Ongoing (unstructured) consultation with the Pioneer project team.
6. Expected Output / Structure of the Final Report
The consultant will submit a comprehensive evaluation report to include evaluation methods, findings, conclusions, and recommendations. The report should be clear and simply written, free of jargon. The structure of the final report (max 25-30 pages excluding appendices) should include:
- Executive summary
- Brief background (as directly relevant to the report¡¦s analysis and conclusions)
- Methodology
- Key findings and conclusions (supported with relevant data)
- Lessons and recommendations for (1) the remainder of the Pioneer project and (2) for wider learning. (Recommendations should include details as to how they might be implemented, e.g. ¡§recommended actions¡¨)
- Annexes including list of all individuals interviewed, interview guides, chronogram / work schedule, and any other technical details
7. Timeframe *
Deliverable By date
Proposal to be received 27 May 2011
Selection of consultant and contract signed 3 June 2011
In country field visit, data collection and interviews TBD according to agreement with consultant (8-10 days fieldwork anticipated; should be completed during the period from 13 June ¡V 9 July 2011)
Submission and presentation of top line findings 15 July 2011
Discussion with and feedback from Malaria Consortium Uganda and Comic Relief UK 22 July 2011
Final Report submission 29 July 2011
Dissemination of the report to project stakeholders (responsibility of Malaria Consortium Uganda) via roundtable discussions and presentations with District Health Teams and other partners 12 August 2011
* Following receipt of proposals, the precise timeframe can be agreed upon with the consultant at the time of contract.
8. Consultant Specification
The lead consultant interested in carrying out this assignment should have a strong background in public health-related evaluations, including the following qualifications and skills:
- Previous experience in evaluating health-related projects based on a theory of change, including qualitative participatory evaluation methodologies that involve the engagement of stakeholders and communities
- Specific experience using the Most Significant Change technique
- Prior experience in and/or familiarity with the Ugandan and/or East African context preferable
- Knowledge of malaria control approaches preferable
- Demonstrated exceptional report writing skills
- Good facilitation skills
- A consultative approach to their work
9. Format for Submission of Proposals
Expressions of interest are welcome from both individual consultants and groups of consultants. The following documentation should be submitted:
1. Cover letter
2. CVs of the consultant(s) (team leader and any members of the evaluation team)
3. A sample evaluation report
4. Proposal (~2500 words in length), including suggested overview of methodology, proposed workplan, and budget (with a breakdown of costs for professional fees including salary history justification, international travel, analysis and report writing. In country travel, accommodation and field costs associated with primary data collection will be covered directly by Malaria Consortium Uganda.)
5. Dates of availability for fieldwork
Post a Comment