Review of HIV programming over a ten year period (2002-2011)
Terms of Reference for External Contractor
HIV Global Learning Review, April 2012
Type of evaluation: Review of HIV programming over a ten year period (2002-2011)
Expected evaluation methodologies: Only Qualitative methods are Anticipated--Document review of available documentation (reports, evaluations, etc), key-informant and in-depth interviews; But other methods may be used with justification if desired.
Number of evaluators: 2 evaluators
Expected number of days: 89 work days initial estimate
Budget Estimated: $60,000 to $70,000 with Maximum of $80,000 USD
Deadline for receiving applications: May 13, 2012
1. Description of project/program to be evaluated
1.1. Background
The American Red Cross (ARC) International Services Department (ISD) is undertaking a review of its global HIV programming from 2002 to 2011. This decision comes at a time when ISD is planning for a significant scale-up of its strategic programming areas. An assessment of ARC's $32 million dollar investment in HIV projects over the past decade will inform this scale-up process by reviewing ISD's HIV work to date.
1.2. Scope and reach of HIV project portfolio
ISD has been engaged in the HIV sector internationally since 2002 beginning with a grant from the Anne Ray Charitable Trust (ARCT) to support children affected by HIV in Russia. From this initial grant, the HIV portfolio has grown over the years including one significant program from 2004-2010 called "Together We Can" that was implemented in Guyana, Haiti and Tanzania with the support of the US Agency for International Development. Since then the portfolio has expanded to cover 11 countries.
Through the support of ARCT, ARC's Strategic Investment Funds and a U.S. government grant, ARC has committed over $32 million dollars over 10 years, making HIV one of the two main pillars of the health portfolio for ISD (the other being the Measles Initiative) . ISD HIV programs focus on providing information on risk reduction, comprehensive HIV prevention interventions, care and support services for People living with HIV (PLHIV), promotion of adherence to antiretroviral and tuberculosis treatment, and reducing stigma and/or discrimination at the household and community levels. As of January 2012, ISD is actively supporting projects in: Bahamas, China, Guyana, Haiti, Jamaica, Kazakhstan, Kenya, Russia, Tanzania, Ukraine and Vietnam. Collaborative efforts are also underway in Belarus and Armenia. A brief overview of the HIV portfolio from 2002-2011 is included in Annex 1.
1.3. Project/program management
Sources of project/program funding include external agencies (e.g. US Government Agencies or private sources like the Anne Ray Charitable Trust), ARC funding, and matching resources (sometimes in kind) from the Host National Societies of the countries in which the projects and programs are implemented. Projects and programs are implemented by Host National Societies (NS) in partnership with ARC, whose resources include field-based Delegates, region-specific Program Officers and Directors at National Headquarters (NHQ) in Washington, D.C. and the support of Technical Advisors. The portfolio includes some multi-country programs, and some programs include projects designed and implemented at the National Society branch level. For ease of language in referring to diverse cases, these TORs will refer to all these projects and programs as "ARC projects/programs" but it should be understood that funding may be from diverse sources, all projects are implemented in partnership with the HNSs and their branches, and implementation responsibilities may be assigned at different levels.
2. Overview of Global HIV Learning Review
2.1. Purpose and objectives of review
The purpose of this global learning review is to provide a retrospective overview of ISD's HIV programming from the past ten years, capturing lessons learned from the perspective of both ARC and host NSs. This global learning review of past and present HIV programming builds a foundation for shared reflection and learning with host NSs. Knowledge garnered from this global learning review will help inform ARC's future programmatic development. ISD will utilize information gleaned from this study for the following three objectives:
1. Lessons learned for ARC: Alignment: To assess how ARC's HIV programming aligns with its own broader priorities and with NSs' priorities and capacities concerning health, particularly HIV. The review should address the following questions:
i. How feasible, relevant and sustainable are ARC HIV projects/programs, from NS' perspective? How do the services and activities of these projects/programs compare to those in the health sector implemented by NSs (past and present)?
ii. What types of resource inputs were a part of the ARC HIV projects/ programs and to what degree were NSs able to absorb resource inputs from ARC HIV projects/programs? What features of design and factors in implementation contributed to successful absorption?
iii. Considering the degree to which ARC's broader programmatic objectives were realized through HIV Projects/programs (strengthening NS' capacity and the resilience of local communities), what factors were important in realizing these objectives?
2. Lessons learned for host NSs and ARC: Results and Sustainability: To assess the results from ARC HIV projects/programs in terms of the effects they had on National Societies' capabilities to serve the community going forward.
Based on knowledge from Objective 1, the review should examine the results of ARC HIV projects/programs (past and present) as they impacted NSs.
i. What effects have ARC HIV projects/programs had on NSs organizationally?
ii. What effects have ARC HIV projects/programs had on NS' technical capacity in HIV?
iii. What effects have ARC HIV projects/programs had on NS' project management capabilities?
iv. What effects have ARC HIV projects/programs had on HIV and health services available to communities?
v. To what degree have ARC HIV projects/programs' services and activities crowded out, supported, complemented and/or strengthened NS capacity to deliver and/or expand HIV and health services available to communities?
vi. Ex-post evaluation: For past projects that have ended, which NSs have sustained service provision to communities? What factors contributed to being able to sustain service provision?
3. Models going forward: To provide a foundation from which ISD can consider options for supporting NS' health and HIV services going forward.
Based on knowledge from Objectives 1 and 2, the review should examine appropriate implementation support and partnership models that ARC can use to help HNSs' sustain service provision to current beneficiaries. The focus of this objective pertains to active ARC HIV projects that include the Caribbean HIV/AIDS Project, Prevention, Care and Support Project in Vietnam, Tanzania Integrated HIV project, Haiti PrevSIDA/HHAP and Regional Health Initiative in Eastern Europe and Central Asia.
i. For current on-going projects, how sustainable are the service provision models that NSs are conducting with communities in relation to NSs' capacity, strategic priorities, partnerships, donor climate, etc.?
ii. What challenges do NSs experience in providing HIV-related services to communities?
iii. What opportunities exist to strengthen NS capacity to provide and sustain HIV-related services to local communities?
iv. What are some ways ARC can capitalize on and/or foster opportunities to enhance NS' capacity to provide and sustain HIV-related services to local communities?
2.4. Main audience of Global HIV Learning Review
The main audience for this evaluation is senior ARC management, who envisages forming conclusions and a strategy, which is then expected to be shared with key donors and partners (e.g. IFRC), in order to determine the future of health programming.
2.5. Coverage of evaluation
Through a desk review, this review will look at the entire HIV program portfolio over the past 10 years (2002-2011), including those completed and those still ongoing, using available project documentation and key staff interviews to be determined in collaboration with the M&E team, regional directors, Senior HIV Advisor, Senior Health Coordinator and senior management. The review will also examine (in less detail) the NS program of health in the countries where the HIV interventions take place.
3. Evaluation criteria and questions
Under the previously mentioned criteria the evaluation will address the following questions:
Criteria
Objective 1:
Alignment: To assess how ARC's HIV programming aligns with its own broader priorities and with NSs' priorities and capacities concerning health, particularly HIV.
Objective 1: Main evaluation questions
1. How feasible, relevant and sustainable are ARC HIV projects/programs, from NS' perspective? How do the services and activities of these projects/programs compare to those in the health sector implemented by NSs (past and present)?
2. What types of resource inputs were a part of the ARC HIV projects/ programs and to what degree were NSs able to absorb resource inputs from ARC HIV projects/programs? What features of design and factors in implementation contributed to successful absorption?
3. To the degree to which ARC's broader programmatic objectives were realized (strengthening NSs capacity and the resilience of local communities), what were important factors?
Objective 1: Sub-questions
1. What health and HIV projects/programs are NSs implementing? How are they being implemented and why?
2. How relevant is addressing HIV to NSs strategic priorities?
3. Did the ARC HIV projects/programs contribute (beyond their projects' service delivery objectives) to strengthening NS' capacity and community resilience?
4. If so, what design features and/or factors in implementation were important?
5. How transferable are these factors across NSs?
Objective 2:
Results and Sustainability: To assess the results from ARC HIV projects/programs in terms of the effects they had on National Societies' capabilities to serve the community going forward.
Objective 2: Main Questions
1. What effects have ARC HIV projects/programs had on NSs organizationally?
2. What effects have ARC HIV projects/programs had on NS' technical capacity in HIV?
3. What effects have ARC HIV projects/programs had on NS' project management capabilities?
4. What effects have ARC HIV projects/programs had on HIV and health services available to communities?
5. To what degree have ARC HIV projects/programs' services and activities crowded out, supported, complemented and/or strengthened NS capacity to deliver and/or expand HIV and health services available to communities
6. Ex-post evaluation: For past projects that have ended, which NSs have sustained service provision to communities? What factors contributed to being able to sustain service provision?
Objective 2: Sub-Questions
1. For projects that have closed, have NSs been able to sustain certain project components to communities? How have they been sustained?
2. For current projects, what actions are NSs doing to sustain services after the grant period ends?
3. What has been the contribution of ARC HIV projects/ programs to the ability of NS' to sustain service delivery?
4. What key elements of those projects can be applied to future efforts to support NSs in sustaining services after the grant periods end?
Objective 3:
Models Going Forward:
Based on Objectives 1 and 2, examine appropriate implementation support and partnership models that ARC can use to help HNSs' sustain service provision to current beneficiaries.
Objective 3: Main Questions
1. For current on-going projects, how sustainable are the service provision models that NSs are conducting with communities in relation to NSs' capacity, strategic priorities, partnerships, donor climate, etc.?
2. What challenges do NSs experience in providing HIV-related services to communities?
3. What opportunities exist to strengthen NS capacity to provide and sustain HIV-related services to local communities?
4. What are some ways ARC can capitalize on and/or foster opportunities to enhance NS' capacity to provide and sustain HIV-related services to local communities?
Objective 3: Sub-Questions
1. Would successful models identified in Objective 2 be appropriate in any of ARC's current HIV projects? (in terms of culture, target group, etc)
2. For models that may be appropriate, how can those models be adapted to help NSs sustain service provision to current beneficiaries?
3. What aspects of the successful partnership models identified in Objective 2 can be effective in any of ARC's current HIV projects?
4. For potentially helpful partnerships, how can ARC help foster and/or strengthen those connections?
4. Scope of work and Evaluation design
4.1. Scope of work
The contractor(s) will be responsible for the following steps and deliverables:
i) An inception report, detailing a recommended methodology and work plan to meet the evaluation objectives satisfactorily in the time and budget provided. The inception report will be discussed at headquarters before further work progresses, particularly to discuss the role and timing of selected country visits;
ii) Before country visits, if any, more detailed review of background documents on the projects and the countries in which they are implemented if needed—e.g. project proposals, as updated, quarterly project reviews, evaluations, completion reports, if any;
iii) Preparation of data collection instruments, list of potential institutions to consult, preliminary interviewees list (can be prepared with input from ARC staff), and interview guides; criteria for site visits/inspections, if any;
iv) Executive summary of initial macro-level findings for organizational management meeting in September 2012;
v) A Draft Report, with list and titles of people interviewed. Separate annexes of country profiles summarizing findings from objectives 1-3 to be subject to review by national stakeholders, for each country portfolio.
vi) Workshop to present findings at headquarters in Power Point format;
vii) Revised Final Report which takes account of all comments received from the workshop and from national stakeholders.
4.2. Evaluation methods
i) Desk review of key documents, including strategy documents, prior evaluation reports, monitoring reports and other documents judged relevant.
ii) Assessment of prior baseline or end line survey/assessments, as relevant.
iii) Literature search and review of material on the environment in which the program operates, and recent developments which impact the feasibility, relevancy and sustainability of NSs conducting health programming, particularly in HIV, including evaluations and annual reports of other local agencies implementing HIV projects in the same sector and with the same/similar target population.
iv) Interviews with key project/program staff—both headquarters and field (could be by phone).
v) Interviews and/or other qualitative methods to hear perspectives and views of representatives of the project/program stakeholders (Host National Society, implementing partners, key informants, past evaluators, donors, etc.)
vi) Other participatory approaches, such as round tables with national stakeholders to assess the "most significant change" as results of ARC interventions.
4.3. Discussion of inception report, and subsequent points of consultation with Evaluation Manager
An inception report discussing the evaluation work plan, research questions, design and methodology will be developed by the consultant after 10 days of selective document/literature review and interviews with a few key staff at NHQ. This inception report will then be shared with the M&E team, Senior HIV Advisor, Regional Directors and International Response and Programs (IRP) Senior Director at NHQ for review and further discussion. Development of data collection instruments or elaboration of methodological approaches will commence once the inception report has been finalized and agreed upon by the consultant and the Senior M and E Advisor.
Data collection instruments or qualitative information collection guides should be submitted to the Evaluation manager for approval, allowing 5 days for coordination of review. In addition, consultation with the Evaluation manager and respective Regional Directors is required before final plans are made for any country or site visits. A minimum of 5 days notice per country is required to coordinate consultation and logistic support for the country visits.
4.4. Logistic and Administrative Support
The M&E team, Senior HIV advisor, program officers and field staff will provide logistical support in coordinating interviews and meetings. Local transportation in the field will be provided whenever possible and may be required in some countries for security reasons. Translation needs leading up to production of the final report, either for documents or field data collection, need to be factored into consultancy cost. While ARC staff may assist in references to qualified personnel, costs need to be borne by the consultants. Translation of the final report or other products will be borne by ARC.
4.5. Reporting relationship
The contractor will report to the designated evaluation manager, the Senior M&E Advisor, International Response and Programs (IRP), or delegee, with technical support from the Senior HIV Advisor, IRP. The Evaluation Manager will coordinate at key points with other technical advisors, and POs as needed, but particularly the Senior HIV Advisor and IRP Senior Director.
4.6. International standards & Presentation of evidence
Standard social science research, evaluation and survey methodologies and good practices utilized in the international humanitarian community should be applied. Such resources should include but are not limited to those promulgated by the Active Learning Network for Accountability and Performance and the Organization for Economic Co-operation and Development.
In particular, all findings and conclusions should be based on evidence which is presented in the final report, and any assumptions and/or limitations clearly stated. For sample surveys, if applicable, detailed information should be presented on the sample design (including sample size calculation, stratification, clustering, allocation, selection, departures from equal selection probability and weighting), the respondent selection methodology, non-response rates, and coefficient of variation, design effect and intra-class correlation for all variables. For focus group discussions, information on selection of participants should be presented. For case studies, the criteria and processes for selecting those cases should be presented.
ARC does not anticipate that the methods envisaged will require review by National Ethical Review Boards. This topic will need to be addressed in the inception report.
4.7. Ethical Guidelines
It is expected that the evaluation will adhere to ethical guidelines as outlined in the American Evaluation Association's Guiding Principles for Evaluators. A summary of these guidelines is provided below, and a more detailed description can be found at www.eval.org/Publications/GuidingPrinciplesPrintable.asp.
1. Informed Consent: All participants are expected to provide informed consent following standard and pre-agreed upon consent protocols.
2. Systematic Inquiry: Evaluators conduct systematic, data-based inquiries.
3. Competence: Evaluators provide competent performance to stakeholders; specifically, they do not accept assignments which are beyond their competency, or for which they cannot mobilize the requisite expertise.
4. Integrity/Honesty: Evaluators display honesty and integrity in their own behavior, and attempt to ensure the honesty and integrity of the entire evaluation process.
5. Respect for People: Evaluators respect the security, dignity and self-worth of respondents, program participants, clients, and other evaluation stakeholders. It is expected that the evaluator will obtain the informed consent of participants to ensure that they can decide in a conscious, deliberate way whether they want to participate.
6. Responsibilities for General and Public Welfare: Evaluators articulate and take into account the diversity of general and public interests and values that may be related to the evaluation.
4.8. Future use of data
All collected data will be the sole property of the American Red Cross. The contractor may not use the data for their own research purposes, nor license the data to be used by others, without the written consent of the American Red Cross. Dissemination of the review report and any component reports will be determined by a dissemination plan to be prepared by the Evaluation Manager in coordination with other technical advisors and Sr. Management.
5. Expected activities and Deliverables
5.1. Expected activities and Level of Effort (proponents may propose changes)
Activities Number of days
Consultant Team – expected to be 2 people
1. Desk review, literature search and discussions with key program and management staff, 8 days
2. Develop Inception Report including choice of data collection and analysis methods, 5 days
3. Review, discuss and revise inception report with M&E managers (by phone), 5 days
4. Develop data collection and analysis plan, 5 days
5. Site selection and data collection schedule outline, 2 days
6. Discuss site selection and schedule outline with evaluation manager and other stakeholders with view to agreeing on exact itinerary for data collection, 3 days
7. Prepare Country visits, 2 days
8. Undertake country visits, 39 days
9. Prepare executive summary of initial macro-level findings, 3 days
10. Data analysis, 5 days
11. Prepare and submit draft Report for review, 3 days
12. Present findings to ARC staff in Washington, DC, 5 days
13. Finalize report in line with ARC feedback, 5 days
Total expected work days: 89
5.2 Deliverables
Steps/Deliverables* Expected deadline
1. Inception report* Mid-June
2. Finalized data collection instruments* and training tools End June
3. Site visit plan and discussion with NHQ Mid-July
4. Site visits to current HIV project countries completed Late August
5. Executive summary of initial macro-level findings*
Mid-September
6. Draft report* Early October
7. Presentation to ARC staff in Washington, DC* Mid-October
8. Final report*
End October
6. Obligations of Evaluators and Associates
a. Inform the evaluation manager in a timely fashion of progress made and of any problems encountered.
b. Submit any survey or qualitative data collection instruments to the evaluation manager with five days notice for coordination of review.
c. Plan travel itinerary and evaluation activities prior to departure with the evaluation manager and regional directors with at least 5 days notice per country, to aid in coordination with field staff and NS. Keep country ARC and NS staff informed of in-country planned activities, so they may plan optimal participation, and if requested by them, provide a debrief upon completion; record any comments or reactions.
d. Implement the activities as expected, and if modifications are necessary, bring to the attention of the Evaluation Manager before enacting any changes.
e. Report on a timely basis any possible conflicts of interest or any issues which seem to warrant further investigation beyond their TOR or authority.
f. Be responsible for the deliverables outlined in this evaluation study, and the conduct of any associates hired (including local hires such as translators or enumerators); specifically, the lead evaluator will be responsible for the quality and timeliness of the work provided by members of his/her team.
g. Coordinate and synthesize data findings into cohesive and balanced products, including the final report and any derivative products such as powerpoints for briefing.
6.1. Obligations of the Evaluation Manager (or delegee)
a. Facilitate the work of the contractor(s) in headquarters and in the field: Make sure that the contractor(s) know where they can receive specified human resources and logistical support; answer or refer any day-to-day enquiries and aim to ensure responsiveness.
b. Liaise and coordinate evaluation activities with NHQ program officers, and in-country ARC and NS staff to ensure smooth data collection.
b. Monitor the work of the contractor(s) and flag any concerns.
c. Receive and sign off on deliverables and authorize payment.
Obligations of in-country ARC Staff:
a. Facilitate the work of the contractor(s) with beneficiaries and other local stakeholders; brief them on any local security or cultural issues of concern which they need to take into account in planning their work.
b. Raise any issues that affect overall costs or quality of evaluation with the evaluation manager.
6.2. Obligations of the NHQ Technical Team
a. Review and approve the proposed methodology and inception report, consulting management and others as appropriate.
b. Provide technical oversight in the review of relevant deliverables, as requested by the evaluation manager.
c. Provide timely comments on any progress reports and the draft report.
7. Required qualifications: Lead Evaluator
1. Demonstrated extensive experience in: a) public health, preferably HIV programming; and b) program evaluation
2. Masters in Public Health, and/or equivalent experience in public health, particularly community health.
3. Demonstrated experience in leading evaluations of humanitarian/development projects/programs.
4. Demonstrated experience in qualitative data collection and analysis.
5. Demonstrated experience in training and managing enumerators or collectors of qualitative information, if required.
6. Excellent analytical and critical thinking skills.
7. Demonstrated experience of written work that has informed practice.
8. Professional work experience in country or region preferred.
9. Strong competence working and communicating cross-culturally.
10. Good oral presentation skills.
11. Fluency in English required, in addition to French and/or Russian desirable.
8. Application and selection details
8.1. Application materials
The application should include the following five items. Please note that any application that does not contain all five items will be rejected. All documents should be single-spaced in Times New Roman font size 12.
1. One-page Summary of experience.
2. Detailed CVs of all professionals who will work on the evaluation. If there is more than one consultant on the proposed evaluation team, please attach a table describing the level of effort (in number of days) of each team member in each of the evaluation activities.
3. Professional references: please provide two references from your previous clients.
4. Writing sample demonstrating use of data and analytical skills (limit of 5 pages).
5. Daily rate: please mention the proposed daily rate in USD.
The Summary of experience should be no more than one page and should include the following:
1. Experience in leading project/program evaluations
o number of evaluations led (with dates, locations and names of organizations)
o number of evaluations served as team member
2. Experience in qualitative and quantitative methods, including those described in these TOR
o numbers of years of experience
o tools/methods used in past
3. Experience in HIV and public health related issues
o number of years of experience
o countries worked in
4. Language proficiency
o clearly state language proficiency in English.
o state proficiency in other languages relevant to this review.
8.2. Application procedures
i) Applications should be sent electronically in one zipped file to ARC to the attention of: Dale.Hill@redcross.org, cc Christine.Fu@redcross.org with a subject line title that contains the words: ARC HIV Review application [add name of candidate or firm]
ii) Questions will be accepted up to 5 days before the deadline (see 8.5 below). Incomplete applications and applications sent after the deadline will not be accepted.
iii) Application files (Word-2003 and above, font 12)
8.3. Deadline for applications
Applications are due 11:59PM EST on May 13, 2012.
8.4. Selection criteria
The criteria that will be used to evaluate the applications are:
For the Consultant:
1. Experience in leading large multi-country project/program evaluations
2. Experience in community health and HIV-specific projects/programs
3. Experience in the evaluation and research methods presented in these TORs
4. Experience in any of the countries to be covered by the review
5. Experience in qualitative research methods
8.5. Questions from applicants
We will accept questions up to 5 days before the deadline and post responses (on ARC website) at latest 3 days before the deadline for receiving applications. Questions should be addressed to Christine.fu@redcross.org.
8.6 Budget for the Review
The budget available for this review is estimated to be between $60,000 and $70,000 USD, depending on fees of principals and range of field visits. The maximum budget available is $80,000. We want to encourage all qualified teams to apply. The selection will be based primarily on technical criteria, but qualifications of principals must be commensurate with fees. If you wish to apply, but feel this budget may not be realistic, please submit a comment explaining the reasons for your assessment. The discussion of the inception report will provide an opportunity to make any adjustments to the proposed methodology or budget as agreed by the contractor (s) and evaluation manager.
HIV Global Learning Review, April 2012
Type of evaluation: Review of HIV programming over a ten year period (2002-2011)
Expected evaluation methodologies: Only Qualitative methods are Anticipated--Document review of available documentation (reports, evaluations, etc), key-informant and in-depth interviews; But other methods may be used with justification if desired.
Number of evaluators: 2 evaluators
Expected number of days: 89 work days initial estimate
Budget Estimated: $60,000 to $70,000 with Maximum of $80,000 USD
Deadline for receiving applications: May 13, 2012
1. Description of project/program to be evaluated
1.1. Background
The American Red Cross (ARC) International Services Department (ISD) is undertaking a review of its global HIV programming from 2002 to 2011. This decision comes at a time when ISD is planning for a significant scale-up of its strategic programming areas. An assessment of ARC's $32 million dollar investment in HIV projects over the past decade will inform this scale-up process by reviewing ISD's HIV work to date.
1.2. Scope and reach of HIV project portfolio
ISD has been engaged in the HIV sector internationally since 2002 beginning with a grant from the Anne Ray Charitable Trust (ARCT) to support children affected by HIV in Russia. From this initial grant, the HIV portfolio has grown over the years including one significant program from 2004-2010 called "Together We Can" that was implemented in Guyana, Haiti and Tanzania with the support of the US Agency for International Development. Since then the portfolio has expanded to cover 11 countries.
Through the support of ARCT, ARC's Strategic Investment Funds and a U.S. government grant, ARC has committed over $32 million dollars over 10 years, making HIV one of the two main pillars of the health portfolio for ISD (the other being the Measles Initiative) . ISD HIV programs focus on providing information on risk reduction, comprehensive HIV prevention interventions, care and support services for People living with HIV (PLHIV), promotion of adherence to antiretroviral and tuberculosis treatment, and reducing stigma and/or discrimination at the household and community levels. As of January 2012, ISD is actively supporting projects in: Bahamas, China, Guyana, Haiti, Jamaica, Kazakhstan, Kenya, Russia, Tanzania, Ukraine and Vietnam. Collaborative efforts are also underway in Belarus and Armenia. A brief overview of the HIV portfolio from 2002-2011 is included in Annex 1.
1.3. Project/program management
Sources of project/program funding include external agencies (e.g. US Government Agencies or private sources like the Anne Ray Charitable Trust), ARC funding, and matching resources (sometimes in kind) from the Host National Societies of the countries in which the projects and programs are implemented. Projects and programs are implemented by Host National Societies (NS) in partnership with ARC, whose resources include field-based Delegates, region-specific Program Officers and Directors at National Headquarters (NHQ) in Washington, D.C. and the support of Technical Advisors. The portfolio includes some multi-country programs, and some programs include projects designed and implemented at the National Society branch level. For ease of language in referring to diverse cases, these TORs will refer to all these projects and programs as "ARC projects/programs" but it should be understood that funding may be from diverse sources, all projects are implemented in partnership with the HNSs and their branches, and implementation responsibilities may be assigned at different levels.
2. Overview of Global HIV Learning Review
2.1. Purpose and objectives of review
The purpose of this global learning review is to provide a retrospective overview of ISD's HIV programming from the past ten years, capturing lessons learned from the perspective of both ARC and host NSs. This global learning review of past and present HIV programming builds a foundation for shared reflection and learning with host NSs. Knowledge garnered from this global learning review will help inform ARC's future programmatic development. ISD will utilize information gleaned from this study for the following three objectives:
1. Lessons learned for ARC: Alignment: To assess how ARC's HIV programming aligns with its own broader priorities and with NSs' priorities and capacities concerning health, particularly HIV. The review should address the following questions:
i. How feasible, relevant and sustainable are ARC HIV projects/programs, from NS' perspective? How do the services and activities of these projects/programs compare to those in the health sector implemented by NSs (past and present)?
ii. What types of resource inputs were a part of the ARC HIV projects/ programs and to what degree were NSs able to absorb resource inputs from ARC HIV projects/programs? What features of design and factors in implementation contributed to successful absorption?
iii. Considering the degree to which ARC's broader programmatic objectives were realized through HIV Projects/programs (strengthening NS' capacity and the resilience of local communities), what factors were important in realizing these objectives?
2. Lessons learned for host NSs and ARC: Results and Sustainability: To assess the results from ARC HIV projects/programs in terms of the effects they had on National Societies' capabilities to serve the community going forward.
Based on knowledge from Objective 1, the review should examine the results of ARC HIV projects/programs (past and present) as they impacted NSs.
i. What effects have ARC HIV projects/programs had on NSs organizationally?
ii. What effects have ARC HIV projects/programs had on NS' technical capacity in HIV?
iii. What effects have ARC HIV projects/programs had on NS' project management capabilities?
iv. What effects have ARC HIV projects/programs had on HIV and health services available to communities?
v. To what degree have ARC HIV projects/programs' services and activities crowded out, supported, complemented and/or strengthened NS capacity to deliver and/or expand HIV and health services available to communities?
vi. Ex-post evaluation: For past projects that have ended, which NSs have sustained service provision to communities? What factors contributed to being able to sustain service provision?
3. Models going forward: To provide a foundation from which ISD can consider options for supporting NS' health and HIV services going forward.
Based on knowledge from Objectives 1 and 2, the review should examine appropriate implementation support and partnership models that ARC can use to help HNSs' sustain service provision to current beneficiaries. The focus of this objective pertains to active ARC HIV projects that include the Caribbean HIV/AIDS Project, Prevention, Care and Support Project in Vietnam, Tanzania Integrated HIV project, Haiti PrevSIDA/HHAP and Regional Health Initiative in Eastern Europe and Central Asia.
i. For current on-going projects, how sustainable are the service provision models that NSs are conducting with communities in relation to NSs' capacity, strategic priorities, partnerships, donor climate, etc.?
ii. What challenges do NSs experience in providing HIV-related services to communities?
iii. What opportunities exist to strengthen NS capacity to provide and sustain HIV-related services to local communities?
iv. What are some ways ARC can capitalize on and/or foster opportunities to enhance NS' capacity to provide and sustain HIV-related services to local communities?
2.4. Main audience of Global HIV Learning Review
The main audience for this evaluation is senior ARC management, who envisages forming conclusions and a strategy, which is then expected to be shared with key donors and partners (e.g. IFRC), in order to determine the future of health programming.
2.5. Coverage of evaluation
Through a desk review, this review will look at the entire HIV program portfolio over the past 10 years (2002-2011), including those completed and those still ongoing, using available project documentation and key staff interviews to be determined in collaboration with the M&E team, regional directors, Senior HIV Advisor, Senior Health Coordinator and senior management. The review will also examine (in less detail) the NS program of health in the countries where the HIV interventions take place.
3. Evaluation criteria and questions
Under the previously mentioned criteria the evaluation will address the following questions:
Criteria
Objective 1:
Alignment: To assess how ARC's HIV programming aligns with its own broader priorities and with NSs' priorities and capacities concerning health, particularly HIV.
Objective 1: Main evaluation questions
1. How feasible, relevant and sustainable are ARC HIV projects/programs, from NS' perspective? How do the services and activities of these projects/programs compare to those in the health sector implemented by NSs (past and present)?
2. What types of resource inputs were a part of the ARC HIV projects/ programs and to what degree were NSs able to absorb resource inputs from ARC HIV projects/programs? What features of design and factors in implementation contributed to successful absorption?
3. To the degree to which ARC's broader programmatic objectives were realized (strengthening NSs capacity and the resilience of local communities), what were important factors?
Objective 1: Sub-questions
1. What health and HIV projects/programs are NSs implementing? How are they being implemented and why?
2. How relevant is addressing HIV to NSs strategic priorities?
3. Did the ARC HIV projects/programs contribute (beyond their projects' service delivery objectives) to strengthening NS' capacity and community resilience?
4. If so, what design features and/or factors in implementation were important?
5. How transferable are these factors across NSs?
Objective 2:
Results and Sustainability: To assess the results from ARC HIV projects/programs in terms of the effects they had on National Societies' capabilities to serve the community going forward.
Objective 2: Main Questions
1. What effects have ARC HIV projects/programs had on NSs organizationally?
2. What effects have ARC HIV projects/programs had on NS' technical capacity in HIV?
3. What effects have ARC HIV projects/programs had on NS' project management capabilities?
4. What effects have ARC HIV projects/programs had on HIV and health services available to communities?
5. To what degree have ARC HIV projects/programs' services and activities crowded out, supported, complemented and/or strengthened NS capacity to deliver and/or expand HIV and health services available to communities
6. Ex-post evaluation: For past projects that have ended, which NSs have sustained service provision to communities? What factors contributed to being able to sustain service provision?
Objective 2: Sub-Questions
1. For projects that have closed, have NSs been able to sustain certain project components to communities? How have they been sustained?
2. For current projects, what actions are NSs doing to sustain services after the grant period ends?
3. What has been the contribution of ARC HIV projects/ programs to the ability of NS' to sustain service delivery?
4. What key elements of those projects can be applied to future efforts to support NSs in sustaining services after the grant periods end?
Objective 3:
Models Going Forward:
Based on Objectives 1 and 2, examine appropriate implementation support and partnership models that ARC can use to help HNSs' sustain service provision to current beneficiaries.
Objective 3: Main Questions
1. For current on-going projects, how sustainable are the service provision models that NSs are conducting with communities in relation to NSs' capacity, strategic priorities, partnerships, donor climate, etc.?
2. What challenges do NSs experience in providing HIV-related services to communities?
3. What opportunities exist to strengthen NS capacity to provide and sustain HIV-related services to local communities?
4. What are some ways ARC can capitalize on and/or foster opportunities to enhance NS' capacity to provide and sustain HIV-related services to local communities?
Objective 3: Sub-Questions
1. Would successful models identified in Objective 2 be appropriate in any of ARC's current HIV projects? (in terms of culture, target group, etc)
2. For models that may be appropriate, how can those models be adapted to help NSs sustain service provision to current beneficiaries?
3. What aspects of the successful partnership models identified in Objective 2 can be effective in any of ARC's current HIV projects?
4. For potentially helpful partnerships, how can ARC help foster and/or strengthen those connections?
4. Scope of work and Evaluation design
4.1. Scope of work
The contractor(s) will be responsible for the following steps and deliverables:
i) An inception report, detailing a recommended methodology and work plan to meet the evaluation objectives satisfactorily in the time and budget provided. The inception report will be discussed at headquarters before further work progresses, particularly to discuss the role and timing of selected country visits;
ii) Before country visits, if any, more detailed review of background documents on the projects and the countries in which they are implemented if needed—e.g. project proposals, as updated, quarterly project reviews, evaluations, completion reports, if any;
iii) Preparation of data collection instruments, list of potential institutions to consult, preliminary interviewees list (can be prepared with input from ARC staff), and interview guides; criteria for site visits/inspections, if any;
iv) Executive summary of initial macro-level findings for organizational management meeting in September 2012;
v) A Draft Report, with list and titles of people interviewed. Separate annexes of country profiles summarizing findings from objectives 1-3 to be subject to review by national stakeholders, for each country portfolio.
vi) Workshop to present findings at headquarters in Power Point format;
vii) Revised Final Report which takes account of all comments received from the workshop and from national stakeholders.
4.2. Evaluation methods
i) Desk review of key documents, including strategy documents, prior evaluation reports, monitoring reports and other documents judged relevant.
ii) Assessment of prior baseline or end line survey/assessments, as relevant.
iii) Literature search and review of material on the environment in which the program operates, and recent developments which impact the feasibility, relevancy and sustainability of NSs conducting health programming, particularly in HIV, including evaluations and annual reports of other local agencies implementing HIV projects in the same sector and with the same/similar target population.
iv) Interviews with key project/program staff—both headquarters and field (could be by phone).
v) Interviews and/or other qualitative methods to hear perspectives and views of representatives of the project/program stakeholders (Host National Society, implementing partners, key informants, past evaluators, donors, etc.)
vi) Other participatory approaches, such as round tables with national stakeholders to assess the "most significant change" as results of ARC interventions.
4.3. Discussion of inception report, and subsequent points of consultation with Evaluation Manager
An inception report discussing the evaluation work plan, research questions, design and methodology will be developed by the consultant after 10 days of selective document/literature review and interviews with a few key staff at NHQ. This inception report will then be shared with the M&E team, Senior HIV Advisor, Regional Directors and International Response and Programs (IRP) Senior Director at NHQ for review and further discussion. Development of data collection instruments or elaboration of methodological approaches will commence once the inception report has been finalized and agreed upon by the consultant and the Senior M and E Advisor.
Data collection instruments or qualitative information collection guides should be submitted to the Evaluation manager for approval, allowing 5 days for coordination of review. In addition, consultation with the Evaluation manager and respective Regional Directors is required before final plans are made for any country or site visits. A minimum of 5 days notice per country is required to coordinate consultation and logistic support for the country visits.
4.4. Logistic and Administrative Support
The M&E team, Senior HIV advisor, program officers and field staff will provide logistical support in coordinating interviews and meetings. Local transportation in the field will be provided whenever possible and may be required in some countries for security reasons. Translation needs leading up to production of the final report, either for documents or field data collection, need to be factored into consultancy cost. While ARC staff may assist in references to qualified personnel, costs need to be borne by the consultants. Translation of the final report or other products will be borne by ARC.
4.5. Reporting relationship
The contractor will report to the designated evaluation manager, the Senior M&E Advisor, International Response and Programs (IRP), or delegee, with technical support from the Senior HIV Advisor, IRP. The Evaluation Manager will coordinate at key points with other technical advisors, and POs as needed, but particularly the Senior HIV Advisor and IRP Senior Director.
4.6. International standards & Presentation of evidence
Standard social science research, evaluation and survey methodologies and good practices utilized in the international humanitarian community should be applied. Such resources should include but are not limited to those promulgated by the Active Learning Network for Accountability and Performance and the Organization for Economic Co-operation and Development.
In particular, all findings and conclusions should be based on evidence which is presented in the final report, and any assumptions and/or limitations clearly stated. For sample surveys, if applicable, detailed information should be presented on the sample design (including sample size calculation, stratification, clustering, allocation, selection, departures from equal selection probability and weighting), the respondent selection methodology, non-response rates, and coefficient of variation, design effect and intra-class correlation for all variables. For focus group discussions, information on selection of participants should be presented. For case studies, the criteria and processes for selecting those cases should be presented.
ARC does not anticipate that the methods envisaged will require review by National Ethical Review Boards. This topic will need to be addressed in the inception report.
4.7. Ethical Guidelines
It is expected that the evaluation will adhere to ethical guidelines as outlined in the American Evaluation Association's Guiding Principles for Evaluators. A summary of these guidelines is provided below, and a more detailed description can be found at www.eval.org/Publications/GuidingPrinciplesPrintable.asp.
1. Informed Consent: All participants are expected to provide informed consent following standard and pre-agreed upon consent protocols.
2. Systematic Inquiry: Evaluators conduct systematic, data-based inquiries.
3. Competence: Evaluators provide competent performance to stakeholders; specifically, they do not accept assignments which are beyond their competency, or for which they cannot mobilize the requisite expertise.
4. Integrity/Honesty: Evaluators display honesty and integrity in their own behavior, and attempt to ensure the honesty and integrity of the entire evaluation process.
5. Respect for People: Evaluators respect the security, dignity and self-worth of respondents, program participants, clients, and other evaluation stakeholders. It is expected that the evaluator will obtain the informed consent of participants to ensure that they can decide in a conscious, deliberate way whether they want to participate.
6. Responsibilities for General and Public Welfare: Evaluators articulate and take into account the diversity of general and public interests and values that may be related to the evaluation.
4.8. Future use of data
All collected data will be the sole property of the American Red Cross. The contractor may not use the data for their own research purposes, nor license the data to be used by others, without the written consent of the American Red Cross. Dissemination of the review report and any component reports will be determined by a dissemination plan to be prepared by the Evaluation Manager in coordination with other technical advisors and Sr. Management.
5. Expected activities and Deliverables
5.1. Expected activities and Level of Effort (proponents may propose changes)
Activities Number of days
Consultant Team – expected to be 2 people
1. Desk review, literature search and discussions with key program and management staff, 8 days
2. Develop Inception Report including choice of data collection and analysis methods, 5 days
3. Review, discuss and revise inception report with M&E managers (by phone), 5 days
4. Develop data collection and analysis plan, 5 days
5. Site selection and data collection schedule outline, 2 days
6. Discuss site selection and schedule outline with evaluation manager and other stakeholders with view to agreeing on exact itinerary for data collection, 3 days
7. Prepare Country visits, 2 days
8. Undertake country visits, 39 days
9. Prepare executive summary of initial macro-level findings, 3 days
10. Data analysis, 5 days
11. Prepare and submit draft Report for review, 3 days
12. Present findings to ARC staff in Washington, DC, 5 days
13. Finalize report in line with ARC feedback, 5 days
Total expected work days: 89
5.2 Deliverables
Steps/Deliverables* Expected deadline
1. Inception report* Mid-June
2. Finalized data collection instruments* and training tools End June
3. Site visit plan and discussion with NHQ Mid-July
4. Site visits to current HIV project countries completed Late August
5. Executive summary of initial macro-level findings*
Mid-September
6. Draft report* Early October
7. Presentation to ARC staff in Washington, DC* Mid-October
8. Final report*
End October
6. Obligations of Evaluators and Associates
a. Inform the evaluation manager in a timely fashion of progress made and of any problems encountered.
b. Submit any survey or qualitative data collection instruments to the evaluation manager with five days notice for coordination of review.
c. Plan travel itinerary and evaluation activities prior to departure with the evaluation manager and regional directors with at least 5 days notice per country, to aid in coordination with field staff and NS. Keep country ARC and NS staff informed of in-country planned activities, so they may plan optimal participation, and if requested by them, provide a debrief upon completion; record any comments or reactions.
d. Implement the activities as expected, and if modifications are necessary, bring to the attention of the Evaluation Manager before enacting any changes.
e. Report on a timely basis any possible conflicts of interest or any issues which seem to warrant further investigation beyond their TOR or authority.
f. Be responsible for the deliverables outlined in this evaluation study, and the conduct of any associates hired (including local hires such as translators or enumerators); specifically, the lead evaluator will be responsible for the quality and timeliness of the work provided by members of his/her team.
g. Coordinate and synthesize data findings into cohesive and balanced products, including the final report and any derivative products such as powerpoints for briefing.
6.1. Obligations of the Evaluation Manager (or delegee)
a. Facilitate the work of the contractor(s) in headquarters and in the field: Make sure that the contractor(s) know where they can receive specified human resources and logistical support; answer or refer any day-to-day enquiries and aim to ensure responsiveness.
b. Liaise and coordinate evaluation activities with NHQ program officers, and in-country ARC and NS staff to ensure smooth data collection.
b. Monitor the work of the contractor(s) and flag any concerns.
c. Receive and sign off on deliverables and authorize payment.
Obligations of in-country ARC Staff:
a. Facilitate the work of the contractor(s) with beneficiaries and other local stakeholders; brief them on any local security or cultural issues of concern which they need to take into account in planning their work.
b. Raise any issues that affect overall costs or quality of evaluation with the evaluation manager.
6.2. Obligations of the NHQ Technical Team
a. Review and approve the proposed methodology and inception report, consulting management and others as appropriate.
b. Provide technical oversight in the review of relevant deliverables, as requested by the evaluation manager.
c. Provide timely comments on any progress reports and the draft report.
7. Required qualifications: Lead Evaluator
1. Demonstrated extensive experience in: a) public health, preferably HIV programming; and b) program evaluation
2. Masters in Public Health, and/or equivalent experience in public health, particularly community health.
3. Demonstrated experience in leading evaluations of humanitarian/development projects/programs.
4. Demonstrated experience in qualitative data collection and analysis.
5. Demonstrated experience in training and managing enumerators or collectors of qualitative information, if required.
6. Excellent analytical and critical thinking skills.
7. Demonstrated experience of written work that has informed practice.
8. Professional work experience in country or region preferred.
9. Strong competence working and communicating cross-culturally.
10. Good oral presentation skills.
11. Fluency in English required, in addition to French and/or Russian desirable.
8. Application and selection details
8.1. Application materials
The application should include the following five items. Please note that any application that does not contain all five items will be rejected. All documents should be single-spaced in Times New Roman font size 12.
1. One-page Summary of experience.
2. Detailed CVs of all professionals who will work on the evaluation. If there is more than one consultant on the proposed evaluation team, please attach a table describing the level of effort (in number of days) of each team member in each of the evaluation activities.
3. Professional references: please provide two references from your previous clients.
4. Writing sample demonstrating use of data and analytical skills (limit of 5 pages).
5. Daily rate: please mention the proposed daily rate in USD.
The Summary of experience should be no more than one page and should include the following:
1. Experience in leading project/program evaluations
o number of evaluations led (with dates, locations and names of organizations)
o number of evaluations served as team member
2. Experience in qualitative and quantitative methods, including those described in these TOR
o numbers of years of experience
o tools/methods used in past
3. Experience in HIV and public health related issues
o number of years of experience
o countries worked in
4. Language proficiency
o clearly state language proficiency in English.
o state proficiency in other languages relevant to this review.
8.2. Application procedures
i) Applications should be sent electronically in one zipped file to ARC to the attention of: Dale.Hill@redcross.org, cc Christine.Fu@redcross.org with a subject line title that contains the words: ARC HIV Review application [add name of candidate or firm]
ii) Questions will be accepted up to 5 days before the deadline (see 8.5 below). Incomplete applications and applications sent after the deadline will not be accepted.
iii) Application files (Word-2003 and above, font 12)
8.3. Deadline for applications
Applications are due 11:59PM EST on May 13, 2012.
8.4. Selection criteria
The criteria that will be used to evaluate the applications are:
For the Consultant:
1. Experience in leading large multi-country project/program evaluations
2. Experience in community health and HIV-specific projects/programs
3. Experience in the evaluation and research methods presented in these TORs
4. Experience in any of the countries to be covered by the review
5. Experience in qualitative research methods
8.5. Questions from applicants
We will accept questions up to 5 days before the deadline and post responses (on ARC website) at latest 3 days before the deadline for receiving applications. Questions should be addressed to Christine.fu@redcross.org.
8.6 Budget for the Review
The budget available for this review is estimated to be between $60,000 and $70,000 USD, depending on fees of principals and range of field visits. The maximum budget available is $80,000. We want to encourage all qualified teams to apply. The selection will be based primarily on technical criteria, but qualifications of principals must be commensurate with fees. If you wish to apply, but feel this budget may not be realistic, please submit a comment explaining the reasons for your assessment. The discussion of the inception report will provide an opportunity to make any adjustments to the proposed methodology or budget as agreed by the contractor (s) and evaluation manager.
Post a Comment