Final Evaluation of UNOPS Transitional Shelter Project American Red Cross
October 26, 2011
Dear Sir/Madam,
Subject: Request for Proposal
1. The American Red Cross, Haiti Delegation hereby solicits your proposal for the provision of a Final evaluation of UNOPS transitional shelter project, Haiti; as per the attached Draft Terms of Reference (TOR) that includes the scope of work.
2. The Request for Proposal (RFP) consists of this letter of transmittal and the following enclosures: Enclosure 1: Specifications/TOR
3. Please note that this document is an RFP and not an invitation to bid. You should also note that the terms set forth in this RFP, the Draft TOR (Enclosure 1) will form a part of any contract should the American Red Cross (ARC) accept your proposal.
4. It is anticipated that any contract or contracts entered into as a result of this RFP will be for a period of 6-7 weeks (approx. 35-40 working days), beginning Early November, 2011.
5. This RFP does not commit the ARC to award a contract or to pay any costs incurred in the preparation or submission of proposals, or costs incurred in making necessary studies for the preparation thereof, or to procure or contract for services or supplies. The ARC reserves the right to reject any or all proposals received in response to this RFP and to negotiate with any of the proposers or other firms in any manner deemed to be in the best interest of the ARC. It also reserves the right to negotiate and award separate or multiple contracts for the elements covered by this RFP in any combination it may deem appropriate, in its sole discretion; modify or exclude any consideration, information or requirement contained in this RFP, and to add new considerations, information or requirements at any stage of the procurement process, including during negotiations with proposers.
6. Proposers must provide all requisite information and clearly and concisely respond to all points set out in this RFP. Any proposal which does not fully and comprehensively address this RFP will be rejected. However, unnecessarily elaborate brochures and other presentations beyond that sufficient to present a complete and effective proposal are not encouraged.
7. The normal terms of payment of the ARC are within 30 days of satisfactory delivery of goods or services and documents in apparent good order. Proposers must therefore clearly specify in their Proposal the payment terms being offered if different from these.
8. The agency/person is expected to submit a combined technical and financial proposal (maximum 5 pages) in English submitted no later than 11:59pm EST, Nov 4, 2011.
9. Any proposals received after the stated opening time and date will be rejected.
10. Proposals must be sent to: arc.evaluation@gmail.com. Please use the following in the subject line of your email: “Consultant – UNOPS Evaluation”.
11. All elements of the proposal should be contained in one single file, in either Word or pdf format. CV and proposed daily fee for each consultant, in one single Word or pdf file. The title of the file should use the following naming convention: Lastname_UNOPSevaluation.
12. The proposal should include
Detailed CVs of the professional(s) who will work on the evaluation. If more than one consultant on the proposed evaluation team, please attach a table describing the level of effort (in number of days) of each team member in each of the evaluation activities.
References: Provide two or three references from your previous clients.
Daily Rate: Please mention the expected daily rate in USD.
13. Proposals will be reviewed and evaluated by the ARC in accordance with the provisions of the ARC’s Procurement Policy as well as the considerations, information and requirements contained in this RFP. The evaluation procedure will consist of a formal, substantive and financial assessment of the proposals received. Price is an important factor; however, it is not the only consideration in evaluating responses to an RFP.
14. Your proposal shall remain valid and open for acceptance for a period of at least sixty (60) days from the submission date. Please indicate in your proposal that it will remain valid for this period.
15. Following submission of the proposals and final evaluation, the ARC will have the right to retain unsuccessful proposals. It is the proposer’s responsibility to identify any information of a confidential or proprietary nature contained in its proposal, so that it may be handled accordingly.
Draft Terms of Reference for External Consultant/Firm
Final evaluation of UNOPS transitional shelter project, Haiti
October 21, 2011
1. Evaluation overview
1.1. Type of evaluation
This evaluation will be an end-of-project evaluation.
1.2. Evaluation methodologies
The evaluation will use qualitative, quantitative or mixed research methodologies.
1.3. Number of evaluators
A team of two evaluators will work on the evaluation:
1. one with demonstrated technical skills and experience in Shelter
2. one with demonstrated experience in leading evaluations of humanitarian projects
The first evaluator (Shelter) has already been selected. Applications will be accepted from individual consultants for the second position (Evaluation).
1.4. Expected start/end dates, Number of work days
The evaluation is expected to begin in early November 2011 and end in mid-December 2011. The expected number of work days are 35-40 days for each member of the evaluation team.
1.5. Budget
The budget for this evaluation is $50,000.
1.6. Deadline for receiving applications
11:59PM EST, Friday November 4, 2011
2. Description of project to be evaluated
2.1. Background and objectives of project
The Haiti earthquake of January 12, 2010 is widely considered to be the most challenging natural disaster in recent history. The American Red Cross (ARC) responded to the devastating earthquake with massive support. ARC human, financial and material resources immediately went to the support of the largest Red Cross/Red Crescent deployments in its history, including twenty-one Emergency Response Units supported by more than seventy Red Cross and Red Crescent National Societies.
To further respond to this disaster, ARC planned its early recovery programming in tandem with ongoing emergency response operations. In this context, ARC launched a Request for Applications (RFA) in April 2010 focused on the transitional shelter and host family support components of ARC’s Haiti Assistance Program. A total of five transitional shelter partners were selected for funding under this objective to construct a total of 6,500 transitional shelters.
The goal of ARC’s shelter program is to meet immediate and long-term shelter needs of people left homeless by the earthquake. Ultimately ARC expects to support shelter through all phases of recovery, but this RFA focused on the transition phase.
In addition to constructing transitional shelters, all projects were required to address access to water and sanitation, either through direct implementation, partnerships or any other mechanisms in order to achieve the Wash cluster guidelines for transitional shelter beneficiaries. DRR, environment, capacity building, long term recovery and gender were to be taken into account as part of project design and implementation activities.
UNOPS was funded under Objective 1 of the RFA, to support vulnerable earthquake-affected families living in inadequate shelter with transitional shelter intended to bridge the gap between relief and reconstruction. The aim of the UNOPS project was to achieve the following objective: to provide durable transitional shelters and WatSan assistance for communities affected by the earthquake.
2.2. Scope and reach of project
Of the five organizations funded under the RFA, UNOPS was to construct 1,500 transitional shelters in accordance with the guidelines set forth in the RFA. Located in Greater Port-au-Prince, UNOPS aimed to reach approximately 1,500 families (or 7,500 direct beneficiaries using an average household size of 5). Other activities of the project were Water and Sanitation (WatSan) and Disaster Risk Reduction (DRR).
2.3. Project management
This project has been directly implemented and managed by UNOPS.
2.4. Previous evaluation activities
Due to the short length of the project, previous evaluations were not conducted.
3. Purpose, objectives, audience and coverage
3.1. Purpose of evaluation
1. To provide inputs for the future meta-evaluation, as well as for the implementation design for the permanent housing program.
2. To determine positioning and lessons learned for similar future shelter projects/programs.
3. To assess compliance with Grant Agreement requirements
3.2. Objectives of the evaluation
1. Describe and assess project results – intended and unintended, positive and negative
2. Assess the major factors which influence results either positively or negatively
3. Draws lessons learned
3.3. Main audience of evaluation
The audience of this evaluation is the American Red Cross Haiti Assistance Program field project/program delegates and management, NHQ technical team, NHQ senior management, American Red Cross Transitional Shelter partners, and RC/RC Movement partners.
3.4. Coverage
The evaluation is intended to be comprehensive and the analysis is expected to cover all areas in which UNOPS has implemented the project (Port au Prince, Martissant and Carrefour- Feuilles). All completed activities covered by the project (Shelter, WASH and DRR) are to be covered, and capacity building, environment, long term recovery and gender are to be taken into account as cross cutting issues.
4. Evaluation criteria and questions
Relevance
1. Was project design relevant and appropriate (taking into account the target populations, timeframe, locations and the Haitian context)? How did project design compare to projects implemented by other humanitarian actors in the same sector and for the same target population?
2. Did the project respond to the specific needs of the target population in both the urban and peri-urban context?
3. Were the T-Shelter designs mindful of the beneficiaries’ needs and wishes?
Effectiveness
4. Were activities implemented as planned? What were the main factors that contributed to whether activities resulted in intended outputs and outcomes?
5. How has land tenure impacted the program, both in implementation and after the handover of the shelter? How are the tenure agreements being respected by land owners and beneficiaries? Are there any cases of eviction of the original beneficiary of the T-shelter since the moment they occupied the shelters?
6. Is the entire beneficiary household living in the T-Shelter? If not, why? Do beneficiaries have other family members residing in the camps to obtain additional services?
7. Were quality standards defined and monitored for all the components of the program, (beneficiary selection and mobilization, shelter construction, capacity building, WASH component, ensuring DRR, gender and environment) in an effective way and did activities achieve high levels of quality in implementation?
8. Effectiveness of Cross-Cutting Themes: How have cross-cutting themes as set forth in the RFA (watsan, DRR, environment, capacity building, long-term recovery, and gender) been incorporated into the overall project design and implementation?
a. How has UNOPS ensured that its timber frame structure has not lead to further deforestation?
b. How has UNOPS assured SPHERE compliance, including the shelter environmental impact standard and WASH standards regarding minimal environmental degradation and watershed protection?
c. How has DRR been taken into consideration in relation to construction site planning, layout and risk reduction techniques in the shelter design?
Results/Outcomes/Impact
9. Did the program achieve its intended outcomes? Were there any important unintended outcomes, either positive or negative?
10. What were the main reasons that determined whether intended outcomes were or were not achieved, and whether there were positive or negative unintended outcomes? Which were under UNOPS control and which were not?
11. Were the t-shelters used for their intended purpose?
Beneficiary perception and satisfaction
12. What was the beneficiaries’ perceived impact of the program? Were beneficiaries satisfied with project outcomes? Were there differences in satisfaction per type of location (urban, peri-urban) or by location?
13. Was there a channel established for beneficiary complaints and response?
Efficiency / cost-effectiveness
14. Were there any noticeable, verifiable instances of waste or inefficiency?
Coverage
15. Did beneficiaries—in general or for specific groups (such as the elderly or disabled, or people with no access to land)—encounter any difficulties accessing the program?
16. Were there any established criteria for beneficiary selection and to what extent were they followed? Did selection criteria evolve over time? If so, why, and what has been the impact of this change both on the program and on the targeted population?
Coherence/connectedness
17. How was the program coordinated with other projects implemented in the area?
18. Were any problems highlighted with program overlap or redundancy?
19. How well did the program adapt its design and objectives to the prevailing humanitarian context for the target population and in the target sector?
Sustainability
20. What sustainability strategies were incorporated into program design?
21. How sustainable were the outcomes of the program? What are the main factors that affect, either positively or negatively, the sustainability of program outcomes?
22. What knowledge have beneficiaries acquired to repair, maintain and/or upgrade their t-shelters and latrines?
Compliance
23. Did UNOPS follow visibility guidelines as set forth by ARC?
24. Did UNOPS meet deadlines as established in the Grant Agreement?
25. Were recommendations and guidance provided from ARC taken into consideration and/or implemented?
5. Scope of work and Evaluation design
5.1. Scope of work
The consultant will be responsible for the following:
1. Developing evaluation methodology and work plan
2. Data collection and analysis
3. Report writing
4. Support of future meta-evaluation of American Red Cross Shelter program
5.2. Methodologies
The following methodologies are expected to be used during the evaluation:
1. Desk review of key documents, to be provided by American Red Cross
2. Literature search and review of evaluations and annual reports of other programs in the same sector and with the same/similar target population in Haiti, in order to a) identify useful secondary data; b) establish benchmarks and/or c) consider good practice indicators or methods of data collection.
3. Interviews with key UNOPS project staff
4. Interviews with representatives of project stakeholders
5. Focus group discussions with stakeholders
Other methodologies can be proposed, including any of the following (list not exhaustive):
6. Probabilistic sample survey of beneficiaries
7. Other participatory qualitative approaches, such as “most significant change”
8. Case studies
9. Physical inspection of facilities
10. Measurement of environmental factors affecting results
5.3. Inception report
After the consultant has had the opportunity to review monitoring data, speak with key project personnel and review background and project materials, the consultant will prepare an inception report that will include details on the proposed methodology, work plan and analysis plan. A discussion will take place with the Evaluation manager to review and finalize the details in the report.
5.4. Logistics and Administrative Support
The American Red Cross can support the consultant with accommodations and transportation (to/from and within Haiti).
5.5. Reporting relationship
The consultant/firm will report to Ranjan Mohnot, Q&L Delegate, who is the designated Evaluation manager.
5.6. International standards & Presentation of evidence
Standard evaluation and survey methodologies and good practices utilized in the international humanitarian community should be applied. Such resources should include but are not limited to those promulgated by the Active Learning Network for Accountability and Performance and the Organization for Economic Co-operation and Development.
In particular, all findings and conclusions should be based on evidence which is presented in the evaluation report. For sample surveys, detailed information should be presented on the sample design (including sample size calculation, stratification, clustering, allocation, selection, departures from equal selection probability and weighting), the respondent selection methodology, nonresponse rates, and coefficient of variation, design effect and intra-class correlation for all variables. For case studies, the criteria and processes for selecting those cases should be presented.
5.7. Ethical Guidelines
It is expected that the evaluation will adhere to ethical guidelines as outlined in the American Evaluation Association’s Guiding Principles for Evaluators. A summary of these guidelines is provided below, and a more detailed description can be found at www.eval.org/Publications/GuidingPrinciplesPrintable.asp.
1. Informed Consent: All participants are expected to provide informed consent following standard and pre-agreed upon consent protocols.
2. Systematic Inquiry: Evaluators conduct systematic, data-based inquiries.
3. Competence: Evaluators provide competent performance to stakeholders.
4. Integrity/Honesty: Evaluators display honesty and integrity in their own behavior, and attempt to ensure the honesty and integrity of the entire evaluation process.
5. Respect for People: Evaluators respect the security, dignity and self-worth of respondents, program participants, clients, and other evaluation stakeholders. It is expected that the evaluator will obtain the informed consent of participants to ensure that they can decide in a conscious, deliberate way whether they want to participate.
6. Responsibilities for General and Public Welfare: Evaluators articulate and take into account the diversity of general and public interests and values that may be related to the evaluation.
5.8. Future use of data
All collected data will be the sole property of the American Red Cross. The consultant/firm may not use the data for their own research purposes, nor license the data to be used by others, without the written consent of the American Red Cross.
6. Expected activities and Deliverables
6.1. Expected activities
Activities Number of days
(illustrative)
1. Desk review, literature search and discussions with key program staff 5
2. Develop and submit inception report for approval 1
3. Develop data collection instruments 3
4. Develop and facilitate data collection training 2
5. Oversee data collection 7
6. Clean and analyze data 3
7. Prepare and submit draft report 7
8. Finalize report in line with ARC feedback 5
9. Work with meta-evaluator during future meta-evaluation (expected March/April) 3
6.2. Deliverables
Deliverables Expected deadline
1. Inception report TBD
2. Finalized data collection instruments TBD
3. Finalized data collection training tools TBD
4. Draft report TBD
5. Final report TBD
6. Participation in workshop with meta-evaluator March/April
7. Obligations of key participants in the evaluation
7.1. Obligations of the NHQ Technical Team
1. Review and approve the proposed methodology via teleconference with ARC field staff on specified date.
2. Provide technical oversight in the review of all deliverables.
3. Provide timely comments on the draft report.
7.2. Obligations of the Evaluation manager
1. Make sure that the consultants are provided with the specified human resources and logistical support, and answer any day-to-day enquiries.
2. Monitor the daily work of the consultants and flag any concerns.
3. Provide timely comments on the draft report
7.3. Obligations of the Consultants
1. Inform the evaluation manager in a timely fashion of progress made and of any problems encountered.
2. Implement the activities as expected, and if modifications are necessary, bring to the attention of the Evaluation manager before enacting any changes.
3. Report on a timely basis any possible conflicts of interest.
8. Required qualifications
Consultant with Evaluation experience
1. Demonstrated experience in leading project/program evaluations
2. Demonstrated professional experience in post-disaster/humanitarian environments
3. Demonstrated experience in qualitative data collection and analysis
4. Demonstrated experience in quantitative data collection and analysis
5. Demonstrated experience in training enumerators and in leading focus group discussions
6. Professional work experience in Haiti strongly preferred
7. Fluency in English required, French strongly preferred
9. Application and selection details
9.1. Application materials
CV and proposed daily fee for each consultant, in one single Word or pdf file. The title of the file should use the following naming convention: Lastname_UNOPSevaluation.
9.2. Application procedures
Send all application materials to the following email address: arc.evaluation@gmail.com. Incomplete applications and applications sent after the deadline will not be accepted.
9.3. Deadline for applications
11:59PM EST, Friday November 4, 2011
9.4. Selection criteria
Applications will be evaluated within one week of receipt according to the following criteria:
1. Experience in leading project/program evaluations
2. Experience in shelter projects/programs
3. Experience in post-disaster/humanitarian context
4. Professional experience in Haiti
5. Cost
Dear Sir/Madam,
Subject: Request for Proposal
1. The American Red Cross, Haiti Delegation hereby solicits your proposal for the provision of a Final evaluation of UNOPS transitional shelter project, Haiti; as per the attached Draft Terms of Reference (TOR) that includes the scope of work.
2. The Request for Proposal (RFP) consists of this letter of transmittal and the following enclosures: Enclosure 1: Specifications/TOR
3. Please note that this document is an RFP and not an invitation to bid. You should also note that the terms set forth in this RFP, the Draft TOR (Enclosure 1) will form a part of any contract should the American Red Cross (ARC) accept your proposal.
4. It is anticipated that any contract or contracts entered into as a result of this RFP will be for a period of 6-7 weeks (approx. 35-40 working days), beginning Early November, 2011.
5. This RFP does not commit the ARC to award a contract or to pay any costs incurred in the preparation or submission of proposals, or costs incurred in making necessary studies for the preparation thereof, or to procure or contract for services or supplies. The ARC reserves the right to reject any or all proposals received in response to this RFP and to negotiate with any of the proposers or other firms in any manner deemed to be in the best interest of the ARC. It also reserves the right to negotiate and award separate or multiple contracts for the elements covered by this RFP in any combination it may deem appropriate, in its sole discretion; modify or exclude any consideration, information or requirement contained in this RFP, and to add new considerations, information or requirements at any stage of the procurement process, including during negotiations with proposers.
6. Proposers must provide all requisite information and clearly and concisely respond to all points set out in this RFP. Any proposal which does not fully and comprehensively address this RFP will be rejected. However, unnecessarily elaborate brochures and other presentations beyond that sufficient to present a complete and effective proposal are not encouraged.
7. The normal terms of payment of the ARC are within 30 days of satisfactory delivery of goods or services and documents in apparent good order. Proposers must therefore clearly specify in their Proposal the payment terms being offered if different from these.
8. The agency/person is expected to submit a combined technical and financial proposal (maximum 5 pages) in English submitted no later than 11:59pm EST, Nov 4, 2011.
9. Any proposals received after the stated opening time and date will be rejected.
10. Proposals must be sent to: arc.evaluation@gmail.com. Please use the following in the subject line of your email: “Consultant – UNOPS Evaluation”.
11. All elements of the proposal should be contained in one single file, in either Word or pdf format. CV and proposed daily fee for each consultant, in one single Word or pdf file. The title of the file should use the following naming convention: Lastname_UNOPSevaluation.
12. The proposal should include
Detailed CVs of the professional(s) who will work on the evaluation. If more than one consultant on the proposed evaluation team, please attach a table describing the level of effort (in number of days) of each team member in each of the evaluation activities.
References: Provide two or three references from your previous clients.
Daily Rate: Please mention the expected daily rate in USD.
13. Proposals will be reviewed and evaluated by the ARC in accordance with the provisions of the ARC’s Procurement Policy as well as the considerations, information and requirements contained in this RFP. The evaluation procedure will consist of a formal, substantive and financial assessment of the proposals received. Price is an important factor; however, it is not the only consideration in evaluating responses to an RFP.
14. Your proposal shall remain valid and open for acceptance for a period of at least sixty (60) days from the submission date. Please indicate in your proposal that it will remain valid for this period.
15. Following submission of the proposals and final evaluation, the ARC will have the right to retain unsuccessful proposals. It is the proposer’s responsibility to identify any information of a confidential or proprietary nature contained in its proposal, so that it may be handled accordingly.
Draft Terms of Reference for External Consultant/Firm
Final evaluation of UNOPS transitional shelter project, Haiti
October 21, 2011
1. Evaluation overview
1.1. Type of evaluation
This evaluation will be an end-of-project evaluation.
1.2. Evaluation methodologies
The evaluation will use qualitative, quantitative or mixed research methodologies.
1.3. Number of evaluators
A team of two evaluators will work on the evaluation:
1. one with demonstrated technical skills and experience in Shelter
2. one with demonstrated experience in leading evaluations of humanitarian projects
The first evaluator (Shelter) has already been selected. Applications will be accepted from individual consultants for the second position (Evaluation).
1.4. Expected start/end dates, Number of work days
The evaluation is expected to begin in early November 2011 and end in mid-December 2011. The expected number of work days are 35-40 days for each member of the evaluation team.
1.5. Budget
The budget for this evaluation is $50,000.
1.6. Deadline for receiving applications
11:59PM EST, Friday November 4, 2011
2. Description of project to be evaluated
2.1. Background and objectives of project
The Haiti earthquake of January 12, 2010 is widely considered to be the most challenging natural disaster in recent history. The American Red Cross (ARC) responded to the devastating earthquake with massive support. ARC human, financial and material resources immediately went to the support of the largest Red Cross/Red Crescent deployments in its history, including twenty-one Emergency Response Units supported by more than seventy Red Cross and Red Crescent National Societies.
To further respond to this disaster, ARC planned its early recovery programming in tandem with ongoing emergency response operations. In this context, ARC launched a Request for Applications (RFA) in April 2010 focused on the transitional shelter and host family support components of ARC’s Haiti Assistance Program. A total of five transitional shelter partners were selected for funding under this objective to construct a total of 6,500 transitional shelters.
The goal of ARC’s shelter program is to meet immediate and long-term shelter needs of people left homeless by the earthquake. Ultimately ARC expects to support shelter through all phases of recovery, but this RFA focused on the transition phase.
In addition to constructing transitional shelters, all projects were required to address access to water and sanitation, either through direct implementation, partnerships or any other mechanisms in order to achieve the Wash cluster guidelines for transitional shelter beneficiaries. DRR, environment, capacity building, long term recovery and gender were to be taken into account as part of project design and implementation activities.
UNOPS was funded under Objective 1 of the RFA, to support vulnerable earthquake-affected families living in inadequate shelter with transitional shelter intended to bridge the gap between relief and reconstruction. The aim of the UNOPS project was to achieve the following objective: to provide durable transitional shelters and WatSan assistance for communities affected by the earthquake.
2.2. Scope and reach of project
Of the five organizations funded under the RFA, UNOPS was to construct 1,500 transitional shelters in accordance with the guidelines set forth in the RFA. Located in Greater Port-au-Prince, UNOPS aimed to reach approximately 1,500 families (or 7,500 direct beneficiaries using an average household size of 5). Other activities of the project were Water and Sanitation (WatSan) and Disaster Risk Reduction (DRR).
2.3. Project management
This project has been directly implemented and managed by UNOPS.
2.4. Previous evaluation activities
Due to the short length of the project, previous evaluations were not conducted.
3. Purpose, objectives, audience and coverage
3.1. Purpose of evaluation
1. To provide inputs for the future meta-evaluation, as well as for the implementation design for the permanent housing program.
2. To determine positioning and lessons learned for similar future shelter projects/programs.
3. To assess compliance with Grant Agreement requirements
3.2. Objectives of the evaluation
1. Describe and assess project results – intended and unintended, positive and negative
2. Assess the major factors which influence results either positively or negatively
3. Draws lessons learned
3.3. Main audience of evaluation
The audience of this evaluation is the American Red Cross Haiti Assistance Program field project/program delegates and management, NHQ technical team, NHQ senior management, American Red Cross Transitional Shelter partners, and RC/RC Movement partners.
3.4. Coverage
The evaluation is intended to be comprehensive and the analysis is expected to cover all areas in which UNOPS has implemented the project (Port au Prince, Martissant and Carrefour- Feuilles). All completed activities covered by the project (Shelter, WASH and DRR) are to be covered, and capacity building, environment, long term recovery and gender are to be taken into account as cross cutting issues.
4. Evaluation criteria and questions
Relevance
1. Was project design relevant and appropriate (taking into account the target populations, timeframe, locations and the Haitian context)? How did project design compare to projects implemented by other humanitarian actors in the same sector and for the same target population?
2. Did the project respond to the specific needs of the target population in both the urban and peri-urban context?
3. Were the T-Shelter designs mindful of the beneficiaries’ needs and wishes?
Effectiveness
4. Were activities implemented as planned? What were the main factors that contributed to whether activities resulted in intended outputs and outcomes?
5. How has land tenure impacted the program, both in implementation and after the handover of the shelter? How are the tenure agreements being respected by land owners and beneficiaries? Are there any cases of eviction of the original beneficiary of the T-shelter since the moment they occupied the shelters?
6. Is the entire beneficiary household living in the T-Shelter? If not, why? Do beneficiaries have other family members residing in the camps to obtain additional services?
7. Were quality standards defined and monitored for all the components of the program, (beneficiary selection and mobilization, shelter construction, capacity building, WASH component, ensuring DRR, gender and environment) in an effective way and did activities achieve high levels of quality in implementation?
8. Effectiveness of Cross-Cutting Themes: How have cross-cutting themes as set forth in the RFA (watsan, DRR, environment, capacity building, long-term recovery, and gender) been incorporated into the overall project design and implementation?
a. How has UNOPS ensured that its timber frame structure has not lead to further deforestation?
b. How has UNOPS assured SPHERE compliance, including the shelter environmental impact standard and WASH standards regarding minimal environmental degradation and watershed protection?
c. How has DRR been taken into consideration in relation to construction site planning, layout and risk reduction techniques in the shelter design?
Results/Outcomes/Impact
9. Did the program achieve its intended outcomes? Were there any important unintended outcomes, either positive or negative?
10. What were the main reasons that determined whether intended outcomes were or were not achieved, and whether there were positive or negative unintended outcomes? Which were under UNOPS control and which were not?
11. Were the t-shelters used for their intended purpose?
Beneficiary perception and satisfaction
12. What was the beneficiaries’ perceived impact of the program? Were beneficiaries satisfied with project outcomes? Were there differences in satisfaction per type of location (urban, peri-urban) or by location?
13. Was there a channel established for beneficiary complaints and response?
Efficiency / cost-effectiveness
14. Were there any noticeable, verifiable instances of waste or inefficiency?
Coverage
15. Did beneficiaries—in general or for specific groups (such as the elderly or disabled, or people with no access to land)—encounter any difficulties accessing the program?
16. Were there any established criteria for beneficiary selection and to what extent were they followed? Did selection criteria evolve over time? If so, why, and what has been the impact of this change both on the program and on the targeted population?
Coherence/connectedness
17. How was the program coordinated with other projects implemented in the area?
18. Were any problems highlighted with program overlap or redundancy?
19. How well did the program adapt its design and objectives to the prevailing humanitarian context for the target population and in the target sector?
Sustainability
20. What sustainability strategies were incorporated into program design?
21. How sustainable were the outcomes of the program? What are the main factors that affect, either positively or negatively, the sustainability of program outcomes?
22. What knowledge have beneficiaries acquired to repair, maintain and/or upgrade their t-shelters and latrines?
Compliance
23. Did UNOPS follow visibility guidelines as set forth by ARC?
24. Did UNOPS meet deadlines as established in the Grant Agreement?
25. Were recommendations and guidance provided from ARC taken into consideration and/or implemented?
5. Scope of work and Evaluation design
5.1. Scope of work
The consultant will be responsible for the following:
1. Developing evaluation methodology and work plan
2. Data collection and analysis
3. Report writing
4. Support of future meta-evaluation of American Red Cross Shelter program
5.2. Methodologies
The following methodologies are expected to be used during the evaluation:
1. Desk review of key documents, to be provided by American Red Cross
2. Literature search and review of evaluations and annual reports of other programs in the same sector and with the same/similar target population in Haiti, in order to a) identify useful secondary data; b) establish benchmarks and/or c) consider good practice indicators or methods of data collection.
3. Interviews with key UNOPS project staff
4. Interviews with representatives of project stakeholders
5. Focus group discussions with stakeholders
Other methodologies can be proposed, including any of the following (list not exhaustive):
6. Probabilistic sample survey of beneficiaries
7. Other participatory qualitative approaches, such as “most significant change”
8. Case studies
9. Physical inspection of facilities
10. Measurement of environmental factors affecting results
5.3. Inception report
After the consultant has had the opportunity to review monitoring data, speak with key project personnel and review background and project materials, the consultant will prepare an inception report that will include details on the proposed methodology, work plan and analysis plan. A discussion will take place with the Evaluation manager to review and finalize the details in the report.
5.4. Logistics and Administrative Support
The American Red Cross can support the consultant with accommodations and transportation (to/from and within Haiti).
5.5. Reporting relationship
The consultant/firm will report to Ranjan Mohnot, Q&L Delegate, who is the designated Evaluation manager.
5.6. International standards & Presentation of evidence
Standard evaluation and survey methodologies and good practices utilized in the international humanitarian community should be applied. Such resources should include but are not limited to those promulgated by the Active Learning Network for Accountability and Performance and the Organization for Economic Co-operation and Development.
In particular, all findings and conclusions should be based on evidence which is presented in the evaluation report. For sample surveys, detailed information should be presented on the sample design (including sample size calculation, stratification, clustering, allocation, selection, departures from equal selection probability and weighting), the respondent selection methodology, nonresponse rates, and coefficient of variation, design effect and intra-class correlation for all variables. For case studies, the criteria and processes for selecting those cases should be presented.
5.7. Ethical Guidelines
It is expected that the evaluation will adhere to ethical guidelines as outlined in the American Evaluation Association’s Guiding Principles for Evaluators. A summary of these guidelines is provided below, and a more detailed description can be found at www.eval.org/Publications/GuidingPrinciplesPrintable.asp.
1. Informed Consent: All participants are expected to provide informed consent following standard and pre-agreed upon consent protocols.
2. Systematic Inquiry: Evaluators conduct systematic, data-based inquiries.
3. Competence: Evaluators provide competent performance to stakeholders.
4. Integrity/Honesty: Evaluators display honesty and integrity in their own behavior, and attempt to ensure the honesty and integrity of the entire evaluation process.
5. Respect for People: Evaluators respect the security, dignity and self-worth of respondents, program participants, clients, and other evaluation stakeholders. It is expected that the evaluator will obtain the informed consent of participants to ensure that they can decide in a conscious, deliberate way whether they want to participate.
6. Responsibilities for General and Public Welfare: Evaluators articulate and take into account the diversity of general and public interests and values that may be related to the evaluation.
5.8. Future use of data
All collected data will be the sole property of the American Red Cross. The consultant/firm may not use the data for their own research purposes, nor license the data to be used by others, without the written consent of the American Red Cross.
6. Expected activities and Deliverables
6.1. Expected activities
Activities Number of days
(illustrative)
1. Desk review, literature search and discussions with key program staff 5
2. Develop and submit inception report for approval 1
3. Develop data collection instruments 3
4. Develop and facilitate data collection training 2
5. Oversee data collection 7
6. Clean and analyze data 3
7. Prepare and submit draft report 7
8. Finalize report in line with ARC feedback 5
9. Work with meta-evaluator during future meta-evaluation (expected March/April) 3
6.2. Deliverables
Deliverables Expected deadline
1. Inception report TBD
2. Finalized data collection instruments TBD
3. Finalized data collection training tools TBD
4. Draft report TBD
5. Final report TBD
6. Participation in workshop with meta-evaluator March/April
7. Obligations of key participants in the evaluation
7.1. Obligations of the NHQ Technical Team
1. Review and approve the proposed methodology via teleconference with ARC field staff on specified date.
2. Provide technical oversight in the review of all deliverables.
3. Provide timely comments on the draft report.
7.2. Obligations of the Evaluation manager
1. Make sure that the consultants are provided with the specified human resources and logistical support, and answer any day-to-day enquiries.
2. Monitor the daily work of the consultants and flag any concerns.
3. Provide timely comments on the draft report
7.3. Obligations of the Consultants
1. Inform the evaluation manager in a timely fashion of progress made and of any problems encountered.
2. Implement the activities as expected, and if modifications are necessary, bring to the attention of the Evaluation manager before enacting any changes.
3. Report on a timely basis any possible conflicts of interest.
8. Required qualifications
Consultant with Evaluation experience
1. Demonstrated experience in leading project/program evaluations
2. Demonstrated professional experience in post-disaster/humanitarian environments
3. Demonstrated experience in qualitative data collection and analysis
4. Demonstrated experience in quantitative data collection and analysis
5. Demonstrated experience in training enumerators and in leading focus group discussions
6. Professional work experience in Haiti strongly preferred
7. Fluency in English required, French strongly preferred
9. Application and selection details
9.1. Application materials
CV and proposed daily fee for each consultant, in one single Word or pdf file. The title of the file should use the following naming convention: Lastname_UNOPSevaluation.
9.2. Application procedures
Send all application materials to the following email address: arc.evaluation@gmail.com. Incomplete applications and applications sent after the deadline will not be accepted.
9.3. Deadline for applications
11:59PM EST, Friday November 4, 2011
9.4. Selection criteria
Applications will be evaluated within one week of receipt according to the following criteria:
1. Experience in leading project/program evaluations
2. Experience in shelter projects/programs
3. Experience in post-disaster/humanitarian context
4. Professional experience in Haiti
5. Cost
| Job Email id: | arc.evlauation(at)gmail.com |
Post a Comment