preparation of guidance material on Impact Evaluation of normative and institutional support work
UN Evaluation Group (UNEG) Impact Evaluation Task Force
Terms of reference for the preparation of guidance material on Impact Evaluation of normative and institutional support work
1 Background
1. In March 2010, the UNEG Task Force (TF) on Impact Evaluation submitted to UNEG members a
concept note based on a two-year process of consultation and stock-taking of the work by UNEG
members on Impact Evaluation (IE), including the assessment of capacity development and guidance
needs and expectations from members on IE from UNEG. The concept note recommended that a
guidance document be prepared by UNEG, that would cover several aspects and issues related to IE in
the UN system.
2. UNEG Annual General Meeting asked the TF to deliver „guidance materials‟ that would be
harmonized and coherent but more flexible for production and diffusion. In September 2010 the TF
agreed to focus on three main areas: a) The Role of IE in the overall M&E system; b) Attribution or
contribution analysis in multi-stakeholder interventions; and c) IE of UN normative and institutional
support work. These TOR specify the scope of work for producing the guidance on “IE of UN
normative and institutional support”.
3. Among other issues, the Concept Note defined „normative and institutional support work‟ as : “all
aspects of the UN‟s efforts in producing, promoting and helping to implement and monitor UN
conventions, protocols, recommendations and declarations, as well as its contributions, to related
national laws, policies, strategies, action plans, programmes, rules, regulations, guidelines, standards,
codes of conduct, manuals, and good manufacturing practices, that give life to international norms at
the country level. Other work under this heading includes institutional support, knowledge production,
capacity development, etc.”
2 Purpose of the assignment and tasks
4. The assignment aims at the preparation, discussion and finalization of a Guidance material on IE of
Normative and Institutional Support Work by the UN, based on the definition above and on the
qualifiers of IE for the UN described in the attached outline. The Guidance Material will help UNEG
members and the evaluation community at large, to conceptualize, design, plan and conduct impact
evaluations of the UN normative and institutional support work.
5. The Guidance Material should be a self-standing document, closely harmonized with all other
Guidance Materials to be produced by the TF, in the format of a users-friendly handbook for
consultation by evaluation managers and evaluators. The target audience is primarily UNEG members,
although the evaluation community at large will also benefit from it.
6. In order to accomplish his/her mandate, the consultant will:
I. take stock of the material produced by the TF since 2009, in particular the Concept
Note prepared in early 2010 and its Annexes and the Outline for the Guidance Material;
II. submit to the TF suggestions for improving the outline, if required;
III. prepare a draft guidance document and submit it to the TF for comments and
suggestions;
IV. finalize the guidance document taking into account the TF feedback and suggestions;
7. Consultation with TF members will be possible by email at any time during the drafting process; as
required, meetings face to face or through teleconference will also be organized as needed. The TF
may consider setting-up a peer-review process involving non TF members to revise the draft guidance
document.
ToR consultant Guidance material on IE of UN normative and institutional support work, January 2011
2
3 Timing, logistic arrangements and deliverables
8. The consultant will work from his/her home station and will interact with the TF members by
teleconference and through email exchange.
9. The consultancy will be for 20 working days. It will be conducted in the period March-April 2011.
10. The consultant will send to the TF co-chairs:
i. proposal for improving the tentative outline (see next page) within 3 working days from
the start of the contract;
ii. the draft guidance material within 12 working days from receiving a feedback from the
TF on the proposed changes to the outline;
iii. the final guidance material within 5 working days from receipt of the feed-back from the
TF and the peer-review, if applicable.
11. Payment will be finalized upon approval by the IE TF co-chairs of the final version of the
Guidance material and related communication to the UNEG secretariat.
4 Qualifications of the consultant
12. The consultant must have in-depth knowledge of IE related issues, including the debate developed
over the last few years within the development community (NONIE, 3IE, etc.) and experience in the
evaluation of normative and institutional support work within the UN. Practical experience in the
conduct of IE is also required. Good writing skills in English are a must.
Annex: Outline for Guidance Material on Impact Evaluation of normative and institutional support
work
ToR consultant Guidance material on IE of UN normative and institutional support work, January 2011
3
UNEG Impact Evaluation Task Force
Guidance Material on Impact Evaluation of normative and institutional
support work
Purpose of a Guidance Material on Impact Evaluation of normative and institutional
support work
1. The Guidance Material has the purpose of helping UNEG members and the evaluation
community at large, to conceptualize, design, plan and conduct impact evaluations of the UN
normative and institutional support work. The Guidance Material should be a self-standing
document, closely harmonized with all other Guidance Materials to be produced by the TF, in
the format of a users-friendly handbook for consultation by evaluation managers and evaluators.
Tentative Outline
5 Definition of impact evaluation – and qualifiers for IE in the UN context
2. A preponderance of UN agencies accept and apply the DAC definition of impact as a basis for
defining impact evaluations and that no alternative definition emerges as a preferred option.
Slide 1. OECD/DAC definition of impact
“Positive and negative, primary and secondary long-term effects produced by a development
intervention, directly or indirectly, intended or unintended.”
3. There are evaluation implications stemming from the DAC definition that are linked to the
phrase „produced by a development intervention’. There is an expectation that the methodology
applied will address the issue of attribution, both in terms of causality and degree. Within this,
the concept of the counterfactual is recognized as an important component for distinguishing an
impact evaluation.
4. Overall, the main points above are consistent with impact evaluation treatment by most UNEG
members but with several important qualifiers:
The importance of mixed methods and triangulation, through which the „why‟ and
„how‟ of an intervention can be explored. It is expected that mixed methods also
facilitate determining the form and extent of indirect and secondary effects.
UN agency-specific mandates and modes of engagement ground the logic of
interventions and shape the mix of evaluation methods to be applied.
The development and design of methodologies for impact evaluation of UN normative
action and related support programmes are of particular interest to the UN Evaluation
Group (UNEG)
The impact of humanitarian operations is primarily short term in nature and diverges
somewhat from the DAC definition. What implication does this have for impact
evaluation designs and -methodologies of normative and institutional support work?
Evaluating the impact of policies and programmes aimed at supporting the
achievement of the MDGs is of special interest to UNEG members.
Impact evaluation within a UN context is ideally embedded within the broader
discussion on monitoring and evaluation systems and evaluation capacity building.
ToR consultant Guidance material on IE of UN normative and institutional support work, January 2011
4
6 Description and definition of ‘normative and institutional support work’
5. The IETF Concept Note presented at the AGM 2010 defined this as: “all aspects of the UN‟s
efforts in producing, promoting and helping to implement and monitor UN conventions,
protocols, recommendations and declarations, as well as its contributions, to related national
laws, policies, strategies, action plans, programmes, rules, regulations, guidelines, standards,
codes of conduct, manuals, and good manufacturing practices, that give life to international
norms at the country level. Other work under this heading includes institutional support,
knowledge production, capacity development, etc.” The Guidance should elaborate on this
concept/definition and fine-tune/streamline it as appropriate.
6. A key point for discussion will be how the guidance material should also include the IE of work
on Human Rights and Gender Equality. It is important that the work done by the UNEG Human
Rights and Gender Equality (HR GE) TF does not get replicated. The guidance document and
handbook produced by the HR GE TF, to be finalised by early 2011, discuss in detail how to
mainstream HR and GE concepts in all kind of evaluations. Thus, this Guidance material should
focus exclusively on the specific issues attached to the evaluation of the impact of HR GE
concepts in normative and institutional support work. Key principles in integrating Human
Rights and Gender Equality in the evaluation process include inclusion, participation, and fair
power relations. The Guidance material should look at how to combine the Human Rights and
Gender Equality principles with Impact Evaluation designs and methods. How will evaluation
design, data collection methods and -strategy be affected? How may integrating Human Rights
and Gender Equality in the impact evaluation process affect principles of credibility, precision,
internal and external validity etc.?
7. For ease of reference, the IETF Concept Note had taken a position in this respect: “A distinct,
but related area for which impact evaluation methods need further elaboration, is that of rightsbased
approaches; notably in the fields of Human Rights and Gender. These concepts and
practices are in principle mainstreamed into all UN activities, which implies some difficulty in
producing analytically discreet cause and effect chains, which can be evaluated. However, as
with the areas previously identified for guidance, there is a body of available documentation,
which can form the basis of further development, tailored to the requirements of impact
evaluation. For example, the Task Force on Gender Equality and Human Rights has recently
completed its report on evaluating results in these areas, which includes impacts. This and other
documents will be analysed and the most effective and useful approaches collated and
summarised. There is also material covering these topics at outcome level, some of which can
be adapted for use in the context of IE and this will be another stream of this component of the
guidance. In addition, an assessment will be made of how generic approaches, such as Theory-
Based Impact Evaluation, impact pathways and contribution analysis can contribute to
strengthening approaches in these normative areas. The overlap between normative and rightsbased
work is an area which may prove particularly interesting and challenging for impact
evaluation. Many of the concepts concerning Human Rights and Gender, which are now
operationalised through mainstreaming, derive from earlier UN normative work and in turn feed
back into it as international understanding of the underlying issues develops. The contribution
of the UN to these complex processes of global social change is a challenging topic, which
could prove worthy of impact evaluation.”
7 Where and how to locate the impact of normative and institutional support
work
8. A key contribution of the Guidance will be a discussion about where and how the expected
impact of UN normative work should materialize. Issues for discussion will include clarification
between outcome and impact at institutional level, immediate and ultimate beneficiaries, level
of responsibility in achieving impacts beyond the institution, etc.
ToR consultant Guidance material on IE of UN normative and institutional support work, January 2011
5
8 Methodological framework and tools for IE of normative and institutional
support work
9. Issues to discuss will include potential evaluation designs, the availability of information and
baseline data and links with monitoring; use of the theory of change; identification of the
counterfactual; etc. Reference will be made to the forthcoming guidance on “Attribution or
contribution analysis”, and is it expected that this guidance will complement the guidance on
“Attribution or contribution analysis” with examples and details relevant to normative and
institutional support work.
9 Timing, human and financial resources and organization of IE of normative and
institutional support work
10. This should include anything specific to IE of normative work that is not captured in the
specific Guidance Material on “Role of Impact Evaluation in the overall monitoring-evaluation
system, including timing, management and conduct”
10 Examples of IE of normative and institutional support work by UNEG
members
11. A few examples exist already; the TF should agree on a format for presenting these as case
studies/examples.
The consultancy will have a duration of 20 working days during the period of February- April 2011, with a draft guidance to be presented to UNEG in mid March.
Please manifest your interest only if available for carrying out the work in the foreseen period. CVs should be submitted by 7th February to Michael.Spilsbury@unep.org ; Tullia.Aiazzi@fao.org and Caspar.Merkle@unwomen.org .
Terms of reference for the preparation of guidance material on Impact Evaluation of normative and institutional support work
1 Background
1. In March 2010, the UNEG Task Force (TF) on Impact Evaluation submitted to UNEG members a
concept note based on a two-year process of consultation and stock-taking of the work by UNEG
members on Impact Evaluation (IE), including the assessment of capacity development and guidance
needs and expectations from members on IE from UNEG. The concept note recommended that a
guidance document be prepared by UNEG, that would cover several aspects and issues related to IE in
the UN system.
2. UNEG Annual General Meeting asked the TF to deliver „guidance materials‟ that would be
harmonized and coherent but more flexible for production and diffusion. In September 2010 the TF
agreed to focus on three main areas: a) The Role of IE in the overall M&E system; b) Attribution or
contribution analysis in multi-stakeholder interventions; and c) IE of UN normative and institutional
support work. These TOR specify the scope of work for producing the guidance on “IE of UN
normative and institutional support”.
3. Among other issues, the Concept Note defined „normative and institutional support work‟ as : “all
aspects of the UN‟s efforts in producing, promoting and helping to implement and monitor UN
conventions, protocols, recommendations and declarations, as well as its contributions, to related
national laws, policies, strategies, action plans, programmes, rules, regulations, guidelines, standards,
codes of conduct, manuals, and good manufacturing practices, that give life to international norms at
the country level. Other work under this heading includes institutional support, knowledge production,
capacity development, etc.”
2 Purpose of the assignment and tasks
4. The assignment aims at the preparation, discussion and finalization of a Guidance material on IE of
Normative and Institutional Support Work by the UN, based on the definition above and on the
qualifiers of IE for the UN described in the attached outline. The Guidance Material will help UNEG
members and the evaluation community at large, to conceptualize, design, plan and conduct impact
evaluations of the UN normative and institutional support work.
5. The Guidance Material should be a self-standing document, closely harmonized with all other
Guidance Materials to be produced by the TF, in the format of a users-friendly handbook for
consultation by evaluation managers and evaluators. The target audience is primarily UNEG members,
although the evaluation community at large will also benefit from it.
6. In order to accomplish his/her mandate, the consultant will:
I. take stock of the material produced by the TF since 2009, in particular the Concept
Note prepared in early 2010 and its Annexes and the Outline for the Guidance Material;
II. submit to the TF suggestions for improving the outline, if required;
III. prepare a draft guidance document and submit it to the TF for comments and
suggestions;
IV. finalize the guidance document taking into account the TF feedback and suggestions;
7. Consultation with TF members will be possible by email at any time during the drafting process; as
required, meetings face to face or through teleconference will also be organized as needed. The TF
may consider setting-up a peer-review process involving non TF members to revise the draft guidance
document.
ToR consultant Guidance material on IE of UN normative and institutional support work, January 2011
2
3 Timing, logistic arrangements and deliverables
8. The consultant will work from his/her home station and will interact with the TF members by
teleconference and through email exchange.
9. The consultancy will be for 20 working days. It will be conducted in the period March-April 2011.
10. The consultant will send to the TF co-chairs:
i. proposal for improving the tentative outline (see next page) within 3 working days from
the start of the contract;
ii. the draft guidance material within 12 working days from receiving a feedback from the
TF on the proposed changes to the outline;
iii. the final guidance material within 5 working days from receipt of the feed-back from the
TF and the peer-review, if applicable.
11. Payment will be finalized upon approval by the IE TF co-chairs of the final version of the
Guidance material and related communication to the UNEG secretariat.
4 Qualifications of the consultant
12. The consultant must have in-depth knowledge of IE related issues, including the debate developed
over the last few years within the development community (NONIE, 3IE, etc.) and experience in the
evaluation of normative and institutional support work within the UN. Practical experience in the
conduct of IE is also required. Good writing skills in English are a must.
Annex: Outline for Guidance Material on Impact Evaluation of normative and institutional support
work
ToR consultant Guidance material on IE of UN normative and institutional support work, January 2011
3
UNEG Impact Evaluation Task Force
Guidance Material on Impact Evaluation of normative and institutional
support work
Purpose of a Guidance Material on Impact Evaluation of normative and institutional
support work
1. The Guidance Material has the purpose of helping UNEG members and the evaluation
community at large, to conceptualize, design, plan and conduct impact evaluations of the UN
normative and institutional support work. The Guidance Material should be a self-standing
document, closely harmonized with all other Guidance Materials to be produced by the TF, in
the format of a users-friendly handbook for consultation by evaluation managers and evaluators.
Tentative Outline
5 Definition of impact evaluation – and qualifiers for IE in the UN context
2. A preponderance of UN agencies accept and apply the DAC definition of impact as a basis for
defining impact evaluations and that no alternative definition emerges as a preferred option.
Slide 1. OECD/DAC definition of impact
“Positive and negative, primary and secondary long-term effects produced by a development
intervention, directly or indirectly, intended or unintended.”
3. There are evaluation implications stemming from the DAC definition that are linked to the
phrase „produced by a development intervention’. There is an expectation that the methodology
applied will address the issue of attribution, both in terms of causality and degree. Within this,
the concept of the counterfactual is recognized as an important component for distinguishing an
impact evaluation.
4. Overall, the main points above are consistent with impact evaluation treatment by most UNEG
members but with several important qualifiers:
The importance of mixed methods and triangulation, through which the „why‟ and
„how‟ of an intervention can be explored. It is expected that mixed methods also
facilitate determining the form and extent of indirect and secondary effects.
UN agency-specific mandates and modes of engagement ground the logic of
interventions and shape the mix of evaluation methods to be applied.
The development and design of methodologies for impact evaluation of UN normative
action and related support programmes are of particular interest to the UN Evaluation
Group (UNEG)
The impact of humanitarian operations is primarily short term in nature and diverges
somewhat from the DAC definition. What implication does this have for impact
evaluation designs and -methodologies of normative and institutional support work?
Evaluating the impact of policies and programmes aimed at supporting the
achievement of the MDGs is of special interest to UNEG members.
Impact evaluation within a UN context is ideally embedded within the broader
discussion on monitoring and evaluation systems and evaluation capacity building.
ToR consultant Guidance material on IE of UN normative and institutional support work, January 2011
4
6 Description and definition of ‘normative and institutional support work’
5. The IETF Concept Note presented at the AGM 2010 defined this as: “all aspects of the UN‟s
efforts in producing, promoting and helping to implement and monitor UN conventions,
protocols, recommendations and declarations, as well as its contributions, to related national
laws, policies, strategies, action plans, programmes, rules, regulations, guidelines, standards,
codes of conduct, manuals, and good manufacturing practices, that give life to international
norms at the country level. Other work under this heading includes institutional support,
knowledge production, capacity development, etc.” The Guidance should elaborate on this
concept/definition and fine-tune/streamline it as appropriate.
6. A key point for discussion will be how the guidance material should also include the IE of work
on Human Rights and Gender Equality. It is important that the work done by the UNEG Human
Rights and Gender Equality (HR GE) TF does not get replicated. The guidance document and
handbook produced by the HR GE TF, to be finalised by early 2011, discuss in detail how to
mainstream HR and GE concepts in all kind of evaluations. Thus, this Guidance material should
focus exclusively on the specific issues attached to the evaluation of the impact of HR GE
concepts in normative and institutional support work. Key principles in integrating Human
Rights and Gender Equality in the evaluation process include inclusion, participation, and fair
power relations. The Guidance material should look at how to combine the Human Rights and
Gender Equality principles with Impact Evaluation designs and methods. How will evaluation
design, data collection methods and -strategy be affected? How may integrating Human Rights
and Gender Equality in the impact evaluation process affect principles of credibility, precision,
internal and external validity etc.?
7. For ease of reference, the IETF Concept Note had taken a position in this respect: “A distinct,
but related area for which impact evaluation methods need further elaboration, is that of rightsbased
approaches; notably in the fields of Human Rights and Gender. These concepts and
practices are in principle mainstreamed into all UN activities, which implies some difficulty in
producing analytically discreet cause and effect chains, which can be evaluated. However, as
with the areas previously identified for guidance, there is a body of available documentation,
which can form the basis of further development, tailored to the requirements of impact
evaluation. For example, the Task Force on Gender Equality and Human Rights has recently
completed its report on evaluating results in these areas, which includes impacts. This and other
documents will be analysed and the most effective and useful approaches collated and
summarised. There is also material covering these topics at outcome level, some of which can
be adapted for use in the context of IE and this will be another stream of this component of the
guidance. In addition, an assessment will be made of how generic approaches, such as Theory-
Based Impact Evaluation, impact pathways and contribution analysis can contribute to
strengthening approaches in these normative areas. The overlap between normative and rightsbased
work is an area which may prove particularly interesting and challenging for impact
evaluation. Many of the concepts concerning Human Rights and Gender, which are now
operationalised through mainstreaming, derive from earlier UN normative work and in turn feed
back into it as international understanding of the underlying issues develops. The contribution
of the UN to these complex processes of global social change is a challenging topic, which
could prove worthy of impact evaluation.”
7 Where and how to locate the impact of normative and institutional support
work
8. A key contribution of the Guidance will be a discussion about where and how the expected
impact of UN normative work should materialize. Issues for discussion will include clarification
between outcome and impact at institutional level, immediate and ultimate beneficiaries, level
of responsibility in achieving impacts beyond the institution, etc.
ToR consultant Guidance material on IE of UN normative and institutional support work, January 2011
5
8 Methodological framework and tools for IE of normative and institutional
support work
9. Issues to discuss will include potential evaluation designs, the availability of information and
baseline data and links with monitoring; use of the theory of change; identification of the
counterfactual; etc. Reference will be made to the forthcoming guidance on “Attribution or
contribution analysis”, and is it expected that this guidance will complement the guidance on
“Attribution or contribution analysis” with examples and details relevant to normative and
institutional support work.
9 Timing, human and financial resources and organization of IE of normative and
institutional support work
10. This should include anything specific to IE of normative work that is not captured in the
specific Guidance Material on “Role of Impact Evaluation in the overall monitoring-evaluation
system, including timing, management and conduct”
10 Examples of IE of normative and institutional support work by UNEG
members
11. A few examples exist already; the TF should agree on a format for presenting these as case
studies/examples.
The consultancy will have a duration of 20 working days during the period of February- April 2011, with a draft guidance to be presented to UNEG in mid March.
Please manifest your interest only if available for carrying out the work in the foreseen period. CVs should be submitted by 7th February to Michael.Spilsbury@unep.org ; Tullia.Aiazzi@fao.org and Caspar.Merkle@unwomen.org .
Post a Comment