Assessing the SMARTness of nutrition commitments

Share section

Introduction

This page describes the application of the SMARTness score, a system for assessing commitments registered through the Nutrition Accountability Framework (NAF). The score evaluates how Specific, Measurable, Achievable, Relevant and Time-bound (SMART) the registered commitments are, by assessing whether commitment makers have provided and clearly described all required information to meet the five dimensions of SMARTness, as defined in The SMARTness of nutrition commitments. The methodology described in Annex 1 is currently being used to assess all commitments made from December 2024 (the previous version of the SMARTness methodology was used from 2021 to late 2024). Meaningful changes in the 2024 version are the removal of additional stakeholder(s) of a joint commitment—due to challenges with ownership in progress tracking and inclusion of details regarding nutrition-specific and –sensitive investment within the Measurement dimension.

The score reveals areas where commitments can be clarified and can be used to spotlight exemplary cases of SMART commitments. Its main purpose is to support stakeholders to improve the formulation—particularly the trackability—of their commitments. It does not assess how impactful a commitment is, whether it has a global- or national-level focus, or whether it targets overall or specific populations.

Share section

Design of the SMARTness score

The application of SMART criteria in the formulation of nutrition commitments ensures that the type of commitments (such as enabling, policy or impact) and their expected outcomes are clear. Making commitments easier to classify and monitor also makes it possible to measure impact and demonstrate success. To facilitate the formulation and assessment of SMART commitments, we identified a set of “ingredients” (individual characteristics that comprise or serve as a proxy) for each of the five dimensions of SMART, as detailed in Annex 1. These ingredients provide the structure for the NAF's Commitment Registration Form; the assessment of their presence then produces the SMARTness score.

Scoring principles and scoring algorithm

The development of the SMARTness score follows two principles:

  • Use of a composite score: To measure each of the SMART dimensions – Specific, Measurable, Achievable, Relevant and Time-bound – ingredients are combined into a single score. The overall SMARTness score is the average of the five dimensions. Several nutrition initiatives use composite scores, such as the Global Hunger Index and the Nutrition Barometer[1] as well as other non-nutrition specific initiatives (e.g. the Multidimensional Poverty Index[2]). Composite scores are more easily interpretable and facilitate comparisons as they reduce the size of a set of ingredients without dropping the underlying information.[3]
  • Apply equal weights to dimensions, and ingredients within dimensions: SMARTness is evaluated by applying equal weights to each dimension. In addition, the ingredients within each dimension are equally weighted. Equal weights across dimensions and ingredients (within dimensions) produce normalised scores.

The SMARTness score is derived through three iterative steps:

  • Step 1: Review of each commitment and ingredient-level scoring as described in Annex 1.
  • Step 2: Calculation of the SMARTness score for each of the SMARTness dimensions and for the overall commitment using the ingredient scores assigned in Step 1.
  • Step 3: Clarification where possible with the commitment makers, to support in potentially improving score and overall trackability. All the ingredients specified in the online resource The SMARTness of nutrition commitments are assessed in the SMARTness score, except for:

Step 1: Reviewing and scoring at the ingredient level

Two assessors are involved in the scoring: One assessor performs the scoring and a second assessor checks the result to confirm agreement. Potential discrepancies are resolved through discussion among the assessors. In case of a disagreement, a third independent reviewer will review and assess the commitment. Annex 1 describes all ingredients that are reviewed for each dimension and defines the scores each ingredient might receive, along with how these are determined (e.g. examples and rationales). While the scores are ingredient-specific (as described in Annex 1) the principle of the scoring system is in common and as follows:

  • 1 point for responses that are complete, clear and in alignment with the content of related fields throughout the form. No clarifications are required from the commitment-maker.
  • 0.5 points for all responses requiring clarifications. We assign this score to any response when we cannot be certain of its content either in terms of clarity (e.g. acronyms that are not easily identifiable, terms with unclear definitions) or coherence (i.e. not in alignment with the content of other fields), which requires clarification from the commitment-maker. There is no further refinement in the score assigned to responses requiring clarifications (e.g. higher than 0.5 if a minor clarification is required and lower than 0.5 for a major one) as such a granular approach would not be cost-effective.
  • 0 points for responses that do not specify the content or are ‘missing’ (i.e. responses that do not specify the answer, such as ‘TBD’, to be defined; and ‘TBC’, to be confirmed) or not at all coherent (e.g. agreement across different fields).

Step 2: Computing the SMARTness score at the commitment level

To compute the score of each SMART dimension and the overall SMARTness of a commitment, we defined the weighting factors for each dimension and ingredient. Specifically:

  1. All five dimensions are considered essential for a commitment to be SMART, hence each of them (S, M, A, R and T) carries an equal weight of 0.2.
  2. Within each SMART dimension, ingredients are equally weighted. The individual weights are determined by the number of ingredients in each dimension.

The SMARTness score for the commitment n (SMARTn) is computed as:

SMARTn = (Sn × w) + (Mn × w) + (An × w) + (Rn × w) + (Tn × w)

Where the five components Sn, Mn, An, Rn and Tn capture the scores of the individual dimensions and w is equal to one-fifth (first-level weight).

Specific (Sn) includes six ingredients (S1 – S6), each weighting one-sixth (wS); Measurable (Mn) six ingredients (M1 – M6), each weighting one-sixth (wM); Achievable (An) four ingredients (A1 – A4), each weighting one-quarter (wA); Relevant (Rn) two ingredients (R1 – R2), each weighting one-half (wR); and Time-bound (Tn) two ingredients (T1 – T2), each weighting one-half (wT). The SMARTness score for the commitment ranges from 0 to 5, while each dimension score ranges between 0 and 1.

The scores of the individual dimensions for the commitment n, are computed as follows:

Sn = S1 × wS + S2 × wS + S3 × wS + S4 × wS + S5 × wS + S6 × wS

Mn = M1 × wM + M2 × wM + M3 × wM + M4 × wM + M5 × wM + M6 × wM

An = A1 × wA + A2 × wA + A3 × wA + A4 × wA

Rn = R1 × wR + R2 × wR

Tn = T1 × wT + T2 × wT

Step 3: Clarification

Assessors will compile and send any questions and clarifications that arise during the SMARTness assessment to each commitment maker using an Excel spreadsheet template. Commitment makers can submit clarifications to their commitments, which are then reviewed by the assessors who may ask for further clarifications as needed, and assist commitment makers in revising their commitments with an overall goal of improving commitment SMARTness and trackability. Once all questions have been clarified with a commitment maker, the assessors will adjust the commitment SMARTness score accordingly and confirm the finalized commitment with the commitment maker for publication on the website with a status of “Verified”. Commitment makers who do not respond for requests for clarification or who only partially respond to assessor clarifications will be published on the website with a status of “Unverified” and “Partially verified”, respectively.

Share section

Annex 1: Scoring per ingredient for each dimension

Share section

Assessing the 'Specific' aspect of commitments

Ingredient S1: Commitment making organisation name

First and third questions in Sign up Form

Compulsory – text (automatically checked for duplicates with other entities already in database); Compulsory – drop-down list of countries

1=Completed

  • A valid organisation name is provided and in agreement with what might be mentioned in other fields (e.g. commitment description).
  • The selection of the country where the commitment-making organisation is located is consistent with other fields.

0.5=Unclear/Not in alignment with other fields

  • A valid organisation name is provided but not in agreement with what is mentioned in other fields.
  • More than one organisation is listed.
  • The organisation name is not clear (e.g. acronyms not commonly used/easily identified).
  • The country listed is inconsistent with other fields (e.g. the country where commitment is focused on rather than the country where commitment-organisation is located).

0=Unspecified

  • Organisation name is not specified (e.g. ‘TBD’, to be defined; and ‘TBC’, to be confirmed)

Weight of ingredients: 1/6

Ingredient S2: Commitment making organisation type

Second question in Sign up Form

Compulsory – multiple choice

1=Completed

  • The response selected is the best match for the organisation and, when applicable, agrees with the content of the commitments (e.g. donor governments are committing to assisting other countries).

0.5=Unclear/Not in alignment with other fields

  • The response selected is not the best match for the organisation (e.g. a low-income country committing to national actions but identified as donor government)

Weight of ingredients: 1/6

Ingredient S3: Description of commitment

Q2.1; Q2.2

Compulsory – text; Compulsory – text

1=Completed

  • The commitment title (Q2.1) is clear; it is one commitment and is in alignment with the commitment description (Q2.2).
  • The commitment is clearly described in Q2.2; it is one commitment (not having included multiple commitments under one) in alignment with the key indicator and in agreement with what might be mentioned in other fields.

0.5=Unclear/Not in alignment with other fields

  • The commitment title or description is unclear (e.g. in terms of purpose).
  • The commitment title or description includes multiple commitments.
  • The commitment title and/or description are not in alignment with each other and/or the key indicator and/or what is mentioned in other fields.
  • The commitment title and/or description are clear and in agreement with other fields but require some clarifications (e.g. definition of chronic malnutrition, spelling out acronyms).
  • The commitment title and/or description are not nutrition-related (e.g. an increase physical activity without any linkage to nutrition outcomes) or the information provided is not sufficient or clear enough to decide whether nutrition-related (e.g. an increase the number of beneficiaries receiving cash transfers to address poverty).

0=Unspecified

  • The commitment title and description are not assessable (e.g. ‘TBD’, to be defined; and ‘TBC’, to be confirmed) or not at all coherent (e.g. agreement across different fields).

Weight of ingredients: 1/6

Ingredient S4: Type of commitment

Q1.2a

Compulsory – multiple choice

1=Completed

  • The commitment type (financial versus political) and type of political commitment selected (if applicable) is in alignment with the information provided in the commitment description and the key indicator.
  • All required information is available and clear for the commitment to be classified using the NAF Nutrition Action Classification System.

0.5=Unclear/Not in alignment with other fields

  • The commitment type (financial versus political) and type of political commitment selected (if applicable) is not in alignment with the information included provided in the commitment description and the key indicator.
  • The required information is available but not clear for the measurable goal to be classified using the NAF Nutrition Action Classification System (e.g. commitment description not in alignment with indicator).

Weight of ingredients: 1/6

Ingredient S5: Geographic coverage of commitment

Q2.3a; Q2.3b; Q2.3c; Q2.3d; Q2.3e; Q2.4

Compulsory – multiple choice; Compulsory (if selected region/multi-region) – multiple choice; Compulsory (if selected multi-country) – drop down; Compulsory (if selected country) – drop down; Compulsory (if selected sub-national) – text; Optional – text

1=Completed

  • The responses selected in Q2.3a, Q2.3b, Q2.3c, Q2.3d, and Q2.3e are in agreement with what might be mentioned in other fields (e.g. commitment description). A valid response is provided in 2.4 – a valid response in this case includes ‘None’ and ‘NA’ as this question might not apply to all commitments (i.e. not having any additional specifications to report).

0.5=Unclear/Not in alignment with other fields

  • The responses selected in Q2.3a, Q2.3b, Q2.3c, Q2.3d, and Q2.3e are not in agreement with what is mentioned in other fields.
  • The response selected in Q2.3a is ‘Regional’, but the region(s) are not specified in Q2.3b.
  • The response selected in Q2.3a is ‘Multi-country’, but the countries are not specified in Q2.3c.
  • The response selected in Q2.3a is ‘National’, but the country is not specified in Q2.3d.
  • The response selected in Q2.3a is ‘Sub-national’ or ‘Local’, but the sub-national or local area is not specified in Q2.3e or is unclear if specified.

Weight of ingredients: 1/6

Ingredient S6: Target population for commitment

Q2.5a; Q2.5b

Compulsory – multiple choice; Compulsory – multiple choice, text

1=Completed

  • The answer in Q2.5a is in agreement with what might be mentioned in other fields (e.g. commitment description).
  • The response provided in Q2.5a on whether there is a dimension of equity is in alignment with the population group characteristic(s) selected or described in Q2.5b.

0.5=Unclear/Not in alignment with other fields

  • The response selected in Q2.5a is not in agreement with what is mentioned in other fields (e.g. commitment description).
  • The response provided in Q2.5a on whether there is a dimension of equity is not in alignment with the population group characteristic(s) selected or described in Q2.5b.
  • The population group characteristic(s) described in Q2.5b are unclear.

Weight of ingredients: 1/6

Score range (dimension): 0–1

Weight of the dimension: 1

Share section

Assessing the 'Measurable' aspect of commitments

Ingredient M1: Primary indicator name and unit

Q3.1a; Q3.1b; Q3.1c; Q3.1d; Q3.1e

Compulsory – text; Compulsory – multiple choice; Compulsory (if currency) – drop down; Compulsory (if absolute number, not currency) – text; Compulsory (if binary) – text

1=Completed

  • A clearly defined indicator along with its unit of measurement (when applicable) is defined in Q3.1a and Q3.1b. The indicator must be further in agreement with the commitment description.
  • The response provided in Q3.1c, Q3.1d or Q3.1e is in alignment with the indicator type selected in Q3.1b.

0.5=Unclear/Not in alignment with other fields

  • The indicator and/or its unit of measurement is not clearly defined.
  • The indicator is not in agreement with the commitment description.
  • More than one indicator is listed for the commitment.
  • The response provided in Q3.1c, Q3.1d or Q3.1e is not in alignment with the indicator type selected in Q3.1b.

0=Unspecified

  • Indicator and/or its unit is not specified (e.g. ‘TBD’, to be defined; and ‘TBC’, to be confirmed)

Weight of ingredients: 1/6

Ingredient M2: Baseline level of indicator

3.2a

Compulsory – text

1=Completed

  • The baseline level of the indicator is clearly described using the same unit of measurement (when applicable) with the one provided in Q3.1a and Q3.1b and the value agrees with is mentioned in other fields (e.g. commitment description).

0.5=Unclear/Not in alignment with other fields

  • The baseline level is not/not clear if in alignment with indicator.
  • The unit of measurement is not in agreement with the one provided in Q3.1a and Q3.1b.
  • The baseline level value is not in agreement with what is mentioned in other fields (e.g. values mentioned in commitment description) or identical with target when it is suggested otherwise (e.g. commitment description mentions reduction or increase).

0=Unspecified

  • The baseline level is not specified (e.g. ‘TBD’, to be defined; and ‘TBC’, to be confirmed)

Weight of ingredients: 1/6

Ingredient M3: Baseline year of assessment

Q3.2b; Q3.2c

Compulsory – drop down, month; Compulsory – drop down, year

1=Completed

  • The response provided is identical or prior to the starting month and year of the commitment (Q3.5a; Q3.5b).

0.5=Unclear/Not in alignment with other fields

  • The response provided is not in agreement with might be mentioned in other fields.
  • The response provided is after the starting month and year of the commitment (Q3.5a; Q3.5b).

Weight of ingredients: 1/6

Ingredient M4: Target level of indicator

Q3.3; Q3.4

Compulsory – multiple choice; Compulsory – text

1=Completed

  • The target level of the indicator is clearly described in Q3.4 using the same unit of measurement (when applicable) with the one provided in Q3.1b and in Q3.2a; the response selected in Q3.3 is consistent with what is entered in Q3.4 and makes sense given the value of the baseline and direction of the baseline to target level (e.g. if the baseline is 100 and the commitment is to increase the level, the overall target entered should be more than 100); the direction of the baseline to target level is in alignment with the commitment description (e.g. the commitment is to reduce stunting, the baseline level is 20% and the target level is 15%); and the value agrees with what is mentioned in other fields (e.g. commitment description).

0.5=Unclear/Not in alignment with other fields

  • The target level is not/not clear if in alignment with indicator.
  • The unit of measurement is not in agreement with the one provided in Q3.1b and/or in Q3.2a.
  • The response value of the target and selection of ‘overall’ or ‘annual’ in Q3.3 does not make sense given the value of the baseline and direction of the baseline to target level (e.g. if the baseline is 100 and the commitment is to increase the level, the overall target entered should be more than 100).
  • The direction of the baseline to target level is not in alignment with the commitment description (e.g. the commitment is to reduce stunting, the baseline level is 20% and the target level is 25%).
  • The target-level value is not in agreement with what is mentioned in other fields (e.g. values mentioned in or commitment description) or identical with baseline when it is suggested otherwise (e.g. commitment description mentions reduction or increase).

0=Unspecified

  • The target level provided in Q3.4 is not assessable (e.g. ‘TBD’, to be defined; or ‘TBC’, to be confirmed).

Weight of ingredients: 1/6

Ingredient M5: Monitoring and evaluation plan

Q3.10a; Q3.10b; Q3.11a; Q3.11b

Compulsory – multiple choice; Compulsory – multiple choice; Compulsory – yes/no; Compulsory – multiple choice, text

1=Completed

  • A monitoring and evaluation plan for the indicator is confirmed by a selection of the first or second response option for Q3.10a, ideally with the frequency if applicable selected in Q3.10b, and in agreement with the indicator fields.
  • If there is a response to Q3.11a, there is an answer to Q3.11b, and the answer makes sense (e.g. if in Q3.11b they selected OECD DAC they are a donor organisation or donor government; if in Q3.11b they selected SUN JAA they are a SUN country; if in Q3.11b they selected ATNI they are a private sector entity).

0.5=Unclear/Not in alignment with other fields

  • Frequency of the monitoring and evaluation plan of the indicator is “Unknown” (Q3.10b)
  • There is a response to Q3.11b, but the answer does not make sense (e.g. if in Q3.11b they selected OECD DAC they are not a donor organisation or donor government; if in Q3.11b they selected SUN JAA but they are not a SUN country; if in Q3.11b they selected ATNI but they are not a private sector entity).

0=Unspecified

  • There is not known plan to assess the indicator (“Unknown” to Q3.10a).

Weight of ingredients: 1/6

Ingredient M6: Nutrition specific or sensitive investments (only for financial commitments)

Q3.8a; Q3.8b.; Q3.8c; Q3.8d; Q3.8e; Q3.9

Compulsory – multiple choice; Compulsory – numerical; Compulsory – numerical; Compulsory – numerical; Compulsory – numerical; Compulsory – multiple choice, text, year

1=Completed

  • If the commitment is not financial (Q1.2a), M6 is given a score of 1.
  • The numbers for nutrition-specific (Q3.8b / Q3.8d) and/or nutrition-sensitive (Q3.8c / Q3.8e) actions are consistent with the information in Q3.4 (primary indicator).
  • The information regarding allocation vs. disbursement (Q3.8a) is consistent with description of commitment (Q2.2) and other data (such as Q3.4, primary indicator).
  • The sectors selected (Q3.9) are consistent with the description of commitment (Q2.2).

0.5=Unclear/Not in alignment with other fields

  • The numbers for nutrition-specific (Q3.8b / Q3.8d) and/or nutrition-sensitive (Q3.8c / Q3.8e) actions are inconsistent with the information provided in the primary indicator and commitment description.
  • The information regarding allocation vs. disbursement (Q3.8a) is inconsistent with description of commitment (Q2.2) and other data (such as Q3.4, primary indicator).
  • The sector(s) selected (Q3.9) are unclear or inconsistent with the description of commitment (Q2.2).

Weight of ingredients: 1/6

Score range (dimension): 0–1

Weight of the dimension: 1

Share section

Assessing the 'Achievable' aspect of commitments

Ingredient A1: Total costs

Q4.1a

Compulsory – yes/no

1=Completed

  • The response provided is ‘Yes’ and is in agreement with what is mentioned in other fields.

0.5=Unclear/Not in alignment with other fields

  • The response provided is ‘Yes’ but is not in agreement with what is mentioned in other fields.

0=Unspecified

  • The response provided is ‘No’

Weight of the ingredients: 1/4

Ingredient A2: Funder

Q4.2

Compulsory – text

1=Completed

  • A valid name of the organisation(s) funding the commitment is provided and in agreement with what might be mentioned in other fields. That is for all known funders; if more are expected but unknown now, the response still receives 1.

0.5=Unclear/Not in alignment with other fields

  • A valid funding organisation name is provided but not in agreement with what is mentioned in other fields.
  • Not all funders are listed (e.g. they provide examples of some and/or they mention ‘and other donors’).
  • The organisation name is not clear (e.g. acronyms not commonly used and easily identified) or vague (UN agencies).

0=Unspecified

  • If ‘Other’ is selected, the funder name is unspecified (e.g. ‘TBD,’ to be defined; or ‘TBC,’ to be confirmed.

Weight of the ingredients: 1/4

Ingredient A3: Funding mechanism

Q4.3

Compulsory – multiple choice, text

1=Completed

  • The funding mechanism selected or described is in agreement with what is mentioned in other fields.

0.5=Unclear/Not in alignment with other fields

  • If ‘Other’ is selected, the funding mechanism is not clearly described (e.g. unclear acronyms, vague description).
  • The funding mechanism is not in agreement with what is mentioned in other fields (e.g. funder = UN agency and funding mechanism = government).
  • The funder is missing hence we cannot cross-check whether the funding mechanism is in alignment.

0=Unspecified

  • If ‘Other’ is selected, the funding mechanism name is unspecified (e.g. ‘TBD,’ to be defined; or ‘TBC,’ to be confirmed.

Weight of the ingredients: 1/4

Ingredient A4: Amount secured

Q4.1b

Compulsory – multiple choice

1=Completed

  • The response option ‘Funds fully secured’ or ‘Funds more than 50% secured’ is selected and is in agreement with what might be mentioned in other fields.

0.5=Unclear/Not in alignment with other fields

  • The response option ‘Funds less than 50% secured’ is selected.
  • Any response selected is not in agreement with what is mentioned in other fields.

Weight of the ingredients: 1/4

Score range (dimension): 0–1

Weight of the dimension: 1

Share section

Assessing the 'Relevant' aspect of commitments

Ingredient R1: Alignment with global nutrition targets

Q1.1a; Q1.1b

Compulsory – yes/no; Compulsory – multiple choice

1=Completed

  • The answer to Q1.1a is ‘Yes’. The global target(s) selected for Q1.1b is in alignment with the commitment description.

0.5=Unclear/Not in alignment with other fields

  • The response(s) selected for Q1.1b do not seem to be in alignment with the commitment description (e.g. the commitment is to reduce stunting and child overweight has been selected as global target).
  • The response(s) selected are not complete (e.g. only one global target has been selected while more apply) or many or all have been selected while they do not seem all to apply.

0=Unspecified/Not aligned with global nutrition targets

  • The response to Q1.1a is ‘No’.
  • The responses selected in Q1.1a and Q1.1b indicate that the commitment aligns with global nutrition targets; however, the commitment title and/or description are not nutrition-related (e.g. an increase physical activity without any linkage to nutrition outcomes) or the information provided is not sufficient or clear enough to decide whether nutrition-related (e.g. an increase the number of beneficiaries receiving cash transfers to address poverty).

Weight of ingredients: 1/2

Ingredient R2: Thematic area

Q1.3a; Q1.3b; Q1.3c; Q1.3d

Compulsory – yes/no; Compulsory – multiple choice; Compulsory – multiple choice; Compulsory – multiple choice

1=Completed

  • A response has been selected for Q1.3d. This is contingent on a ‘Yes’ response having been selected for Q1.3a, and ‘2025 Paris N4G Summit’ as one of the responses selected for Q1.3b. We are not validating the response against the content of the commitment; we are classifying all commitments using the NAF Nutrition Action Classification System.
  • If the commitment is not linked to the N4G Summit (in Q1.3b), this ingredient is not scored and R1 has an ingredient weight of 1.

0.5=Unclear/Not in alignment with other fields

0=Unspecified

Weight of ingredients: 1/2

Score range (dimension): 0–1

Weight of the dimension: 1

Share section

Assessing the 'Time-bound' aspect of commitments

Ingredient T1: Start date

Q3.5a; Q3.5b

Compulsory – drop-down menu, month; Compulsory – drop-down menu, year

1=Completed

  • The response provided is in agreement with what might be mentioned in other fields.

0.5=Unclear/Not in alignment with other fields

  • The response provided is not in agreement with what might be mentioned in other fields.
  • The starting date is before the baseline year of the indicator assessment.

Weight of ingredients: 1/2

Ingredient T2: End date

Q3.6a; Q3.6b

Compulsory – drop-down menu, month; Compulsory—drop-down menu, year

1=Completed

  • The response provided is in agreement with what might be mentioned in other fields and it is not before or identical to the start date (Q3.5a and Q3.5b).

0.5=Unclear/ Not in alignment with other fields

  • The response provided is not in agreement with what might be mentioned in other fields.
  • The response provided is before or identical to the start date (Q3.5a and Q3.5b).

Weight of ingredients: 1/2

Score range (dimension): 0–1

Weight of the dimension: 1

As of 2024, each commitment constitutes a separate submission rather than a submission constituting several bundled goals. We assess each commitment for SMARTness.

Share section

Annex 2: Pilot testing the SMARTness score

We piloted the SMARTness score on a random sample of 85 commitments (147 commitment goals) from the first version of the questionnaire rolled out in 2021. Our sample corresponded to around 20% of the commitments received through the NAF up to mid-February 2022. The SMARTness assessment pilot was performed in two rounds, in which two researchers reviewed the commitments independently and in duplicate. After reviewing 45 commitments (84 commitment goals), the two reviewers’ results were compared and goals were ranked based on the number of differences between the reviewers. Differences were identified between SMART dimensions and between ingredients. This step helped to identify problematic cases, the limitations of the preliminary scoring system, and any post-processing errors. The following step was re-assessing the selected subset, focusing on goals with the highest number of differences. The re-evaluation was performed by the same reviewers working together. This step led to a strengthening of the scoring system and the establishment of clearer scoring criteria. The step was also used to further quality assure the post-processing methods. The second round of scoring (40 commitments and 63 commitment goals) aimed to evaluate the improvement in the scoring system and identify any additional adjustments needed in the scoring protocol.

Robustness checks cover three dimensions:

  • Reliability (consistency of the score across time and different researchers): This is achieved by the combination of comprehensive and strict scoring guidelines with a scoring protocol developed with and by the reviewers and tested over multiple pilots.
  • Validity (a measure of the accuracy of the score): As there is not a gold standard (or benchmark), validity is assessed internally. The methodological approach was reviewed by the lead authors and no major inconsistencies or flaws have been identified (face validity). The content validity is embedded in the methodology by constructing the score with all the compulsory elements in the survey. We aim to assess the construct validity by testing group differences and estimating correlation matrices.
  • Sensitivity (how the SMARTness score is susceptible to changes to the data): This is tested by randomly introducing small changes to the original dataset and statistically comparing the different SMARTness scores generated by the two datasets.

Testing the scoring methodology

At the end of the first phase of the pilot (a review of 10% of commitments), the scoring disagreement between the two reviewers ranged from 5.8% to 24.3% across the ingredients, with a mean value of 12.4%. Following the score, the two researchers systematically reviewed and recorded differences in their scores. The issues identified were discussed with the lead IEG lead authors to reach an agreement, and the protocol was updated to address cases that caused confusion (e.g. specifying examples). Further refining the score assigned to unclear responses (i.e. those requiring clarification from commitment-makers) was also discussed. We considered identifying ingredients for which scores could be further refined (e.g. four- or five-value scoring instead of the current three-value approach), so that responses that required only minor clarifications would receive a higher score than those requiring major clarifications. However, we refrained from doing this given the complexity of defining minor versus major clarifications and the subsequent high risk of introducing wider disagreement between the reviewers. As any minor or major unclear point will require clarification from commitment-makers, the same score (i.e. 0.5) will be assigned to any ingredient whose response is not complete and clear. Finally, the two reviewers’ assigned scores were amended based on the updated protocol.

At the end of the second phase (a review of an additional 10% of commitments), the scoring disagreement between the two reviewers ranged from 1.7% to 45.6% across ingredients, with a mean value of 5.3%. Inconsistencies across one ingredient (R2) stood out and these were due to changes to the criteria in the protocol following the first phase. Following further discussions between the reviewers and the lead IEG authors, we updated the protocol to clarify the interpretation of the scoring for R2, reverting to the original scoring method for this ingredient. The scoring disagreement for the remaining dimensions (S, M, A and T) ranged from 1.7% to 7.8%, and the protocol was refined further to include additional examples for these dimensions from the second phase. It was agreed that one reviewer would score the remaining commitment goals, with the second reviewer consulted for any commitments that had potential for ambiguity.

SMARTness score pilot results

Tables 1a and 1b report the SMARTness score summary statistics, at commitment and goal levels. The S, M, R and T dimensions have the highest scores. The Achievable dimension has the lowest score of 0.66 and 0.65, at commitment and goal level respectively. Table 2 breaks down each dimension by ingredients and reports the percentage of goals that do not achieve the full score by each ingredient. Goals that will require clarifications from stakeholders range between 3–12% within the Specific dimension, 3–27% for Measurable, 25–68% for Achievable, 14–25% for Relevant, and 0–3% for Time-bound.

Table 1a: SMARTness score summary statistics, by commitment

Dimensions Mean SD Min Max
S 0.96 0.02 0.79 1
M 0.87 0.08 0.10 1
A 0.66 0.09 0.00 1
R 0.89 0.02 0.50 1
T 0.99 0.01 0.75 1
SMART 4.37 0.10 3.11 5

Commitments (N=85)

Table 1b: SMARTness score summary statistics, by goal

Dimensions Mean SD Min Max
S 0.96 0.06 0.79 1
M 0.87 0.19 0.10 1
A 0.65 0.27 0.00 1
R 0.90 0.14 0.50 1
T 0.99 0.05 0.75 1
SMART 4.37 0.42 3.11 1

Goals (N=147)

Table 2: Summary statistics of scoring for each ingredient, by goal

Ingredients Mean SD Min Max % of < 1
S1 0.99 0.08 0.5 1 2.7
S2 0.96 0.13 0.5 1 7.5
S3 0.95 0.15 0.5 1 9.5
S4 0.94 0.16 0.5 1 11.6
S5 0.97 0.12 0.5 1 6.1
S6 0.95 0.15 0.5 1 9.5
S7 0.97 0.11 0.5 1 5.4
M1 0.90 0.22 0 1 17
M2 0.85 0.30 0 1 21.8
M3 0.96 0.14 0.5 1 8.1
M4 0.82 0.31 0 1 2.8
M5 0.80 0.35 0 1 27.2
A1 0.37 0.46 0 1 68
A2 0.72 0.37 0 1 40.8
A3 0.74 0.35 0 1 40.1
A4 0.78 0.40 0 1 24.5
R1 0.93 0.17 0.5 1 14.3
R2 0.87 0.24 0 1 24.5
T1 1.00 0.00 1 1 0
T2 0.98 0.09 0.5 1 3.4

Note: Refer to the Appendix for the description of the ingredients.

Last updated: 10 April 2025.

Jump to section Jump to section

Footnotes

  1. Save the Children and World Vision International. The nutrition barometer – Gauging national responses to undernutrition. 2012.

    The Global Hunger Index. 2021; published online October). https://www.globalhungerindex.org/download/all.html

    Pangaribowo, E. H., Gerber, N., Torero, M. (2013) Food and nutrition security indicators: A review. www.zef.de.

    Return to source text

  2. Alkire, S., Santos, M. E. Measuring Acute Poverty in the Developing World: Robustness and Scope of the Multidimensional Poverty Index. World Development 2014; 59: 251–274

    Return to source text

  3. OECD. Handbook on Constructing Composite Indicators: Methodology and User Guide. 2008.

    Return to source text