Let's Stop Trying to Quantify Household Vulnerability: The Problem With Simple Scales for Targeting and Evaluating Economic Strengthening Programs

Côte d'Ivoire Assessment

We used a rigorous psychometric approach to develop an economic vulnerability scale in Côte d'Ivoire based on the Sustainable Livelihoods approach.⁷ We conducted a cross-sectional survey of 3,749 households in 5 health regions of Côte d'Ivoire in an attempt to develop a measure of USAID/Côte d'Ivoire's broad definition of vulnerability as the degree of inability of households to provide for the health, education, and nutritional needs of household members with and without HIV infection to mitigate the economic and health impact of HIV, cope with infection, and reduce their risk for acquiring HIV (for those without HIV).

The instrument was derived from existing survey tools to assess a comprehensive set of asset “capitals,” including financial, social, natural, financial, and physical capital, country-specific indicators from national household surveys, and formative qualitative data. In the absence of an existing validated measure of the broad definition of vulnerability that we attempted to measure, we used validated food security and poverty measures, which we expected to correlate with economic vulnerability, to test the validity of our tool. We used the PPI to measure poverty and the Food Consumption Score (FCS) and Reduced Coping Strategies Index (rCSI) to measure food security. Additionally, data collectors trained on the PEPFAR classifications of household status provided their own subjective assessments of each household as a fourth validation measure. Indicators were reviewed by a stakeholder advisory group and the survey was pretested and refined before data were collected.

Using principal component analysis (PCA), we attempted to identify sets of correlated vulnerabilities and derive a small number of composite scores (components) to create an index for targeting vulnerable households for enrollment into an economic strengthening program and matching them to appropriate interventions. We compared the mean component scores from the PCA to data collector classifications of household economic vulnerability using ANOVA. We also classified the households into 4 different vulnerability categories using distributions derived from a participatory vulnerability ranking activity.

We selected 65 of 98 variables in the PCA based on completeness and variability. Based on scree plot data, we retained 4 components but these 4 components explained only 21% of the total variance among the items. This means that our measure explained only 21% of what distinguished households as more or less vulnerable based on the variables we included in our measure. Although there is no minimum accepted threshold, a rule of thumb proposed for multivariate analysis in the social sciences is that a solution should explain roughly 60% of variance to be considered satisfactory.⁸ The first (and largest) component, which captured food security and wealth measures, explained only 8% of the variance. In other words, households classified at the same level of vulnerability could have vastly different sets of responses to the questions in our survey, suggesting that very different sets of variables could lead to the same economic status outcomes.

Overall, we were unable to reduce the variables for our broad construct of HIV-related economic vulnerability down to a scale. We concluded that there were many pathways to household economic vulnerability in our sample and that appropriate programmatic responses should be tailored to individual household needs using a case management approach rather than an economic classification based on a scale.

Uganda Assessment

ASPIRES' Family Care activity is managing 2 learning projects in Uganda to build evidence on how economic strengthening and social support can prevent unnecessary family–child separation and reintegrate separated families: Family Resilience (FARE), implemented by AVSI, and Economic Strengthening to Keep and Reintegrate Children into Families (ESFAM), implemented by ChildFund. In 2016, we worked with implementers to develop a short tool to assess the economic status of program beneficiaries for matching households to interventions and M&E purposes.

The HVAT is the basis for the tools used by both projects. Both programs modified the government tool to add indicators relevant to family–child separation and economic vulnerability: AVSI developing the FARE Household Vulnerability Assessment Tool (FARE HVAT) and ChildFund adapting the Family Status Vulnerability Index (FSVI) that it had developed for a related project. The scoring of these tools reflects these changes.

At the same time, Family Care developed a 9-item “Simple Economic Strengthening Tool” to generate common household economic classifications that could be used for cross-project comparison (Table 1). Family Care selected among existing economic indicators that were consistent across the FSVI and HVAT tools and that corresponded with the PEPFAR categories and LIFT (Livelihoods and Food Security Technical Assistance) livelihoods framework.⁹ An analysis of these frameworks showed 5 main domains of economic status:

1. Ability to pay for basic needs

2. Consistency/volatility of income

3. Availability of liquid assets and savings

4. Food security

5. Availability of assets to respond to shocks

To ensure that all domains were covered, Family Care had requested the 2 projects to add or harmonize some indicators across the FSVI and HVAT. A total of 9 questions were selected for the Simple Economic Strengthening Tool. Each was scored on a scale of 0–4, in accordance with the scoring systems already in place for the HVAT and FSVI. Individual question scores were used to generate a classification based on the PEPFAR descriptions of economic status (destitute, struggling, prepared to grow, or not vulnerable) for each domain. The 5 domain classifications were then used to determine the household classification based on an algorithm developed by Family Care. This complex scoring method was meant to account for the vulnerability dynamics implied by different combinations of scores across questions and domains.

Family Care then analyzed baseline data for 114 FARE target households using the Simple Economic Strengthening Tool. Due to poor correlation with the PPI, Family Care's selected validation measure, the scoring method was then revised based on implementer input so that each question was scored on a scale of 0–3, with scores corresponding to the PEPFAR categories as below:

• 0 points=not vulnerable

• 1 point=prepared to grow

• 2 points=struggling to make ends meet

• 3 points=destitute

Each question was weighted evenly, and final classifications were based on an unweighted average across all scores. The question on shocks originally had multiple responses with the same point value but was revised so that questions that previously had the same value now had different values. As such, Family Care was unable to convert the raw scores for this domain to the new scoring scheme and this domain was dropped from the revised tool. Four of the original 5 domains were retained, with 2 questions each, so domain scores were evenly weighted. FARE used the Simple Economic Strengthening Tool scoring method during baseline data collection among 114 households.

Because poverty likelihood, as measured using the PPI, was expected to vary in the same direction as economic vulnerability as measured by the Simple Economic Strengthening Tool, the PPI was conducted simultaneously with all households as a validation measure. Among the poverty line calculations provided by the PPI, Family Care selected the US$2/day poverty line calculated at 2005 Purchasing Power Parity (PPP), which is the most updated measure of PPP. The correlation between the revised Simple Economic Strengthening Tool scores and PPI-calculated poverty likelihoods at the $2/day level increased to a low, positive correlation (r=.43). Though there is no single accepted method for interpreting correlation coefficients, a common rule of thumb would suggest that a moderate correlation should reach at least .50.¹⁰

Although the Simple Economic Strengthening Tool contained useful indicators for program monitoring, its ultimate output was only moderately correlated with our poverty measure, which also casts doubt on the validity of the economic vulnerability classifications it generated.

South Africa Assessment

Structural economic drivers at the household and individual levels play a major role in driving disproportionate rates of HIV infection among adolescent girls in South Africa.¹¹ PEPFAR program implementers using economic strengthening interventions to address these drivers need to monitor progress with high-quality M&E tools. Although there are some existing measures of adolescent girls' vulnerability to HIV, none focus specifically on the pathway between individual economic status and HIV risk. The purpose of this assessment was to develop and validate M&E survey tools to assess economic vulnerability for households and individual girls participating in socioeconomic interventions offered by the South African NGO Children in Distress Network (CINDI) to enhance resilience against HIV.

Between October 20 and December 7, 2016, 87 individual interviews with girls and 93 household-level interviews were conducted among a sample of girls aged 10–19 participating in CINDI programs and their caregivers.

The household-level tool (Household Tool) was developed using the same domains derived from the PEPFAR guidance as the Simple Economic Strengthening Tool, with similar questions. Each question was scored 0–3 points and averaged to generate domain scores. Domain scores were then averaged to generate a total score, which was rounded to the nearest whole number to generate a classification. Validity measures included the PPI, household rankings derived from participatory exercises, and subjective classifications generated by the data collector and the respondent.

Thirteen items for the individual-level tool (Girl Tool) were derived from a review of the literature on the links between economic status and HIV for adolescent girls in sub-Saharan Africa based on an assessment of potential variability and on consultation with implementers and an outside expert in health and economic empowerment programs for adolescent girls. Each item was equally weighted, with each indicator scored at a range of 0–3, for an overall range of 0–39. The Vulnerable Girl Index, a measure of adolescent girls' HIV vulnerability validated in several Southern African countries,¹² was used as a validity measure for the Girl Tool. Structured interviews were conducted with 4 program staff members to assess the acceptability, feasibility, and perceived validity of the Household and Girl Tools.

Neither the Household Tool nor the Girl Tool could be validated using the selected validation measures, and both tools had low inter-item reliability (α=.45 and .21, respectively). We also found that the VGI scores accounted for little variance in the data collected and had poor inter-item reliability (α=.19), where a typical benchmark for high inter-item reliability is at least .70.¹³

Table 2 summarizes the 4 ASPIRES vulnerability tools and assessments, including the definition of economic vulnerability, domains assessed, validation measures, and findings.