Research Identifies Limitations to Reliability of Service Prioritization Decision Assistance Tools (SPDATs) Used to Assess Vulnerability of People Experiencing Homelessness

By the NLIHC Research Team

A recent study published in Housing Policy Debate, “Assessing the Reliability of SPDAT Homelessness Vulnerability Tools and the Impact of Assessor Consistency and Changes to Homeless Vulnerability Over Time,” examined the reliability of the Service Prioritization Decision Assistance Tools (SPDATs), a suite of four instruments used to assess the vulnerability of people experiencing homelessness and prioritize high-risk individuals and families for short-term rapid rehousing (RRH) or longer-term permanent supportive housing (PSH) vouchers. The study finds that the SPDATs were most reliable when the longer-form versions were used, when reassessments were conducted by the same service provider, and when reassessments were conducted more frequently. 

In many local continuums of care (CoC), intake of people experiencing homelessness begins with a prescreening using shorter, easier-to-administer versions of the SPDAT instruments—the Vulnerability Index-SPDAT (VI-SPDAT) for individuals, and the Family Vulnerability-Index SPDAT (F-VI-SPDAT) for families. These prescreens identify people experiencing homelessness as experiencing low, moderate, or high levels of vulnerability. People who receive higher scores on the VI-SPDAT and F-VI-SPDAT are then eligible for a more intensive evaluation using the full versions of the tools, simply referred to as SPDAT for individuals and Family-SPDAT (F-SPDAT) for families. The researchers emphasize that the consistent reliability of these instruments is essential, as “there is a high cost to misallocate” the already-limited supply of RRH opportunities and PSH vouchers. 

The researchers sought to evaluate the reliability of the SPDATs to help inform proposed changes to these instruments, as well as the development of new instruments for assessing vulnerability among people experiencing homelessness. They used data from the Utah Homeless Management Information System (UHMIS) on all single adults and families experiencing homelessness who participated in at least one SPDAT assessment between August 2014 and October 2022. After data cleaning, this included 28,825 people experiencing homelessness, with 7,169 participating in the VI-SPDAT, 4,526 in the SPDAT, 1,696 in the F-VI-SPDAT, and 1,444 in the F-SPDAT; these counts include people who participated in both the prescreen and full-length versions of the individual or family instruments.  

The researchers then used standardized metrics to evaluate the internal, inter-rater, and intra-rater reliability of each of the four SPDAT instruments. Internal reliability examined whether all the components within an instrument are doing a good job of measuring the same concepts. If an instrument has low internal reliability, it implies that participants may not give consistent answers because the questions are not well-constructed. Inter-rater reliability examined whether an instrument gives consistent results when administered by different service providers, while intra-rater reliability examined whether an instrument gives consistent results when administered by the same service provider at different points in time. High inter-rater and intra-rater reliability suggests that the instrument produces consistent, dependable results over time and across different service providers.   

The researchers found that the SPDAT and F-SPDAT demonstrated acceptable to good internal reliability across all sections of the instruments examined and good internal reliability overall. In contrast, the prescreen VI-SPDAT and F-VI-SPDAT instruments showed poor to questionable reliability across all sections examined, and poor overall internal reliability. The findings for the VI-SPDAT echo concerns that have arisen in recent years about the consistency of VI-SPDAT scores, particularly across different racial groups. The authors speculate that the SPDAT and F-SPDAT may demonstrate higher reliability than the VI-SPDAT and F-VI-SPDAT because administrators of the former are required to undergo more comprehensive training than administrators of the latter, equipping them to better “engage with and elicit clear and accurate responses from people experiencing homelessness.” 

The researchers also found that reliability was consistently greater among assessments completed by the same provider over time than among different providers (i.e., higher intra-rater reliability than interrater reliability) across all four SPDAT instruments. The authors posit that when a service provider has repeated encounters with the same individual or family experiencing homelessness, their familiarity may result in more open communication, dependable evaluations of need, and quicker identification of changes in circumstances that contribute to greater reliability. 

However, the reliability of all four instruments was found to decrease over time, regardless of whether the assessment was given by the same service provider or a different provider. Although it is common practice to reassess an individual or family’s vulnerability on an annual basis, the authors found that the most significant declines in reliability scores occurred among reassessments conducted between 6 months to 1 year after the previous assessment. The researchers suggest that because the circumstances of people experiencing homelessness can frequently change more frequent reassessments are warranted to avoid delays in the identification of new or worsening vulnerabilities of people experiencing homelessness.   

Based on their findings, the authors conclude that the SPDAT and F-SPDAT instruments may be more appropriate tools for reliably assessing the vulnerabilities of people experiencing homelessness as compared to the VI-SPDAT and F-VI-SPDAT. However, they caution that the SPDAT and F-SPDAT are themselves imperfect, and that more frequent reassessments and better training for SPDAT administrators are warranted to ensure timely, reliable information while new and better tools for vulnerability assessment are developed. 

Read the report at: http://bit.ly/4kR09TO