How a study on giving the homeless $7,500 arrived at its flawed conclusions

A Vancouver homeless man's sign notes he was originally from Winnipeg. (Kenneth Chan/Daily Hive)

Since last week, there has been much media coverage about the University of British Columbia’s (UBC) study findings on what individuals experiencing homelessness would do if they were each provided with $7,500 cash.

Does it sound familiar? It should, as UBC first announced the preliminary findings of such a study in October 2020, which was favourably reported by Daily Hive Urbanized at the time.

Nearly all of the widespread media coverage — including much national and international attention — over the past week did not acknowledge this is the same study now finally being published in a scientific journal, The Proceedings of the National Academy of Sciences (PNAS). Planning for the study first began in 2016, and it was registered in a database in 2018.

In essence, the study’s lead researcher, Dr. Jiaying Zhao, painted a rosy picture that the cash transfer was a major success, as a means to “debunk” stereotypes and persistent public biases of homeless people and their spending habits — that they would spend a large sum of money on housing, clothing, food, and public transit, and not on alcohol and drugs.

You might also like:

But the newly published journal’s in-depth details on how the study was conducted pours much doubt on its framing of the findings and inputs, as in layman’s terms, it essentially attempts to have its cake and eat it too.

It makes a major attempt to filter out people with major challenges — individuals who are most likely to have addiction and/or mental health issues, even though this is core to the crisis in the streets and presents the greatest health, social, community, and economic implications.

The researchers established a pre-registered screening criteria of individuals who have a non-severe level of substance use, alcohol use, and mental health symptoms. Individuals must also be a Canadian or permanent resident, between the ages of 19 and 65, and have been experiencing homelessness over the short-to-medium term — specifically for under two years.

The rationale for this filtering? “These screening criteria were used to reduce any potential risks of harm (e.g., overdose) from the cash transfer.” While this is a very important consideration, it created a very shaky foundation for the study.

A total of 115 participants were enrolled in the year-long study, entailing 50 homeless individuals who received the cash transfer and 65 homeless individuals who received nothing as the control group.

The researchers chose their participants by screening a total of 732 participants from 22 shelters operated by four shelter organizations across Metro Vancouver.

Only 229 individuals or 31% passed all of the aforementioned criteria.

But over the course of the year-long study, the researchers lost contact with 114 of the 229 participants. This ultimately means just under 16% of the original pool of participants reached the finish line.

And when they could not reach statistical power, which refers to the likelihood of a hypothesis test detecting a true effect if there is one, they combined the study’s condition one and two groups together to form the cash condition, and condition three and four groups to form the control condition. The first two conditions are cash, and the latter two conditions are non-cash.

“Together, this research offers a new approach to address homelessness and provides insights into homelessness reduction policies,” reads the study’s abstract. But no, in actuality, it did not.

As the attrition in participants was higher in the control group (63%) than in the test group (51%), this undermines randomization, according to Jon Baron, who is the founder and president of the US-based Coalition for Evidence-Based Policy, the former chair of the US government’s National Board for Education Sciences, and a former Democratic candidate for the Governor of Maryland.

The study “found no discernible impact on any primary or secondary study outcome. The claimed effect on homelessness (an exploratory outcome) is unreliable due to sample loss of over 50%,” he wrote on X, formerly known as Twitter.

“The study found no discernible impact on any pre-specified primary or secondary outcome (related to cognitive functioning, subjective well-being, and self-efficacy). The study abstract doesn’t mention this fact and instead portrays the findings as unambiguously positive.”

Furthermore, he says, “such inaccurate/unbalanced reporting of findings in study abstracts is unfortunately too common, even in top journals like PNAS. I focus on abstracts because readers — who may be too busy to review a full study — often rely on abstracts for an impartial summary of the results.”

Other researchers in this same topical area also separately confirmed to Daily Hive Urbanized the study’s gaping issues.

@PNASNews published an RCT titled “Unconditional Cash Transfers Reduce Homelessness.” Quick take: The RCT found no discernible impact on any primary or secondary study outcome. The claimed effect on homelessness (an exploratory outcome) is unreliable due to sample loss >50%. 1/6

— Jon Baron (@JonBaronforMD) August 31, 2023

The study found no discernible impact on any prespecified primary or secondary outcome (related to cognitive functioning, subjective well-being, & self-efficacy). The study abstract doesn’t mention this fact & instead portrays the findings as unambiguously positive: 3/ pic.twitter.com/uCdXtW577c

— Jon Baron (@JonBaronforMD) August 31, 2023

Also, I think the claimed effect on homelessness over 1 y (an exploratory outcome) is unreliable for multiple reasons, but most notably: The study lost over half its sample over the year, & the loss was higher in the C group (63%) than T group (51%), undermining randomization. 4/

— Jon Baron (@JonBaronforMD) August 31, 2023

Vinay, the CONSORT diagram on appx p. 8 shows that the original intent-to-treat sample was 53 in the two cash groups (treatment) & 91 in the two non-cash groups (control). The table on appx p. 9 shows that 26/53 (49%) of Ts & 34/91 (37%) of Cs completed the 12-month survey. pic.twitter.com/PmeNEJaI8A

— Jon Baron (@JonBaronforMD) September 1, 2023

You might also like: