Using LLMs to identify Social Determinants of Health (SDOH) Domains in Electronic Health Records at NYC H+H
Written by Tauhid Tanjim
The Challenge
The largest municipal health system, New York City Health and Hospitals (NYC H+H), which includes 11 hospitals, 29 Gotham Health Centers, and five Long-Term Care Centers, is dedicated to reducing health disparities by providing the highest quality healthcare services to all New Yorkers. Achieving health equity requires a detailed understanding of Social Determinants of Health (SDoH), which the World Health Organization defines as the conditions in which people are born, grow, live, work, and age. These factors, shaped by the distribution of money, power, and resources, are critical because they account for most of the modifiable factors impacting health outcomes.
SDoH are not always captured in a structured manner within Electronic Health Records (EHRs). At H+H, they are often buried in the free text of clinical notes. For example, when a patient visits a clinic, their demographic information is captured via a structured form, however, critical information about their social situation is often stored in an unstructured way by healthcare professionals. Their social situations often contain important information on patients’ living conditions, environmental or social problems (such as lack of access to public or private transportation), or financial challenges. This process of storing information, or lack thereof, makes it difficult to fully understand the impact of SDoH on health outcomes. It creates a significant barrier to effective research and proactive care. For NYC H+H to truly move the needle on health equity, they should find ways to systematically detect and address SDoH across their entire healthcare system. By doing so, NYC’s health system can better ensure that every patient receives the care and support they need, no matter their social or economic background.
As a Siegel PiTech PhD Impact Fellow this summer, I looked at various SDoH domains, including lack of education, poor housing quality, housing instability, homelessness, unemployment, and financial insecurity. I explored applying natural language processing (NLP) tools to identify these SDoH in EHRs. After thorough analysis, I identified gaps and pitched possible solutions to H+H.
Motivation
Recent advancements in Large Language Models (LLMs) inspired me and my collaborators at H+H to explore LLMs’ potential for extracting SDoH from unstructured clinical notes. After developing a proof-of-concept, we ultimately decided to focus on housing situations as the most promising SDoH for LLM detection.
Discovery and Investigation Process: Examining Cases for Housing Instability
To test the idea of using LLMs, I first ran a search across 3.5 million records using keywords such as "shelter", "unhoused", "home", "supportive housing", "living in subway", and "living on the street," and found 405 relevant cases to work with. I then manually reviewed 405 unstructured clinical notes to determine whether they contained relevant information about housing instability, poor housing quality, or street homelessness. I asked myself specific questions to filter out SDoH domains:
1. Does this note contain information about living outside, in a shelter, or the subway?
2. Is the patient having trouble with electricity or hot water?
3. Is the patient anxious about losing their home in the immediate term?
I identified 26 relevant cases of people experiencing homelessness out of the 405 clinical notes With this refined dataset, I moved on to testing various large language models (LLMs) to automate the detection of these SDoH factors.
At the beginning, the LLMs were producing a lot of false positives, but after refining the prompts, the accuracy of the results improved gradually. The final results are shown in the figure below across various models. The findings demonstrate the potential of LLMs in identifying SDoH factors within EHRs.
Fig 1: Comparison of the Number of Instances of Housing Instability and Homelessness Identified by Different Models, with Human Evaluation as a Benchmark.
Fig. 2: Accuracy Percentage of Various LLMs Compared in Model Performance Evaluation
Fig. 3: Processing Time (in seconds) for Various LLMs Across Model Components.
Impact and Path Forward
Tauhid Tanjim
Ph.D. Student, Information Science, Cornell Tech
LLMs show promising results in this proof-of-concept stage, and I believe that they will become a powerful tool to analyze and emphasize patients’ SDoH in the provision of high quality healthcare services , helping reduce health disparities and promote health equity. .
For next steps, I’d like to scale my approach to cover a broader range of SDoH domains beyond housing-related factors. Further refinement of data for LLMs and their prompts will also be necessary to improve accuracy and reduce false positives or hallucinations. Finally, integrating this system into the NYC H+H workflow could promote real-time identification of SDoH factors, enabling healthcare providers to offer more targeted and timely interventions.