Student Conference Tackles AI & DEI in Healthcare
Table of Contents
- AI in Healthcare: navigating Innovation with Equity
- AI in Healthcare: Navigating Innovation with Equity – Q&A
- Key Questions and Answers:
- Why is data equity crucial in AI healthcare applications?
- What are algorithmic bias and its effects on healthcare?
- How can we ensure a human element in AI-driven healthcare?
- What are the implications of DEI (Diversity, Equity, and inclusion) Initiatives for Healthcare?
- What were the key insights from student perspectives at the conference?
- How did the algorithm used by an insurance company demonstrate algorithmic bias, and what would have been a remedy?
- What risks does dependence on AI for health decisions bring?
- What steps can be taken to improve AI development?
- AI and Healthcare: Key Considerations
- Moving Forward
- Key Questions and Answers:
published: March 12, 2025
The Promise and Peril of Artificial Intelligence in Healthcare
As artificial intelligence (AI) increasingly integrates into healthcare systems, researchers emphasize the necessity for continuous monitoring and refinement of these technologies. The 46th annual Minority Health Conference at UNC Chapel Hill’s Friday Conference Center highlighted both the potential benefits and inherent risks of AI in healthcare.
The conference literature emphasized that, “While technological innovations hold great promise for improving access to care, they often risk amplifying inequities when not designed with marginalized communities in mind.”
This sentiment underscores the critical need for data equity in the advancement and deployment of AI tools within the healthcare sector.
The Human Element in AI-Driven Healthcare
Fay Cobb Payton, a NC State University professor emeritus specializing in information technology and business analytics, stressed the importance of human oversight.”You always need a human in the loop, notably in health care,” she stated, emphasizing that healthcare decisions require discernment beyond mere data analysis.
cobb Payton cautioned that without careful attention to equity in data collection and usage, AI outcomes can be skewed, potentially harming patients. ”In health care, you’re not selling widgets, you’re not selling books, you’re not buying a piece of clothing,” she explained. “you still need discernment.”
DEI Under Scrutiny: Implications for Healthcare equity
The conference took place amidst a backdrop of increasing pressure to curtail Diversity, Equity, and inclusion (DEI) programs. Recent actions aim to eliminate DEI initiatives from federally funded programs, with parallel efforts in the state Senate and state House to limit such programs in public schools and state agencies.
Despite these challenges, the minority Student Caucus at UNC, which has organized the conference since 1977, continues its work. Conference organizers believe that student-run groups like the Minority Student Caucus fall outside of the new constraints on university policies that had addressed diversity, equity and inclusion.
The conference drew a large crowd, with over 600 attendees at the Friday Center and an additional 200 participating virtually. Discussions spanned a wide array of topics, from enhancing access to reliable health resources for underserved populations to leveraging telehealth to better serve transgender individuals.
Algorithmic Bias: A Threat to Equitable Healthcare
The discussion around artificial intelligence revealed important concerns about potential biases embedded within these systems.
Cobb Payton cited Harvard University research indicating that AI could potentially halve treatment costs by enabling quicker and earlier diagnoses, leading to potentially 40 percent better health outcomes.She also noted that AI innovation in healthcare could become a $200 billion market by 2030, creating strong incentives for adoption and development.
However, she cautioned that, “Largely tech developers don’t have public health experts and clinical experts in the room.There needs to be some clarity when it comes to how the technology is being developed, even when it’s developed.”
She highlighted instances where tools developed without diverse input resulted in “algorithmic bias,” skewing results and perpetuating inequities.
Example of Algorithmic Bias
One notable example involved an algorithm used by an insurance company to assign patients to a care management program. this program aimed to provide additional resources to patients with chronic conditions. However, the algorithm relied on cost as a predictive variable, rather than patients’ actual diagnoses.
A seminal paper published in 2019
in Science revealed that this approach disproportionately disadvantaged black patients. Researchers found that, “The algorithm falsely concluded that black patients were healthier to equally sick white patients,” because Black patients often spend less on healthcare due to wealth disparities and barriers to access.
The study demonstrated that removing cost from the equation or incorporating race into the algorithm would have significantly increased the percentage of Black patients assigned to the care management program, from 17.7 percent to 47 percent.
Cobb Payton emphasized, “Here, the risk scores have become very critical because Black patients are considered sicker than white patients given the same risk score but receive fewer resource allocations.” She also cited examples of algorithmic bias leading to lower priority for Black kidney transplant candidates and increased rates of caesarean deliveries for Black mothers. “The technology is running wild while the health policy is still trying to figure it out,” she added.
Student Perspectives: Bridging the Digital Divide
Justin Wang,a UNC Chapel Hill senior majoring in public health,highlighted the growing interest in artificial intelligence and the importance of being aware of biases in machine learning algorithms. “It’s significant that we’re conscious of all these biases that are built into machine learning, AI algorithms that we deal with every single day,” he said.
Wang, along with mercy Adekola, organized the conference, focusing on how statistics can both exacerbate and improve health disparities. Wang said, “Minority health has always been something that I’ve been interested in, just my experiences within clinics and seeing how minorities might be treated differently just because of their background — or not even treated differently, but their experiences are different.”
Adekola, reflecting on the current political climate, noted, “Many people have fought to get here, to be able to use words like diversity, equity, inclusion, racism, bias, prejudice — and it all came from things like slavery, civil rights.”
Wang added, “There’s a lot of discouraging things going on in the world right now. But the important thing, I think, that I try to keep in mind, and I think this conference really tries to emphasize, is you can really only do what you can do. Nonetheless of what else is going on outside… there are always things that you can do in your own space,with the community around you,that you can help or improve the lives of.”
Despite the challenges, Adekola expressed feeling rejuvenated after the conference. “We are the future, you know? And I feel like tomorrow is now,” she said.”I feel like,as students,that sense of urgency is there to craft innovative solutions that help our tomorrow,because tomorrow is today,you know,and we are the tomorrow.”
Adekola concluded, ”We have to start being inspired to innovate solutions.”
Artificial intelligence (AI) is rapidly changing the healthcare landscape, offering the potential to improve access, diagnoses, and health outcomes. Though, the integration of AI also introduces risks, particularly concerning equity and bias. This Q&A explores these critical issues, drawing insights from experts and real-world examples presented at the 46th annual Minority Health Conference at UNC chapel Hill.
Key Questions and Answers:
Why is data equity crucial in AI healthcare applications?
what is data equity in AI? Data equity in AI refers to the fair and impartial representation of diverse populations and their specific health needs within the datasets used to train AI algorithms. Without data equity, AI systems can produce biased results that reinforce or even amplify existing health disparities.
Why does data inequity impact AI in healthcare?
Marginalized communities are frequently enough underrepresented: AI algorithms trained on incomplete or biased data may not accurately reflect the health realities of minority groups.This can lead to misdiagnoses, inappropriate treatment recommendations, and reduced access to essential healthcare services.
AI algorithms can amplify existing inequalities: If the data reflects existing biases in healthcare, the AI will learn and perpetuate this bias. This could lead to decisions that disadvantage specific patient populations.
What are algorithmic bias and its effects on healthcare?
What is algorithmic bias in healthcare? Algorithmic bias occurs when systematic and unfair outcomes result from the use of algorithms in healthcare decisions. These biases arise from flawed data, poorly designed algorithms, or a lack of diverse perspectives in the development process.
How does algorithmic bias affect healthcare equity? Instances of Algorithmic bias impacting equitable healthcare include:
Inaccurate Risk Assessments: As highlighted by Cobb Payton, AI-driven risk scores can misclassify Black patients as healthier than white patients with the same risk profile, leading to reduced access to care management programs.
Organ Transplant Access Disparities: Algorithmic bias can affect waiting list priority for Black patients needing kidney transplants.
disparities in Maternal Care: Increased rates of caesarean deliveries for Black mothers can stem from biased algorithms influencing clinical decisions.
How can we ensure a human element in AI-driven healthcare?
The Importance of Human Oversight: Fay Cobb Payton emphasizes that “you always need a human in the loop” when using AI in healthcare. Algorithmic outputs should be reviewed by clinicians who can apply their judgment and experience to make informed decisions.
The Limits of data analysis: Healthcare decisions require nuance and understanding that go beyond what data analysis alone can provide. Human discernment is essential to ensure that AI systems serve patients’ best interests.
What are the implications of DEI (Diversity, Equity, and inclusion) Initiatives for Healthcare?
The Conference Context: The 46th annual Minority health Conference took place amidst rising pressure to curtail DEI programs, wich aim to promote diverse participation, fair resource allocation, and inclusive healthcare practices.
The Value of Student-run Groups: Even as formal DEI initiatives face constraints, student-led organizations like the Minority Student Caucus continue to play a critical role in advocating for health equity.
What were the key insights from student perspectives at the conference?
Growing Awareness Among Students: Justin Wang highlighted the increasing student interest in AI and the meaning of understanding biases in machine learning algorithms.
Actionable Steps for Promoting Equity: Wang and Mercy Adekola organized the conference to explore how statistics can be used to both worsen and improve health disparities. By focusing on individual contributions and community-level actions, students can make a tangible difference.
Innovative Solutions: Adekola notes the urgency among students to “craft innovative solutions that help our tomorrow” and be inspired to start innovating.
How did the algorithm used by an insurance company demonstrate algorithmic bias, and what would have been a remedy?
The Problem: The algorithm used cost as a primary predictor to assign patients to a care management programs.Though,it failed to consider the structural inequities that affect Black patients,causing the algorithm to evaluate them as healthier.
The Study: A study published in science found that black patients with similar health needs were less frequently assigned due to lower healthcare spending stemming from systemic issues.
The Solution: Incorporating race into the algorithm or removing cost as this would lead to better,equitable assignment.
What risks does dependence on AI for health decisions bring?
Dependence on AI in healthcare may lead to risks like:
Algorithmic bias can create inequities by resulting in incorrect diagnoses or inappropriate treatment;
Privacy concerns from data sharing and confidentiality violations;
Over-reliance on AI can diminish human expertise, empathy in patient care
What steps can be taken to improve AI development?
Ensuring inclusion of diverse public health and clinical experts in the development teams;
Improving clarity standards for the way the technology is being developed
AI and Healthcare: Key Considerations
| Issue | description | Potential Impact |
| :——————– | :————————————————————————————————————————— | :—————————————————————————————————————————— |
| Data Equity | Fair and impartial representation of diverse populations in AI training datasets. | Reduces biases and ensures that AI systems accurately reflect the health needs of all patient groups. |
| Algorithmic Bias | Systematic and unfair outcomes resulting from flawed AI algorithms. | Reinforces or amplifies existing health disparities; can lead to misdiagnoses and inappropriate treatment recommendations.|
| Human Oversight | The need for clinician review and judgment in AI-driven healthcare decisions. | Ensures that AI systems are used responsibly and that patient care remains personalized and empathetic. |
| DEI Initiatives | Programs aimed at promoting diversity, equity, and inclusion in healthcare and technology.| Addresses health disparities and promotes equitable access to healthcare resources for marginalized communities. |
| Student Engagement | Encouraging student involvement in identifying and addressing AI-related health inequities. | Fosters innovation and ensures future healthcare professionals are equipped to navigate the ethical challenges of AI. |
Moving Forward
Addressing the ethical dimensions of AI in healthcare is essential for ensuring equitable access and optimal health outcomes for all. The insights and perspectives shared at the 46th annual Minority Health Conference offer guidance for promoting responsible AI innovation and advancing health equity.