Dissecting Racial Bias in an Algorithm that Guides Health Decisions for Millions
For millions across the US, hospitals use commercial risk scores to target patients needing extra help with complex health needs. We examine a widely used commercial algorithm for racial bias. Thanks to a unique dataset, we also study the algorithm’s construction, gaining a rare window into the mechanisms of bias.
We find significant racial bias: at the same risk score, blacks are considerably sicker than whites. Removing bias would double the number of high-risk blacks auto-identified for extra help, from 17.7% to 46.5%. We isolate the problem to the algorithm’s objective function: it predicts costs. But since, conditional on health, blacks incur lower costs than whites, accurate cost predictions produce racially biased health predictions. We find suggestive evidence that this represents a ‘problem formulation error’: as algorithmic prediction is in a nascent stage, convenient choices of proxy labels (in this case, cost) can inadvertently produce biases at scale. We support this view by presenting early results from a collaboration with the algorithm manufacturer, in which we improve the algorithm's ability to predict important health outcomes while simultaneously reducing racial bias.
Ziad Obermeyer is an associate professor in the School of Public Health. His research combines insights from medicine with methods from biostatistics, computer science, and econometrics.
The I School Research Exchange offers I School faculty and Ph.D. students opportunities to learn about, discuss, and contribute to research developing within the school, across the campus, and in the region.
Lunch, for those who have signed up, will be served at 12:00. The talk start at 12:30 and we try to wrap up between 1:45 and 2:00.
Meetings are open to I School faculty, I School Ph.D. students, I School visiting scholars, and invited guests