Dissecting Racial Bias in an Algorithm that Guides Health Decisions for Millions

For millions across the US, hospitals use commercial risk scores to target patients needing extra help with complex health needs. We examine a widely used commercial algorithm for racial bias. Thanks to a unique dataset, we also study the algorithm’s construction, gaining a rare window into the mechanisms of bias.

We find significant racial bias: at the same risk score, blacks are considerably sicker than whites. Removing bias would double the number of high-risk blacks auto-identified for extra help, from 17.7% to 46.5%. We isolate the problem to the algorithm’s objective function: it predicts costs. But since, conditional on health, blacks incur lower costs than whites, accurate cost predictions produce racially biased health predictions.  We find suggestive evidence that this represents a ‘problem formulation error’: as algorithmic prediction is in a nascent stage, convenient choices of proxy labels (in this case, cost) can inadvertently produce biases at scale. We support this view by presenting early results from a collaboration with the algorithm manufacturer, in which we improve the algorithm's ability to predict important health outcomes while simultaneously reducing racial bias.