Auditing the AI auditors: A framework for evaluating fairness and bias in high stakes AI predictive models.

Although artificial intelligence (AI) is now being used to make decisions about people’s employment, education, healthcare, and experiences with law enforcement, external evaluators do not often agree on what is necessary to show that an AI is “unbiased” or “fair.” This is in part because “bias” and “fairness” mean different things to different people. We created a framework for auditing that respects these differences in pursuit of better, fairer AI.