First-Person Fairness in Chatbots

This Colloquium is Co-sponsored with the Center for Data Science for Enterprise and Society’s Data Science Distinguished Lecture Series.

Abstract: 
Much research on fairness has focused on institutional decision-making tasks, such as resume screening. Meanwhile, hundreds of millions of people use chatbots like ChatGPT for very different purposes, ranging from resume writing and technical support to entertainment. We study “first-person fairness,” which means fairness toward the user who is interacting with a chatbot. This includes providing high-quality responses to all users regardless of their identity or background, and avoiding harmful stereotypes. We propose a scalable, privacy-preserving method for evaluating one aspect of first-person fairness across a large, heterogeneous corpus of real-world chatbot interactions. Specifically, we assess potential bias linked to users’ names—which can serve as proxies for demographic attributes like gender or race—in chatbot systems like ChatGPT that can store and utilize user names. Our method leverages a second language model to privately analyze name-sensitivity in the chatbot’s responses. We verify the validity of these annotations through independent human evaluation. In addition to quantitative bias measurements, our approach also identifies common tasks, such as "career advice" or "writing a story" and gives succinct descriptions of subtle response differences across tasks. Finally, we publish the system prompts necessary for others to conduct similar experiments that faithfully simulate ChatGPT conversations with arbitrary user profiles.

This is joint work with Tyna Eloundou, Alex Beutel, David G. Robinson, Keren Gu-Lemberg, Anna-Luisa Brakman, Pamela Mishkin, Meghan Shah, Johannes Heidecke, and Lilian Weng.

Bio: 
Adam Tauman Kalai is a Research Scientist at OpenAI working on AI Safety and Ethics. He has worked in Algorithms, Fairness, Machine Learning Theory, Game Theory, and Crowdsourcing. He received his PhD from Carnegie Mellon University. He has served as an Assistant Professor at Georgia Tech and the Toyota Technological Institute at Chicago. He is a member of the science team of Project CETI, an interdisciplinary initiative to understand the communication of sperm whales. He has co-chaired AI and crowdsourcing conferences including COLT (the Conference on Learning Theory), HCOMP (the Conference on Human Computation) and NEML. His honors include the Majulook prize, best paper awards, an NSF CAREER award, and an Alfred P. Sloan fellowship.