What Gets Measured Gets Governed: Benchmarking Privacy in Frontier AI Development and Deployment

In this article for the Future of Privacy Forum, authors Jameson Spivack & Alex Rosenberger (Future of Privacy Forum) discuss the growing need for standardized benchmarks to evaluate privacy risks in frontier AI systems. While benchmarks are widely used to measure model capabilities and safety, there is no commonly accepted standard for comparing AI systems’ privacy performance. Such benchmarks could help developers establish measurable privacy targets, enable organizations to compare vendors, and give regulators and independent researchers a consistent framework for assessing risks.

The authors highlight several privacy challenges associated with advanced AI, including the memorization and disclosure of personal information contained in training data, the re-identification of anonymized or pseudonymized data, and the inference of sensitive characteristics from seemingly innocuous information. Increasingly autonomous AI agents introduce further risks through their access to personal data and external systems, potentially collecting, retaining, or sharing more information than necessary. Other concerns include nonconsensual deepfakes, impersonation, and AI-driven decisions affecting employment, healthcare, and other consequential areas.

Standardized privacy benchmarks could embed privacy considerations into AI development and provide a repeatable means of evaluating models throughout their lifecycle. They could help developers test privacy protections, assist organizations with vendor selection and risk assessments, and give policymakers and independent evaluators more reliable evidence when assessing AI systems. Existing initiatives, including ConfAIde, PrivacyLens, and AgentDAM, already examine specific privacy risks, but a broadly accepted, cross-industry standard has yet to emerge.

However, the authors caution that benchmarks cannot capture every dimension of privacy. Quantitative assessments may overlook systemic harms such as surveillance, fail to represent diverse populations, or measure imperfect proxies for real-world risks. Results can also be distorted when developers optimize models specifically to perform well on tests, while differences in privacy laws and norms complicate international standardization. The authors point to the MLCommons Privacy and Confidentiality Working Group, in which the Future of Privacy Forum participates, as an emerging effort to develop standardized evaluations of sensitive information disclosure and data minimization in AI agents.

CONTINUE READING…