The AI Lab is hosting the Policy Fellows Seminar with speakers Aisha Nájera, Neil Perry, Ryan Steed, and Rajesh Veeraraghavan. As part of the AI Lab’s Policy Fellows Program, these researchers work closely with governmental agencies at the state and federal level to advance the responsible development and utilization of AI. Join us as they share insights from their work at the intersection of technology and public policy.
AI Lab Policy Fellows:
Aisha Nájera and Rajesh Veeraraghavan
Talk Title: Coding for Safety and Expertise: Assessing Government AI in Health and Human Services
Abstract: We analyze a database of government AI use cases in welfare to ask what kinds of risks and forms of expertise are made visible, and which remain opaque. By coding these cases against emerging safety categories, we sketch a mapping schema that links AI functions to potential risks while also tracing how expertise is implicated, redistributed, or obscured in these systems. In doing so, we also consider how existing AI risks taxonomies can be assessed and adapted to the specific operational needs, identifying practices for managing risks in context. Our aim is to open a way of thinking about how welfare AI inventories surface questions of safety and expertise.
Neil Perry
Talk Title: A Measurement Platform for Prompt Injection in AI Agents
Abstract: My research at CAISI focuses on measuring the security of AI agents against prompt injection to better understand current vulnerabilities and defenses. I am designing a crowdsourced platform that captures the creative breadth of attacks and defense designs from real people, provides a continuously updated leaderboard, and enables large-scale comparative studies. The platform makes it easy for anyone to write and share modular agent architectures, which can then be evaluated for formal security properties. By making measurement an ongoing, community-driven effort, the system captures new attacks and defenses as they emerge and compares the performance of different models and architectures to better understand how to use and secure agents.
Ryan Steed
Talk Title: Does It Work? Open Measurement Science Questions for AI Evaluation
Abstract: Integrating AI systems into real-world workflows is a complex process that can have unexpected effects. Industry-standard benchmarks fail to capture these complexities—they generally involve simplistic tasks in isolated settings. If our most popular measures may not tell us much about performance in the real world, how should we measure whether AI systems really work? How do we evaluate risks associated with their use? In this talk, I’ll share some open measurement science questions facing policymakers and talk about the future of evaluation and procurement standards in the workplace.