The talk will present three tales told by Scheherazade to the Sultan in Arabian Nights about AI and Science.
1. Sinbad and the Poisoned Datasets: There have been many cases where organizations with vested interests (e.g., tobacco companies) have looked to manipulate public opinion and policies towards their own interests. They have historically done so by bankrolling research designed to serve those interests. This was challenging and expensive. Given we are in this new AI age, what more can they do now, and can we mitigate that?
2. Ali Baba and the 40 Prompts: P-hacking involves researchers torturing data until they get desired -- but often spurious -- results. When using LLM as a judge or using LLMs for annotation, p-hacking is easily done by simply trying many prompts until a desired result is obtained. The conventional way of mitigating p-hacking is preregistration, where researchers must register their analysis plan before collecting any data. But that doesn't work here. So what can we do about it?
3. Aladdin and the Magic Latex: The Magic Latex (i.e., AI) has led to a rapid increase in the number of submissions. Some conferences and journals are putting fixed caps, e.g., nobody can submit more than a certain number of papers. But a large fixed cap leads to a ton of single-author submissions (e.g., in the TMLR journal we have seen many cases of 4-8 single-author submissions by the same person in a span of 1-2 weeks) and a small fixed cap means that advisors with multiple students cannot submit. So where do we draw the line? Or perhaps we draw a curve?
Bio: Nihar B. Shah is an associate professor at Carnegie Mellon University, with joint appointments in the Machine Learning and Computer Science departments. His research is centered around the evaluation of science, including peer review and autonomous AI scientist alignment. His research has already been used in the evaluation of several hundred thousand research papers and thousands of grant proposals, in over 200 venues. The algorithms we have developed are also deployed in diverse applications such as admissions decisions and competition judging. He is a recipient of The Allen Newell Award for Research Excellence at CMU, a Young Alumnus Medal from the Indian Institute of Science, the David J. Sakrison memorial prize from UC Berkeley for a "truly outstanding and innovative PhD thesis," and several Best Paper awards.
To request accommodations for a disability please contact Emily Lawrence, emilyl@cs.princeton.edu, at least one week prior to the event.