By Julia Schwarz
Zhuang Liu has joined Princeton as an assistant professor of computer science, bringing expertise in deep learning and computer vision. He started July 1, 2025.
Liu’s research focuses on building multimodal AI models that can see and understand the world through images, video and text.
Large language models, unlike multimodal models, are trained only on text and language. Language models are quite advanced, Liu said, but they often miss key details about the world that are obvious to humans because they aren’t trained on visual data.
"Humans survived in nature for millions of years before we developed language,” Liu said. “The way we interact in the world, the vast majority of the data is visual.”
AI systems that can learn visually are essential for tasks like allowing a car to see and react to objects on the road and enabling a robot to pick up trash up off the floor. Multimodal models are also useful for reading charts and graphs and searching images.
The challenge in developing multimodal models has been the size and complexity of visual data, Liu said. One image of a city street, for example, has so much data in it that many paragraphs of text would be needed to represent the whole scene in language. Multimodal models need to process more data while still operating efficiently and accurately.
Finding ways to do this is the subject of Liu’s research. He is researching what mix of data should be used to train these models and how to build a model that can be generalized across data types.
Liu has been interested in artificial intelligence since he was an undergraduate at Tsinghua University. He liked that it was a combination of math, engineering and experimentation. He then went on to graduate school at the University of California-Berkeley, where he joined one of the first research groups working on deep learning and computer vision.
Looking forward, Liu is interested in building AI models that can work on hard scientific problems. “I think the most important thing we can do is let AI do useful research for us, meaningful and high-impact research,” he said. Multimodal systems are essential for this, he added, because so many hard scientific problems are visual.
This is true even in an abstract field like math, he said. “When I learned geometry, it’s very hard to just operate on a symbolic level. It’s most useful when we think of it as spatial.”
Liu joined Princeton after four years as a research scientist at Meta. This semester, he is co-teaching COS 324: Introduction to Machine Learning. Last semester he taught a graduate seminar on deep learning. He is excited to be back in academia, he said, to pursue his research interests and “to work with a lot of amazing students.”