10-13

Julian Ost FPO

Julian Ost will present his FPO "3D Driving Scene Generation and Understanding Through Neural Rendering and Generative Priors" on Tuesday, October 13, 2026 at 1:30 PM in CS 105.

The members of Julian’s committee are as follows:
Examiners: Felix Heide (Adviser), Szymon Rusinkiewicz, Jia Deng
Readers: Adam Finkelstein, Dhruv Shah

A copy of his thesis is available upon request.  Please email gradinfo@cs.princeton.edu if you would like a copy of the thesis.

Everyone is invited to attend the talk.

Abstract follows below:

Simulation provides an essential tool for robot learning, and from bipedal robots to autonomous driving, researchers have been successful in transferring models trained and validated in simulation to real-world environments. Simulators for these tasks rely on physics-based dynamics and abstract scene models that are often analyzed after perception– this paradigm implicitly considers perception solved and generalizing. At the same time, neural scene reconstruction methods have promised the ability to rerun recordings and allow scenario changes in large outdoor scenes. However, their baked-in static environments only allow for limited scene control, constrained by the diversity of scenes and trajectories captured.

This thesis addresses both sides of this gap through the lens of neural rendering and data priors: generating diverse 3D environments going beyond what reconstruction can offer, and making 3D perception generalizable by grounding it in explicit geometric priors. I will first give a short overview on scene reconstruction, its challenges, and the limits of pure reconstruction at scale. I will then present geometry-grounded 3D scene generation methods that create entirely new driving and indoor environments at scale through flow matching across 3D data distributions combined with score distillation from 2D image models. By combining coarse 3D and high-fidelity 2D priors, we are constraining the generation process to consistent scene geometry for photo-realistic, large-scale results. Finally, I will present an inverse neural rendering approach to 3D perception that recast vision problems as an inverse rendering problem by optimizing over generative 3D object priors through a differentiable rendering pipeline, providing interpretable and dataset-agnostic inference - This is then validated on the task of 3D multi-object tracking across unseen driving datasets.