Picture of me

Geoffrey Roeder

I am a PhD student at Princeton working on statistical machine learning (as part of the Artificial Intelligence and Machine Learning research group), advised by Ryan Adams, in the Laboratory for Intelligent Probabilistic Systems. In 2018, I completed my MSc with David Duvenaud at the Vector Institute for Artificial Intelligence, while a student in Machine Learning group at the University of Toronto. I completed my BSc (2016) at the University of British Columbia in Statistics and Computer Science.

I spent summer 2016 working in Mark Schmidt's Machine Learning Lab where I developed unsupervised learning algorithms for a Matlab machine learning toolbox. I spent fall 2017 working with Ferenc Huszár on improving black-box optimization methods for general non-differentiable functions. During summer 2018, while an intern at Microsoft Research Cambridge, I collaborated on a novel class of deep generative models for understanding and programming information processing in biological systems. As of summer 2019, I am an intern at Google Brain in San Francisco, working with Durk Kingma on a better understanding of representation learning in deep generative models.

More broadly, I am motivated in my research to push forward a theoretical understanding of deep learning, in support of improving robustness and reliability of deep statistical models, while exploring how new affordances in deep generative models can improve existing practices in scientific discovery and engineering design.


Curriculum Vitae

Email: roeder@princeton.edu

Selected Research

Computational flow

Efficient Amortised Bayesian Inference for Hierarchical and Nonlinear Dynamical Systems

We introduce a flexible, scalable Bayesian inference framework for nonlinear dynamical systems characterised by distinct and hierarchical variability at the individual, group, and population levels. Our model class is a generalisation of nonlinear mixed-effects (NLME) dynamical systems, the statistical workhorse for many experimental sciences. We cast parameter inference as stochastic optimisation of an end-to-end differentiable, block-conditional variational autoencoder. We specify the dynamics of the data-generating process as an ordinary differential equation (ODE) such that both the ODE and its solver are fully differentiable. This model class is highly flexible: the ODE right-hand sides can be a mixture of user-prescribed or "white-box" sub-components and neural network or "black-box" sub-components. Using stochastic optimisation, our amortised inference algorithm could seamlessly scale up to massive data collection pipelines (common in labs with robotic automation). Finally, our framework supports interpretability with respect to the underlying dynamics, as well as predictive generalization to unseen combinations of group components (also called "zero-shot" learning). We empirically validate our method by predicting the dynamic behaviour of bacteria that were genetically engineered to function as biosensors.

Accepted for publication and short oral at ICML 2019: arXiv link; poster link

Design manifold

Design Motifs for Probabilistic Generative Design

Generative models can be used to produce designs that obey hard-to-specify constraints while still producing plausible examples. Recent examples of this include drug design, text with desired sentiment, or images with desired captions. However, most previous applications of generative models to design are based on bespoke, ad-hoc procedures. We give a unifying treatment of generative design based on probabilistic generative models. Some of these models can be trained end-to-end, can take advantage of both labelled and unlabelled examples, and automatically trade off between different design goals.

Submitted to ICLR 2018 workshop track.

Surface plot depicting problem

Backpropagation through the Void: Optimizing Control Variates for Black-Box Gradient Estimation

Gradient-based optimization is the foundation of deep learning and reinforcement learning. Even when the mechanism being optimized is unknown or not differentiable, optimization using high-variance or biased gradient estimates is still often the best strategy. We introduce a general framework for learning low-variance, unbiased gradient estimators for black-box functions of random variables. Our method uses gradients of a neural network trained jointly with model parameters or policies, and is applicable in both discrete and continuous settings. We demonstrate this framework for training discrete latent-variable models. We also give an unbiased, action-conditional extension of the advantage actor-critic reinforcement learning algorithm.

Accepted as a contributed talk at the Deep Reinforcement Learning Symposium, NIPS 2017.

I gave a talk on the paper at the University of Cambridge in November, 2017

Accepted for publication at ICLR 2018

Surface plot depicting problem

Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference

We propose a simple and general variant of the standard reparameterized gradient estimator for the variational evidence lower bound. Specifically, we remove a part of the total derivative with respect to the variational parameters that corresponds to the score function. Removing this term produces an unbiased gradient estimator whose variance approaches zero as the approximate posterior approaches the exact posterior. We analyze the behavior of this gradient estimator theoretically and empirically, and generalize it to more complex variational distributions such as mixtures and importance-weighted posteriors.

A short version of the paper was published at NIPS 2016's Advances in Approximate Bayesian Inference workshop

The full length version of the paper was published at NIPS 2017

Andrew Miller wrote a great blog post exploring the key ideas of the paper.

Manifold to learn with t-SNE

MatLearn: Machine Learning Algorithm Implementations in Matlab

Link to website

I merged multiple code bases from many graduate student contributors into a finished software package, and added a variety of new unsupervised learning algorithms including sparse autoencoders, Hidden Markov Models, Linear-Gaussian State Space Models, t-Distributed Stochastic Neighbour Embedding, and Convolutional Neural Networks for image classification.

Download package