Deep-Inverse Correlography: Towards Real-Time High-Resolution Non-Line-of-Sight Imaging

Christopher Metzler
Felix Heide
Prasana Rangarajan
Muralidhar Madabhushi Balaji
Aparna Viswanath
Ashok Veeraraghavan
Richard Baraniuk

Optica 2020

Deep Inverse Correlography: A camera uses light scattered off of a rough wall, known as a virtual detector, to reconstruct an image of the hidden object. When using a continuous-wave laser, the camera records speckle. Inset, non-line-of-sight (NLOS) correlography estimates an object’s autocorrelation using speckle images. It then uses this autocorrelation estimate to recover the object’s shape by solving a Phase Retrieval problem. Because it is more robust to noise, the CNN-based method can operate with far less light and thus at higher frame rates than a system relying on traditional PR algorithms like HIO or Alt-Min (exposure time length indicated in the bottom legend).

Low signal-to-noise ratio measurements, primarily due to the quartic attenuation of intensity with distance, are arguably the fundamental barrier to real-time, high-resolution, non-line-of-sight (NLoS) imaging at long stand-offs. To better model, characterize, and exploit these low SNR measurements, we use spectral estimation theory to derive a noise model for NLoS correlography. We use this model to develop a speckle correlation based technique for recovering occluded objects from indirect reflections. Then, using only synthetic data sampled from the proposed noise model, and without knowledge of the experimental scenes nor their geometry, we train a deep convolutional neural network (CNN) to solve the noisy phase retrieval problem associated with correlography. We validate that the resulting Deep-Inverse Correlography approach is exceptionally robust to noise, far exceeding the capabilities of existing NLOS systems both in terms of spatial resolution achieved and in terms of total capture time. We use the proposed technique to demonstrate NLoS imaging with 300 μm resolution at a 1 m standoff, using just two 18-th second exposure-length images from a standard CMOS detector.


Christopher A. Metzler, Felix Heide, Prasana Rangarajan, Muralidhar Madabhushi Balaji, Aparna Viswanath, Ashok Veeraraghavan, Richard G. Baraniuk

Deep-inverse correlography: towards real-time high-resolution non-line-of-sight imaging

Optica 2020


In the News

OSA News:



Nvidia News:


Phys Org:

Golem News:

Yahoo News: