Hybrid Geometry- and Image-Based Modeling and Rendering for Interactive Visualization of Architectural Interiors

Wagner Toledo Corrêa
Eric Chang
Thomas A. Funkhouser

Traditional computer graphics uses geometric and physics information to synthesize images. Using this traditional approach, to get photorealistic results, the geometric model has to be very detailed, and the image synthesis algorithms have to be very complex. Creating a detailed geometric model by hand is difficult, time-consuming, and error prone, and devices for automatic acquisition of 3D models are expensive and limited to small objects. Besides, photorealistic rendering of complex models is still far from real-time rates.

An alternative to the traditional approach is image-based modeling and rendering (IBMR). IBMR systems acquire a collection of images of a environment, reconstruct from these images the plenoptic function of the environment, and create new images by resampling the plenoptic function at new viewing parameters. IBMR systems give us shorter modeling times, faster rendering speeds (independent of scene complexity), while maintaining high level of photorealism.

These are some desirable properties of IBMR systems:

• Capture
  • Unconstrained camera motion
  • Little user intervention
  • Gaze planning
  • Real-time feedback
• Representation
  • Highly concave environments
  • Small number of images
  • Multiresolution
• Reconstruction
  • Use standard 3D polygon texturing hardware
  • Eliminate disocclusions
  • Guaranteed real-time frame rates

There is no single system that meet all those goals.

In our approach, we assume we have photographs or video sequence of the environment and a coarse level 3D model of the environment. This 3D model does not have to be detailed (just boxes for rooms and major pieces of furniture are enough). Such a model is easy to be created manually or even automatically (e.g., extract from blueprints).

Our approach is capable of meeting all those goals:

• Capture
  • Unconstrained camera motion: matching edges in images with 3D geometry provides for more robust camera pose estimation.
  • Little user intervention: use 3D model and frame-to-frame coherence to guide robust edge matching.
  • Gaze planning: use 3D model to predict subset of useful viewpoints can be used for robotic capture
  • Real-time feedback: use pre-planned paths to guide user during capture, and use 3D model to determine coverage in real-time and alert user about deficiencies
• Representation
  • Highly concave environments: 4D representation of plenoptic function stored on the surfaces for each 3D polygon in the approximate model: images, texture coordinates, and coverage region
  • Small number of images: use 3D model to compute minimal subset of images required for coverage
  • Multiresolution: for each 3D polygon, capture images from different distances, and store a hierarchy of textures for efficient management of texture memory
• Reconstruction
  • Use standard 3D polygon texturing hardware: reconstruct images with hardware projective texture mapping and texture blending
  • Eliminate disocclusions: use 3D model to guide interpolation of neighboring pixels values
  • Guaranteed real-time frame rates: render approximate geometry using the hierarchy of texture maps just a few texture mapped polygons per frame