Programming Assignment 2: Light Fields

Overview

For this assignment, you will implement a light field viewer based on the techniques described by Levoy & Hanrahan 96.

I. Deriving Equations

The basic operation performed by the light field viewer is looking up the color corresponding to a given ray in space. Since the light field is represented by "slabs" (pairs of planes in space), you will need to derive code for intersecting a ray in space with a rectangular region of a plane (specified with the region's four corners).

Hints:

• Make sure to report "no intersection" if the ray intersects the plane outside the valid region or if the intersection is behind the viewer.
• Once you know the point of intersection, and given that the region is represented with mxn samples, you will have to find either the closest sample or, for bilinear interpolation, the four closest samples.
• Because light fields sometimes use slabs with planes (especially the uv plane) at infinity, you should derive your equations such that they take corners specified in homogeneous coordinates (with the w coordinate possibly 0).

II. Basic Light Field Viewer (60%)

1. Begin with code to read a single-slab, uncompressed light field given as a pair of files:
   • A "parameters" file that contains the following in plain ASCII format

     	samples_u samples_v samples_s samples_t
     
     	uv_corner1_x uv_corner1_y uv_corner1_z uv_corner1_w  uv_corner1_p uv_corner1_q
     	uv_corner2_x uv_corner2_y uv_corner2_z uv_corner2_w  uv_corner2_p uv_corner2_q
     	uv_corner3_x uv_corner3_y uv_corner3_z uv_corner3_w  uv_corner3_p uv_corner3_q
     	uv_corner4_x uv_corner4_y uv_corner4_z uv_corner4_w  uv_corner4_p uv_corner4_q
     
     	st_corner1_x st_corner1_y st_corner1_z st_corner1_w  st_corner1_p st_corner1_q
     	st_corner2_x st_corner2_y st_corner2_z st_corner2_w  st_corner2_p st_corner2_q
     	st_corner3_x st_corner3_y st_corner3_z st_corner3_w  st_corner3_p st_corner3_q
     	st_corner4_x st_corner4_y st_corner4_z st_corner4_w  st_corner4_p st_corner4_q
     

     For example, the following parameters

     	16 16  256 256
     
     	-2 -2  2  1   0  0
     	 2 -2  2  1   1  0
     	 2  2  2  1   1  1
     	-2  2  2  1   0  1
     
     	-1 -1  0  1   0  0
     	 1 -1  0  1   1  0
     	 1  1  0  1   1  1
     	-1  1  0  1   0  1
     

     specify a light field with 16x16 samples in uv and 256x256 samples in st. The lower-left (0,0) corner on the uv plane is at (-2,-2,2,1), the lower-right (1,0) corner is at (2,-2,2,1), etc.

   • A "data" file of length samples_u * samples_v * samples_s * samples_t * 3 that contains the raw uvst data, given as RGB triples. This is raw binary data expressed as bytes in RGB order with no padding or line breaks. In C, you'd read it using something like

             FILE *lfdatafile = fopen("data", "rb");
             for (i = 0; i < uwidth*uheight*vwidth*vheight; i++) {
                     unsigned char buf[3];
                     fread(buf, 3, 1, lfdatafile);
                     unsigned char red = buf[0];
                     unsigned char green = buf[1];
                     unsigned char blue = buf[2];
                     /* Do something with the data... */
             }
     

     The order of the samples varies fastest in "t", next fastest in "s", then "v" and "u". That is, looking at (u,v,s,t) indices, the first sample is (0,0,0,0), the next one is (0,0,0,1), and so on.

   Some sample data files are here.

2. For each frame, do the following:
   • Implement a mouse-based camera controller.
   • For each pixel, generate a ray starting at the camera position and in the pixel's direction.
   • Intersect the ray with the uv and st planes, finding the closest sample in each.
   • If both intersections are valid, look up the color for that pixel, else color the pixel black.
   • Once all pixels have been assigned a color, use glDrawPixels() to render to the screen.

III. Extensions (30%)

Implement extensions (A and B) xor C:

A. Interpolation

In addition to the "closest-point" sampling implemented above, implement the following interpolation schemes:

• Nearest in uv, bilinear in st.
• Bilinear in uv, nearest in st.
• Quadrilinear in uvst.

Which of the first 2 appears to produce better results? Why?

B. Multiple Slabs

Extend your viewer to handle light fields consisting of multiple slabs. Some data is here.

C. VQ Decompression

Extend your system to handle VQ-compressed data. The changes in the data format are as follows:

• The "parameters" file will contain an additional line indicating the VQ tile size and codebook length. For example,

  	1 1 2 2  16384
  

  indicates that the VQ tiles have extent 1x1 in uv, 2x2 in st, and that valid codewords are 0 through 16383.
• The "data" file will have length

  	(samples_u / tilesize_u) * (samples_v / tilesize_v) * 
  	(samples_s / tilesize_s) * (samples_t / tilesize_t) * 2
  

  with 2-byte codebook indices replacing the raw RGB data.
• An additional "codebook" file will contain the VQ codebook entries. For the example above, there will be 16384 entries, each of length 1*1*2*2*3 (an RGB color for each entry in a 1x1x2x2 tile).

Some data to get you started is here. However, once you have VQ decompression implemented, it should be possible to view the .lif files available here. The only tedious part is parsing the .lif header, documented here.

Note that for this assignment, it is acceptable to decompress the data in memory before the start of rendering - it is not necessary to render directly from the VQ-compressed data.

IV. Writeup and analysis (10%)

Please submit your source code together with a writeup (as plain text, HTML, or PDF) that contains a description of the design decisions you made, options you implemented, and sample screen-shots.

Submitting

Please make your writeup and code accessible via the web, and send the URL to funk@princeton.edu with "COS526" in the subject line. Please see the general notes on submitting your assignments, as well as the late policy and the collaboration policy.