Programming Assignment 7: Seam Carving


Warning: this assignment has not been battle-tested. It is likely that there are more ambiguities and bugs. Please bring those to our attention and we will do our best to clarify and fix.


Seam-carving is a content-aware image resizing technique where the image is reduced in size by one pixel of height (or width) at a time. A vertical seam in an image is a path of pixels connected from the top to the bottom with one pixel in each row. (A horizontal seam is a path of pixels connected from the left to the right with one pixel in each column.) Below left is the original 505-by-287 pixel image; below right is the result after removing 150 vertical seams, resulting in a 30% narrower image. Unlike standard content-agnostic resizing techniques (e.g. cropping and scaling), the most interesting features (aspect ratio, set of objects present, etc.) of the image are preserved.

As you'll soon see, the underlying algorithm is quite simple and elegant. Despite this fact, this technique was not discovered until 2007 by Shai Avidan and Ariel Shamir. It is now a feature in Adobe Photoshop (thanks to a Princeton graduate student), as well as other popular computer graphics applications.

Dr. Hug in the ocean Dr. Hug in the ocean

In this assignment, you will create a data type that resizes a W-by-H image using the seam-carving technique.

Finding and removing a seam involves three parts and a tiny bit of notation:

  1. Notation. In image processing, pixel (x, y) refers to the pixel in column x and row y, with pixel (0, 0) at the upper left corner and pixel (W − 1, H − 1) at the bottom right corner. This is consistent with the Picture data type in stdlib.jar. Warning: this is the opposite of the standard mathematical notation used in linear algebra where (i, j) refers to row i and column j and with Cartesian coordinates where (0, 0) is at the lower left corner.

    a 3-by-4 image
      (0, 0)     (1, 0)     (2, 0)  
      (0, 1)     (1, 1)     (2, 1)  
      (0, 2)     (1, 2)     (2, 2)  
      (0, 3)     (1, 3)     (2, 3)  

    We also assume that the color of a pixel is represented in RGB space, using three integers between 0 and 255. This is consistent with the java.awt.Color data type.

  2. Energy calculation. The first step is to calculate the energy of each pixel, which is a measure of the importance of each pixel—the higher the energy, the less likely that the pixel will be included as part of a seam (as we'll see in the next step). In this assignment, you will implement the dual gradient energy function, which is described below. Here is the dual gradient of the surfing image above:

    Dr. Hug as energy

    The energy is high (white) for pixels in the image where there is a rapid color gradient (such as the boundary between the sea and sky and the boundary between the surfing Josh Hug on the left and the ocean behind him). The seam-carving technique avoids removing such high-energy pixels.

  3. Seam identification. The next step is to find a vertical seam of minimum total energy. This is similar to the classic shortest path problem in an edge-weighted digraph except for the following:

    Vertical Seam

  4. Seam removal. The final step is remove from the image all of the pixels along the seam.

The SeamCarver API. Your task is to implement the following mutable data type: 

public class SeamCarver {
   public SeamCarver(Picture picture)
   public Picture picture()                       // current picture
   public     int width()                         // width of current picture
   public     int height()                        // height of current picture
   public  double energy(int x, int y)            // energy of pixel at column x and row y
   public   int[] findHorizontalSeam()            // sequence of indices for horizontal seam
   public   int[] findVerticalSeam()              // sequence of indices for vertical seam
   public    void removeHorizontalSeam(int[] seam)   // remove horizontal seam from picture
   public    void removeVerticalSeam(int[] seam)     // remove vertical seam from picture

   public static void main(String[] args)         // unit testing

}

Analysis of running time.  Analyze your approach to this problem giving estimates of its time and space requirements by answering the relevant questions in the readme.txt file.

Extra Credit. Submit an image (either your own, or one in the public domain) for which the technique works particularly well (or poorly). Include a brief description in the readme of what makes the image particularly (un)amenable to the technique.

Challenge for the bored.

The challenges worth extra credit are quite difficult, and you certainly shouldn't attempt them just for the points. Make sure to note if you've successfully completed them in your readme.

  1. Your energy() method implemented the dual gradient energy function. Try out other energy functions. No extra credit.

  2. Optimize multiple calls to the find seam and remove seam methods by caching the previous call's information. One point of extra credit if you can get the algorithm running at interactive speeds for small images (say 200 x 200).

  3. Implement an interactive object-removal feature: The user highlights an area of the image, and that portion of the image is forced to zero energy. Rows and columns are then successively removed until every pixel in that zero-energy region has been removed. One point of extra credit.

Submission.  Submit SeamCarver.java, and any other files needed by your program (excluding those in stdlib.jar and algs4.jar). Finally, submit a readme.txt file and answer the questions.

This assignment was developed by Josh Hug, Maia Ginsburg, and Kevin Wayne.