COS 432: Information Security

Design Reviews

Some assignments ask you to do a "design review". This Web page describes what a design review is, and gives you hints about how to carry one out.

What is a Design Review?

In a design review, a group of "reviewers" evaluates a design and implementation project done by a group of "designers". The reviewers get a copy of all documents produced by the designers, and they get a chance to ask the designers questions in a face-to-face meeting.

Design reviews are an important part of security practice in the real world. Bringing in a set of "fresh eyes" to look over a project, and requiring the designers to go through the exercise of justifying their design, can provide invaluable improvement in a design.

Done properly, design review is not an adversarial process. The designers must approach the review as an opportunity to learn; the reviewers must approach the review with an attitude of respect for the designers and for what they have done right. The key is to take a "bugs are good" attitude --- you know the bugs are there, and you're happy to find them so you can exterminate them.


Design reviews will be done in groups. If you are reviewing a design done in assignment N, then you'll work in the same group in which you did assignment N. At the beginning of a design review period, we will email each group an assignment, telling them which group's design they will be reviewing.

We will make a group's handed-in code and report available to their reviewers. Reviewers should read these carefully, and the reviewers should discuss them briefly among themselves before the review meeting.

The reviewers and designers must pick a mutually agreeable time and place for their face-to-face meeting. When you schedule your meeting, please email the time and location to the instructor. The instructor will attend some of the meetings to observe the design review process at work. The meeting should last at least a half hour, and no more than an hour.

After the meeting, the reviewers must write a report summarizing their conclusions regarding the design. The report should be frank about the design's good and bad points, while being written in a tone respectful of the designers' efforts. Where possible, the review should suggest specific ways to improve the design.

As always, the group of reviewers should write and submit a single, joint report, with all of their names clearly listed.

Grading the Reports

We will grade the reports based on our evaluation of how useful they would be to the designers in understanding the results of their work and how to improve their design in the future.

In order to encourage frankness in the design reviews, we will maintain a "Chinese wall" policy to ensure that the results of design reviews do not influence the grades we give for the original designs. In other words, if assignment N is a design and assignment N+1 is a review of that design, we will make sure that your grade on assignment N is completely determined before we look at what your reviewers wrote in their report for assignment N+1. Because of this rule, you can provide constructive criticism in your design review without worrying that your criticism is undermining anybody's grade.


Here is some advice about how to do a good design review.

You have limited time in the face-to-face meeting, so try to use it wisely. Review the documents in advance so you don't spend meeting time learning things that you could have gotten from the documents.

The review team might want to meet briefly before you meet with the designers, to help each other understand what is in the documents, and to discuss what questions you want to ask in the main meeting. This kind of pre-meeting works well if it is held immediately before the main meeting; then everything is fresh in your mind when you need it.

Try to focus your attention on the hard problems and tough design choices that the designers had to make. Doing this will focus your attention on the places where mistakes are most likely, and will give you the best opportunity to find mistakes or to notice something clever that the designers did.

Copyright 2000-2004, Edward W. Felten.  All rights reserved.