LESSON 12

Peer Reviews

A peer review is a formal or informal method used by development groups for sharing techniques and expertise, for recognizing potential problems early in the design cycle, and for appraising product usability.

A peer review is never a management tool for evaluating personnel or affixing blame. It does not replace the initial design review, for which an interdisciplinary team is assembled to judge whether a design meets its written requirements. It also does not replace the design change review, which also uses interdisciplinary meetings to alert others of planned design changes.

Today’s first article, called "Bringing Engineering Disciple to Game Development - How We Approached an Engineering Discipline," describes how a game development company uses peer reviews. (The applicable text is included in this email.)

The second article, "Peer Reviews," explains how to set up a peer review process.

Two additional Web site links are included for reference. First, the Software Engineering Institute provides downloadable articles on software development reviews, management practices, and technical practices. The final article details how the United States Navy’s Space and Naval Warfare Systems Center conducts peer reviews.
--------------------------------------------------

Bringing Engineering Disciple to Game Development

This article gives an overview of a game company’s efforts to reengineer their software development process.

“Kesmai Corporation, through its GameStorm, ARIES Online Games, and Kesmai Studios business units, is the world's premier developer, publisher, and retailer of multiplayer online games.” - Kesmai Corporation

In this article’s introduction, the manager of Kesmai’s development studios states, “A year ago, we decided to change radically our methods of developing games. We were dissatisfied with our maintenance costs, the effort required to achieve our desired level of quality, and our inability to predict production schedules.”

Kesmai considered several existing development structures but found that they “contained elements that could stifle creativity and ignored the importance of having fun software...”

This article discusses how the company stressed three points to achieve their goals:
  1. programming standards
  2. peer review process
  3. software process working group

Note:  For the online description of Kesmai's success with peer reviews, scroll down in the online article and read the three paragraphs just below Figure 1, starting with “Peer Review Process.”

Or, the Peer Review Process text follows here.


****************************
Why Use Paper Prototypes?

"The definition of the term "peer review" is encapsulated in this quote from the SEI Capability Maturity Model: "The purpose of peer reviews is to remove defects from the software work products early and efficiently. An important corollary effect is to develop a better understanding of the software work products and of defects that might be prevented." Peer reviews, in this context, have no impact on personnel evaluation at all; they're simply a method of process improvement - never a methodology for placing blame. This peer review process only took a few weeks to hammer out, and is now embraced by our developers. Reviews go on almost every week in our studio. The primary value of peer reviews is in finding problems and potential problems long before they would normally be found in the testing stage. A less obvious but equally powerful benefit is that all participants in the peer review process learn to avoid both common and subtle errors in their own work. We encountered some resistance to peer reviews, because there was concern that management would use these reviews to rate our developers. Once it was clear that this was not the case though, our developers embraced the process.

“Our first peer reviews found significant problems that had defied other debugging methods, and every peer review we've conducted so far has found and corrected problems. As our people gain more experience with peer reviews, they're uncovering problems earlier in the development process - bugs might otherwise have taken weeks of debugging had they still existed when the game was in beta testing.

“When used effectively, peer reviews are a powerful tool to improve both the quality and the effectiveness of your development staff. The relative cost of finding and correcting errors early in the development of a title is incredibly low compared to trying to correct bugs when you're about to ship. If you don't already conduct peer reviews, I recommend starting today.”
*******

For the entire article see:

Bringing Engineering Disciple to Game Development - How We Approached an Engineering Discipline
http://www.gamasutra.com/features/production/19981218/eng_disc_02.htm
--------------------------------------------------

Establishing a Peer Review Process - SEI’s Guidelines

In this 1999 draft article, Carnegie Mellon University’s Software Engineering Institute (SEI) provides a step-by-step process for conducting peer reviews. A quick scan of the article will acquaint you with the breadth of the information available. Topics include goals, management commitment, resources, responsibilities, and activities.

If you are setting up a peer review process, use this article as a reference to provide you with ideas for your own system.

The article’s headings are:

GOALS
--defined process
--peer reviews
--defect removal
COMMITMENT TO PERFORM
--policy
ABILITY TO PERFORM
--plan
--resources
--responsibility
--training
ACTIVITIES PERFORMED
--perform
--prepare
--conduct
--implement actions
--record data
MEASUREMENT AND ANALYSIS
--insight
VERIFYING IMPLEMENTATION
--process assurance
--product assurance
--senior management reviews
--project manager reviews

To read the online article, scroll down approximately two screens to the first table.  The text starts just below the table at the Peer Review heading.

Peer Reviews
http://www.sei.cmu.edu/activities/cmm/draft-c/c37prv.html
----------------------------

Reference Materials

1. Carnegie Mellon Software Engineering Institute
2. United States Navy’s Space and Naval Warfare Systems Center

*******

Carnegie Mellon Software Engineering Institute

What is the Carnegie Mellon Software Engineering Institute (SEI)?

From their Web site at http://www.sei.cmu.edu/ :

“The Software Engineering Institute (SEI) is a federally funded research and development center sponsored by the U.S. Department of Defense through the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics [OUSD (AT&L)]. The SEI contract was competitively awarded to Carnegie Mellon University in December 1984. It is staffed by technical and administrative professionals from government, industry, and academia.

“Mission:  The U.S. Department of Defense established the Software Engineering Institute to advance the practice of software engineering because quality software that is produced on schedule and within budget is a critical component of U.S. defense systems. The SEI mission is to provide leadership in advancing the state of the practice of software engineering to improve the quality of systems that depend on software. The SEI accomplishes this mission by promoting the evolution of software engineering from an ad hoc, labor-intensive activity to a discipline that is well managed and supported by technology.

“Principal Areas of Work:  The SEI carries out its mission through two principal areas of work:  management practices and technical practices.”

Use the SEI site map to access their many products and services.

Carnegie Mellon Software Engineering Institute
http://www.sei.cmu.edu/about/website/sitemap.html
--------------------------------------------------

United States Navy’s Space and Naval Warfare Systems Center

The United States Navy’s Space and Naval Warfare Systems Center conducts peer reviews of their products.

Their Peer Review Process article states:  “The objective of a peer review is to involve a group of people (peers) to examine part or all of the product to discover potential problems. The results of the peer review are either informally or formally recorded and passed on to the author or person responsible for correcting the discovered problem. This document describes three types of Peer Reviews; (1) Walkthroughs, (2) Technical Reviews, and (3) Formal Inspections.”

This thorough and easy-to-read [!] source reference provides many ideas for conducting peer reviews. (Flowcharts and procedures suggest detailed controls that are often characteristic of a military development center, so tread lightly!)

From the Peer Review Process Table of Contents -------------

PROCESS DESCRIPTION
Planning Phase
--Assign Roles and Responsibilities
--Conduct Planning Activities
--Select Type of Peer Review for Product
Implement Selected Peer Review Procedures
--Walkthrough
--Technical Review
--Formal Inspection (FI)
Measurements of Peer Review Process
Review and Audit of Peer Reviews

Peer Review Process
http://sepo.spawar.navy.mil/sepo/docs.html   (Click the Peer Reviews link for additional links to MS Word documents or PDF documents.)