Software Reviews. Fernando Brito e Abreu Universidade Nova de Lisboa (

(1)

Software Reviews

Fernando Brito e Abreu ([email protected]) Universidade Nova de Lisboa (http://www.unl.pt)

QUASAR Research Group (http://ctp.di.fct.unl.pt/QUASAR)

ABSTRACT

WHAT IS A REVIEW ?

WHY REVIEWS ?

DEFECT CLASSIFICATION SCHEDULE AND DURATION

¡ REVIEW TYPES

¢ WALKTHROUGHS

£ INSPECTIONS

¤ THE FINAL REPORT

¥ DISCUSSION

(2)

What is a Review ?

An evaluation of any deliverable done by other team members besides the author

A validation technique (e.g.: detection of discrepancies with the requirements spec)

A verification technique (e.g.: detection of non conformities with adopted standards)

Why Reviews ?

To counteract against perceptive mismatch

Inability to deal with detailed information chunks and loosely defined complex interactions

George Miller, "The Magical Number 7 ± 2 : Some Limits in our Capacity for

Processing Information", Psychological Review, n.63, p.81-97, 1956.

(3)

Why Reviews ?

To share responsibility

In most Engineering areas (e.g. Civil or

Mechanic) projects are signed both by author and reviewer (thus becoming co-responsible)

Why Reviews ?

Defects found earlier cost less to remove

Better visibility and control of the development process

Continuous learning

Positive impact in teamwork

Easier maintenance

Less harm when somebody leaves a project

Considerable reduction of testing effort and

(4)

Why Reviews?

Reviews are complementary to tests

Examples where reviews can be used and tests cannot:

To V&V non-executable deliverables

e.g. specifications, models, user manuals, incomplete versions of source code, …

To assess several quality attributes

e.g. adherence to standards, maintainability, inefficient algorithm, poor programming style, …

Schedule and Duration

Schedule:

Immediately after the conclusion of a given stage of an identified deliverable

Included in Project Plan and done regularly and repetitively

No immunity!

Duration

Depends on deliverable dimension and complexity

Hint: a fixed percentage of the time spent developing it

[Britcher88] proposes they should not exceed 2 hours

Short reviews (e.g.: 1 h), done often, are preferable

(5)

What to Review?

All kinds of deliverables are eligible

If possible we should review everything

but often we cannot afford that extra effort

therefore we must select a sample

Sampling has a statistic connotation

How to sample the material to review?

What to Review?

Shall we select:

a random sample (like in other areas of industry?

a representative sample

of best practices?

of average practices?

of worst practices?

The aim is to maximize review efficiency!

Complexity metrics can (should) be used ...

(6)

Reviews Types

REVIEWS Informal Formal Internal Walkthroughs Inspections External Demonstrations Audits

Walkthroughs

Revisions in which one participant makes a description of the analyzed deliverable

Usually the presenter is the author (not compulsive)

All participants try to identify defects and non conformities

No distinct role is assigned to participants (except presenter)

Detail and speed depend on presenter preparation

This type of review should not be long, therefore also not the sample being reviewed

(7)

Walkthroughs

Bad aspects

Lack of formality often leads to uncontrolled situations

Less effectiveness and efficiency compared to Inspections

Defects found are often neither classified nor recorded

Good aspects

Small cost (no preparation time is required ...)

Not much intrusive

Applicable in small organizations

Can bootstrap Inspections!

Inspections (Peer Reviews)

Initially proposed by M. E. Fagan at IBM [Fagan76]

and matured during a decade of usage [Fagan86]

Effective technique for defects detection

Formal review technique with well-defined input and output requirements

(8)

Inspections

A group of participants is nominated

Deliverable to review, as well as related others, must be made available beforehand (e.g.: 3 or 4 days)

Participants must be familiar with inspections procedures

specific training is required to be able to participate

Each participant has a well defined role:

moderator, reader, producer and recorder

Inspections - Moderator

Directs the meeting, registering attendance and individual preparation times

Avoids deviations during the inspection (e.g.:

tendency to propose solutions)

Avoids or solves conflicts

Must adopt a non authoritarian role

Registers the total meeting duration

Submits results to the defect tracking system

Deliberates about the meeting follow-up

(9)

Inspections - Reader

Divides (before the meeting) the reviewed deliverable in small sections for presentation purposes

Describes each section, with adequate granularity, during the meeting

Guarantees the inspection pace, adopting a rhythm, neither too fast, nor too slow

Example:200 documented LOCs per hour

Inspections - Producer (author)

Assures that the deliverable to be reviewed, as well as others inter-related ones, are available for all participants in advance

During the meeting will contribute with his/her particular knowledge of the deliverable being reviewed to solve emerging doubts

After the meeting must guarantee that defects found are removed (other person can be assigned)

(10)

Inspections - Recorder

Responsible for the synthesis of opinions expressed by all participants

Registers the following:

physical location of defects found (note: that is why all material should be printed with line numbers)

description of defects

category of each defect (from checklist)

defects root cause (if identified)

Inspections - All participants

Try to identify defects, conformance to internally adopted standards and synchronic traceability before the meeting

Only report a given defect when the reader goes through the corresponding section of the

deliverable being reviewed

Help to classify and identify the root cause of defects found

Sign the participation document

(11)

Inspection role accumulation

Reader Recorder Author

Moderator

Yes Yes No

Reader

No No

Recorder

No

The Final Report

In all types of review a final report must be produced. It should include:

detected defects

defects classification

meeting date

meeting duration

participants

total preparation time

estimated time/effort for detected defects removal

(12)

The Final Report (continued)

Plays a very important role for all participants in the development process:

project manager

client

programmer

quality assurance team

Data collection and report generation should be computer supported!

Inspections - Gold Rules

Inspections are for peers. Upper management representatives should not be present!

Solution for defects removal should not be sought!

The product, not the producer, is being reviewed!

Checklists should be used for defect mining (e.g.

JavaCheck)

Inspection team becomes, as a whole, co- responsible for the quality of the deliverable

Remaining defects must be assumed by the team

(13)

Inspection Costs

FIXED

Planning (schedule/staff)

Setting sampling criteria

Setting defect and fault classification scheme

Form development

Building / selecting tool

Training

FOR EACH MEETING

Organizing effort

selecting “sample”

contacting participants

Meeting room

Preparation effort

Meeting effort

Reporting defects found

Correction effort (?…)

Inspection Benefits (revisited)

Better teamwork / motivation

Exchange of experience / knowledge between the participants (acts as actual training)

Identification of specific training needs

Break in the monotony (by exchanging roles)

Incentive to increase performance (have less errors than the average)

Promotion of a Quality Culture

(14)

Inspection Benefits (revisited)

Standards enforcement

Reduction in defect rate and in V&V costs

Increased understanding / visibility of ongoing projects (may improve deadline meeting

performance by early detection of problems)

Reduction of personal turnover

Database of inspection results - allows to base defect prevention actions (causal analyses)

Several of these benefits are not tangible!

Assessing cost vs. benefits

The error amplification factor [IBM81]

Error – quality problem detected before the software is released to the client

Defect – quality problem detected after the software is released to the client

Rationale: design activities introduce between 50% and 65% of all errors

Formal design reviews have been shown to

detect up to 75% of those errors, reducing

the costs in subsequent phases

(15)

Error amplification model

Errors Passed Through

Amplified Errors 1:x

Newly Generated Errors

Efficiency of Error detection Errors from the

previous step

Errors passed to the next step Errors Error Detection

Relative cost Phase

1,5 Analysis &

Design

6,5 Code / Modular

Testing

15 Pre-installation

/ Integration testing

67 Production

Errors, with and without revisions

0

10 70%

2

1*1,5

25 50%

5

10*3

25 60%

24

0

0 50%

12

0

0 50%

6

0

0 70%

3 2

1 15

5

10

24 12 6

3

0

10 0%

6

4*1,5

25 50%

10

27*3

25 20%

94

0

0 50%

47

0

0 50%

24

0

0 50%

10 6

4 37

10

27

94 47 24

12 Preliminary

design

Detailed design

Code / Unit test

Integration test

Validation test

System test

Preliminary design

Detailed design

Code / Unit test

Integration test

Validation test

System test

(16)

Cost/Benefit ratio

(with and without reviews)

783 82

2177 116

Totals

201 3

804 12

67,0 Production

315 21

1230 82

15,0 Pre-installation/Integration

testing

234 36

143 22

6,5 Coding / Modular testing

33 22

1,5 Analysis and Design

Cost

# Def.

Cost

# Def.

defect PHASE

REVIEWS WITH

REVIEWS NO

Cost p/

Note: a cost reduction of 64% occurred!

Source: [Pressman97]

Inspection effectiveness (%)

[Laitenberger98]

57,9 55,1

43,4 Wood et al.

33,5 53,4

50,3 Kamsties & Lott

(2nd replication)

36,0 37,0

40,0 Kamsties & Lott

(1st replication)

41,2 54,6

54,1 Basili & Selby

38,0 36,0

30,0 Myers

46,7 47,7

37,3 Hetzel

Structural testing Functional

testing Inspections

Authors

(17)

The Human Factor

As all other Software Quality Improvement activities and techniques, reviews may be:

considered as a threat to individual ”freedom”

banned with the excuse of being an overhead

Counter measures:

Publicize defect efficiency numbers asap!

Recognize and reward the work of best reviewers!

Variants / Critics to Fagan

Inspections

Simple Review (author plus another)

usually adopted in other fields of Engineering

Active Design Reviews [Parnas87]

group meetings can be skipped (lower cost)

Meeting Gain (MG) must be calculated:

MG = Defects found during meeting / All defects found

(18)

Variants / Critics to Fagan

Inspections

Phased Inspections - using InspeQ toolset

[Knight91]

Department of Computer Science / University of Virginia

Cost-Benefit assessment of alternatives

[McCarthy96]

ATT Bell Labs/University of Maryland experiments

Inspection alternatives:

PI: Preparation-Inspection (1st round is for understanding)

DC: Detection-Collection (1st round emphasis on detection)

DD: Detection-Detection (2 rounds with no meeting)

DD alternative had the best cost-benefit ratio!

Controversy on cost-effectiveness

Structural factors have no significant effect on effectiveness

More reviewers don’t necessarily find more errors (2, or 3 reviewers should be enough)

Meetings are less cost effective, and provide a similar effectiveness, when compared to asynchronous (meetingless) inspections

Inspection effectiveness is best explained by reviewer expertise

Individual effects are more influential than group effects

Improved individual analysis methods significantly improve performance

(19)

Review Tools Web Links

InstantQA (http://www.reasoning.com/)

ReviewPro ( http://www.sdtcorp.com/)

CheckMate ( http://www.sybernet.ie )

Leap (http://csdl.ics.hawaii.edu/Tools/LEAP/LEAP.html)

CSRS ( http://www.ics.hawaii.edu/~csdl/csrs.html)

Assist

(http://www.cs.strath.ac.uk/research/efocs/assist.html)

CodeReviewer (http://www.codehistorian.com/)

Relevant standards

ANSI/IEEE Std 1012 : "Standard for Software Verification and Validation Plans"

ANSI/IEEE Std 1028 : "Standard for Software Reviews and Audits"

DoD Mil Std 1521 B : "Technical Reviews and Audits for Systems, Equipments and Computer Software"

(20)

Bibliography

⁽¹⁾

[Ackerman89] Ackerman, A. Frank &

Bushwald, Lynne S. & Lewski, Frank H. :

"Software Inspections: An Effective Verification Process", IEEE Software, 6(3), p.31-36, May 1989.

[Britcher88] Britcher, Robert N. : "Using Inspections to Investigate Program Correctness", IEEE Computer, p.38-44, November 1988.

[Collofello88] Collofello, James S. : "The Software Technical Review Process", Curriculum Module SEI-CM-3-1.5, Software Engineering Institute, Carnegie Mellon University, Pittsburgh, June 1988.

[Cross88] Cross, John A. (ed), "Support Materials for the Software Technical Review Process", Support Materials SEI-SM-3-1.0, Software Engineering Institute, Carnegie- Mellon University, Pittsburgh, April 1988.

[Fagan76] Fagan, Michael E. : "Design and Code Inspections to Reduce Errors in Program Development", IBM Systems Journal, 15(3), March 1976.

[Fagan86] Fagan, Michael E. : "Advances in Software Inspections", IEEE TSE, 12(7), p. 744- 753, July 1986.

[Freedman90] Freedman, Daniel P. & Weinberg, Gerald M. : "Handbook of Walkthroughs, Inspections and Technical Reviews: Evaluating Programs, Projects and Products" (3rd edition), Dorset House Publishing, NY, 1990.

[Gale90] Gale, J. L. & Tirso, J. R. & Burchfield, C. A. : "Implementing the Defect Prevention Process in the MVS Interactive programming organization", IBM Systems Journal, 29(1), p.33- 43, 1990.

[Gilb93] Gilb, Tom & Graham, Dorothy: “Software Inspection”, Addison-Wesley, ISBN 0-201-63181- 4, 1993.

[Hollocker90] Hollocker, Charles P. : "Software Reviews and Audits Handbook", John Wiley, New York, 1990.

Bibliography

⁽²⁾ ^[Parnas87] Parnas, D.L. & Weiss, D.M. : "Active Design Reviews: Principles and Practices", Journal of Systems and Software, nº7, p.259-265, 1987.

[Pressman97] Pressman, Roger S. :

"Software Engineering: a Practitioner's Approach" (4th edition), McGraw-Hill, 1997.

[Russell91] Russell, Glen W. :

"Experience with Inspection in Ultra large- Scale Developments", IEEE Software, 8(1), p.25-31, January 1991.

[Strauss94] Strauss, Susan H. &

Ebenau, Robert G., Software Inspection Process, McGraw-Hill, 1994.

[Votta93] Lawrence G. Votta,

“Does Every Inspection Need a Meeting?”, SIGSOFT'93 Symposium on Foundations of Software Engineering, ACM Press, 1993.

[Yourdon89] Yourdon, Edward : Structured Walkthroughs (4th edition), Yourdon Press, Prentice-Hall, 1989.

[IBM81] IBM Corporation, “Implementing Software Inspections,” course notes, IBM Systems Sciences Institute, 1981.

[Knight91] Knight,John C. & Myers, E. Ann : "Phased Inspections and Their Implementation", ACM Sigsoft, SW Engineering Notes, 16(3), p.29, July 1991.

[Laitenberger98] Laitenberger, Oliver:

“Studying the effects of code inspection and structural testing on software quality”, ISERN-Report ISERN-98-10, 1998.

[Mays90] Mays, R.G. & Jones, C.L. &

Holloway, G.J. & Studinski, D.P. :

"Experiences with Defect Prevention", IBM Systems Journal, 29(1), 1990.

[McCarthy96] Patricia McCarthy et al.“An Experiment to Assess Cost-Benefits of Inspection Meetings and their Alternatives: A Pilot Study”, 3rd Int. Software Metrics Symposium, IEEE, Berlin, March 96.