Rubric: Software Engineering : Factual Claims : Defect Cost Increase : Pressman Ratios
Background: I have been researching quantity and quality of empirical evidence underlying claims in software engineering. What do we know, and how well-established is that? See in particular https://leanpub.com/leprechauns which concludes that the answer is in (too) many cases "not much, and poor".
This applies in particular to the "Defect Cost Increase" claim, which is poorly supported by evidence. The claim states that the longer a defect stays undiscovered after being introduced in a software system's artefacts, the more expensive it is to correct.
The "Pressman Ratios" are a specific quantitative assessment of this claimed effect:
"Assume that an error uncovered during design will cost 1.0 monetary unit to correct. Relative to this cost, the same error uncovered just before testing commences will cost 6.5 units; during testing, 15 units; and after release, between 60 and 100 units."
We can compress this to "1:6.5:15:60-100", for ease of distinguishing with other variants of the same claim that differ only in the precise ratios.
The reference cited by Pressman is initially formatted as follows:
[IBM81] "Implementing Software Inspections," course notes, IBM Systems Sciences Institute, IBM Corporation, 1981
Evidence that the Systems Sciences Institute even existed is scant and hard to come by. A search of the IBM corporate web site yields nothing, for instance.
Using Google Scholar with search terms "IBM Systems Sciences Institute" (Scholar) yields a short list of people who seem to have been affiliated with the organization:
- Claude Wesley Burrill Burrill
- Arnold Oral Allen Allen Allen2
- Janice Richmond Lourie Lourie
- Thomas M. Yokoyama Yokoyama
- Paul R. Melichar Melichar
I am using middle names and middle initials where available to be allow specific Google searches and disambiguations, the usual citation style only has the last name in full, e.g. "A. O. Allen".
Quoting Claude W. Burrill:
The Systems Science Institute is an educational organization and is a unit of IBM. We offer a variety of advanced courses for management and for professionals in the computer field.
The Insitute was, in essence, an internal training program for employees at IBM.
Confusingly, several biographies of the people listed above associate them with the "IBM Systems Research Institute" (for instance, Burrill's obituary). A New York Times article describes it as follows:
(...) virtually all I.B.M. employees receive some kind of company-financed education or training each year beyond basic job training. The education might range from attendance at special lectures to full-fledged courses, inside or outside corporate facilities. Despite its other programs, I.B.M. believes it needs a graduate-level school for its own use. The Systems Research Institute, founded in 1960, is the closest thing to an academic center at I.B.M.. (...) But the Systems Research Institute is not an academic place. ''It's all business,'' said its associate director, Joseph E. Flanagan.
The IBM Systems Research Institute was based in the state of New York, with offices in Manhattan and Thornwood. (Giant, Origins, Thornwood)
The IBM Systems Science Institute was based in Los Angeles, at 3550 Wilshire Boulevard. (Address).
It started operations as early as 1967 (according to an IBM ad referring to its post-1982 name, see next paragraph). Started
It ceased to operate under that name before 1982, according to a different ad. (Rename) It became the "Information Systems Management Institute".
Another article will deal in more detail with the credibility of the Pressman Ratios, and how they are widely but generally inappropriately cited throughout the literature on software engineering.
The main findings of the present entry on the "IBM Systems Science Institute" are as follows:
- the Institute was a corporate training program, not a research body; as such it is inappropriate to cite the source of the ratios as "an IBM study" or "a study by the IBM Systems Science Institute", in the total absence of any claim that the Institute was the primary source;
- the original project data, if any exist, are not more recent than 1981, and probably older; and could be as old as 1967.
References
Yep, that's one of the ur-papers that you get to if you chase citations around long enough, the "telephone game" pattern I explore at length in Leprechauns.
But it has its problems - some of the data was obtained from student projects Boehm was supervising, for instance. Another problem is that you still have to check the papers cited by Boehm to follow the "chain of data custody" - Boehm's towering reputation notwithstanding.
To pick just one (I have a more thorough survey in the book) the source of the GTE data is a paper by Daly (paywalled by the usual suspects, but Sci-Hub is your friend), the only relevant portions of which are shown below:
As you can see from the figure, the categories of interest are labeled differently. Daly was comparing "reading code" to "machine debugging", not phases. Also importantly, Daly was talking about the cost incurred to find a bug, not the time to fix one. And in the 1970s the primary cost of "machine debugging" would have been the cost of running the computer, which tended to dwarf the labor costs of the human programmers.
Anyway, it's not clear on what basis it was decided that "Eyeball" corresponds to "Requirements", "Prototype system" to "Development test", and "Post cutover" to "Operation" - other than Boehm wanting to shoehorn the data into his model. (And what does "eyeball" even mean in this context ? Not clear.)
Finally, it's somewhat obvious that Boehm made up the error bars around the log-graph data point which stands for "fifteen" - why would he do that, other than to lend an air of statistical legitimacy to what was in fact a wet-finger-in-the-air approximation from Daly? The intellectually honest thing to do here isn't to draw fake error bars, it's just to plot the damn points, quote the small paragraph above from Daly, and warn the reader about the approximate nature of the equivalence being made. Yes, that adds half a page to the paper, but that's how scientific publications are supposed to be, long and painstakingly detailed and rigorous.
But Boehm's paper is not science, it's a marketing brochure, selling a new-fangled concept called "Software Engineering". So it has to be short and convincing, and scientific accuracy is a secondary concern.