Cost Risk Analysis Data and Methods
A risk and uncertainty analysis can be conducted by a three-point estimate methodology, a risk driver approach, or a combination of both.
Cost Risk Analysis with Three-Point Estimates
Three-point estimates typically use a minimum, most likely, and maximum34 to describe the range of possible costs for each element. To capture cost estimate risk and uncertainty, analysts may collect various estimates from subject matter experts, use data from actual program performance, or use historical data from similar programs. Table 12 shows a traditional approach to cost risk analysis with three-point estimates applied directly to the WBS elements of a nominal cost estimate for the air vehicle of an Unmanned Aerial Vehicle (UAV) program. In an actual cost risk analysis, these estimates would be developed from a combination of historical data and in-depth interviews with persons who are knowledgeable in each area of the program.
Table 12: Ranges of Cost by WBS
| WBS element | WBS description | Minimum cost ($K per air vehicle) | Most likely cost ($K per air vehicle) | Maximum cost ($K per air vehicle) |
|---|---|---|---|---|
| 1.1 | Airframe | 17,955 | 19,645 | 21,785 |
| 1.2 | Propulsion | 8,500 | 10,553 | 13,000 |
| 1.3 | Communications | 4,000 | 4,306 | 5,100 |
| 1.4 | Navigation | 4,100 | 4,305 | 4,650 |
| 1.5 | Central Computer | 550 | 600 | 650 |
| 1.6 | Software | 32,300 | 35,547 | 52,500 |
| 1.7 | Integration, Assembly, test and check-out | 1,950 | 2,136 | 2,600 |
Source: GAO. | GAO-20-195G
For this example, the UAV program is procuring 32 air vehicles. Summing the most likely costs yields an estimate of $77 million per air vehicle, which results in a point estimate of $2.47 billion for all 32 air vehicles.
To model the risks in the simulation, the risks are typically represented as triangular distributions, but other distributions can also be used. Choosing the right probability distribution for each WBS element is important for accurately capturing risk and uncertainty. For any WBS element, selecting the probability distribution should be based on how effectively it models actual outcomes. Because different distributions model different types of risk, knowing the shape of the distribution helps to visualize how the risk will affect the overall cost estimate uncertainty. (Appendix X gives a variety of probability distribution shapes that are used for modeling cost risk.) Thus, the shape of the distribution should be determined by the characteristics of the risks they represent.
For a cost estimating relationship (CER), it is a best practice to use prediction interval statistical analysis to determine the bounds of the probability distribution because it is an objective method for determining variability. The prediction interval captures the error around a regression estimate and results in a wider variance for the CER. A CER input may also be uncertain and have a probability distribution that describes its range.
Once the distributions have been established, a statistical simulation—typically a Monte Carlo simulation—uses random sampling to select specific costs from each WBS element probability distribution, and a new program cost estimate is calculated.35 The simulation repeats this random selection thousands of times, creating a new program cost estimate with each iteration. The resulting probability distribution displays the range of possible program costs along with their confidence levels, as seen in figure 15.
Figure 15: Air Vehicle Cost Cumulative Probability Distribution from a Three-Point Estimate
Figure 15 shows that once risks and uncertainty are accounted for, the program cost at the 50 percent confidence level is $2.61 billion, which is $140 million higher than the point estimate of $2.47 billion. The cumulative distribution shows that the likelihood is about 15 percent that the program will cost $2.47 billion or less given the costs and the risk ranges used—an optimistic estimate in light of the risks to the program. Conversely, a program manager that desired an 80 percent confidence level would budget for a program cost of $2.79 billion, or about $320 million more than the point estimate.
One disadvantage of using three-point ranges to represent all the risk in an analysis is that probability distributions for costs cannot be attributed to individual risk events. It can be difficult to know or quantify the multiple risks inherent in the historical data. Similarly, it is difficult to quantify the risks in interviewees’ inputs. For the example in table 12, a program manager may suggest a worst-case scenario of $22 million per airframe. However, the higher cost may be caused by lack of materials, poor labor productivity, or some combination of those risks. It is also possible that an interviewee has increased the pessimistic estimate to account for general uncertainty in addition to discrete risks. Therefore, the result of the three-point method is a recommended amount of cost contingency that covers both quantified risks and some level of uncertainty, but gives no indication which risks are most likely to affect the program cost.
The minimum and maximum values may not be the actual minimum and maximum of the possible range of costs. Expert opinion has been shown to represent only 60 percent to 85 percent of the possible outcomes, so cost estimators sometimes make adjustments to account for a wider range. Further discussion is included later in this chapter.↩︎
The most common technique for combining the individual elements and their distributions is the Monte Carlo simulation. In the simulation, the distributions for each cost element are treated as individual populations from which random samples are taken. A Monte Carlo simulation randomly generates values for uncertain variables multiple times to simulate a model.↩︎