Chapter 12: Step 9: Conduct Risk And Uncertainty Analysis
In the previous chapter, we discussed sensitivity analysis and how it is useful for identifying cost drivers by determining how sensitive the estimate is to changes in input parameters, developing ranges of potential costs, and performing what-if analyses. By understanding which input variables have a significant effect on a program’s final costs, management can devote resources to acquire better knowledge about those inputs to respond to their risks. But while sensitivity analysis measures the effects of changing one parameter at a time, in reality, many parameters can change at the same time. Quantitative risk and uncertainty analysis uses statistical techniques to predict the probability of successfully executing a program within its budget by capturing the cumulative effect of program risks and uncertainty.
A risk and uncertainty analysis is one way to ascertain whether a program is realistically budgeted because it can determine the probability associated with achieving the cost estimate for the program. The analysis provides a way to assess the variability in the estimate by quantifying cost, schedule, and technical risks. A cost estimator can model such effects as changing technical parameters, schedule delays or accelerations, labor productivity, and changing missions, thus creating a range of potential costs. A range of costs is more useful to decision-makers than a point estimate because a range helps them better understand program risk.
A risk and uncertainty analysis requires the cost estimating team to collect program risk data.29 Risk data should be derived from a quantitative risk assessment and should not be based on arbitrary percentages or factors. A risk assessment is a part of the program’s overall risk management process in which risks are identified and analyzed and potential consequences are determined. As risks are identified and prioritized, risk response plans are developed and incorporated into the program’s cost estimate and schedule, as necessary. Ultimately, management needs to understand that a risk and uncertainty analysis is only as good as the comprehensiveness of risks and uncertainties quantified at a point in time. Without a risk and uncertainty analysis, the program estimate will not reflect the degree of uncertainty and a level of confidence cannot be given about the estimate. Unless a range of costs is provided, decision-makers will lack information on cost, schedule, and technical risks, and will not have insight into the likelihood of executing the program within the budget.
We have found that budgeting to a risk-adjusted estimate is critical to successfully achieving a program’s objectives. Programs have developed overly optimistic estimates with narrow uncertainty ranges for many reasons: cost estimators may have minimized program risk, ignored data outliers, relied on historical data that may not have been representative of a new technology, or assumed higher productivity than what had previously been achieved. In addition, decision-makers may influence the estimate with bias for political or budgetary reasons. For example, they may assume that a new program will perform much better than its predecessor in order to fit the program within an unrealistic budget, or just to sell the program. To counter over-optimism and bias, a risk analysis must consider all risks and uncertainty as completely and objectively as possible. Case study 17 provides an example of performing an inadequate risk and uncertainty analysis.
Case Study 17: Incomplete Risk and Uncertainty Analysis, from Coast Guard Acquisitions, GAO-18-600
To maintain heavy polar icebreaking capability, the Coast Guard—a component of the Department of Homeland Security (DHS) — and the Navy are collaborating to acquire up to three new heavy polar icebreakers through an integrated program office. The National Defense Authorization Act for Fiscal Year 2018 included a provision for GAO to assess issues related to the acquisition of the icebreaker vessels. In addition, GAO was asked to review the heavy polar icebreaker program’s acquisition risks.
GAO found that the Coast Guard did not have a sound business case in March 2018 when it established the cost, schedule, and performance baselines for its heavy polar icebreaker acquisition program, because of risks in four key areas, one of which was the program cost estimate. The life cycle cost estimate that informed the program’s $9.8 billion cost baseline substantially met GAO’s best practices for being comprehensive, well documented, and accurate, but only partially met best practices for being credible. In particular, the cost estimate failed to quantify the range of possible costs over the entire life of the program.
GAO found that the Navy only modeled cost variation in the detail design and construction portion of the program, and excluded from its analyses any risk impacts related to the remainder of the acquisition—operating and support and disposal phases—which altogether comprised about 75 percent of the life cycle cost. Without performing a risk and uncertainty analysis on the entire life cycle cost of the three ships, it was not possible for the Navy to determine a level of confidence associated with the overall cost estimate. By not quantifying important risks, the Navy may have underestimated the range of possible costs for about three-quarters of the entire program.
The term “risk data” comprises the parameters and data sets that are used in performing a risk and uncertainty analysis. These data are both cost and non-cost and may include costs, durations, performance parameters, and requirements, among other types. Risk data also include statistics and factors used to define probability distributions used in simulations.↩︎