Sensitivity Analysis in Cost Estimating
Typically performed on high-cost and high-risk elements, sensitivity analysis examines how the cost estimate is affected by a change in a parameter or assumption. For example, it might evaluate how the point estimate varies with different assumptions about system reliability values, or how costs vary in response to additional system weight growth.
Factors that have the most effect on the cost estimate warrant further study to ensure that the best possible value is used. This analysis helps assure decision-makers that sensitive parameters and assumptions have been carefully investigated and the best possible values have been used in the final point estimate. If the cost estimate is found to be sensitive to several factors, the estimate’s input values and underlying assumptions should be reviewed.
A sensitivity analysis can provide useful information for the system designer because it highlights elements that are cost sensitive. In this way, sensitivity analysis can be useful for identifying areas where more design research can result in less production cost or where increased performance can be implemented without substantially increasing cost. This type of analysis is typically called a what-if analysis and is often used for optimizing cost estimate parameters and assumptions.
Sensitivity analysis also helps decision-makers choose a program alternative. For example, it can help a program manager determine how sensitive a program is to changes in gasoline prices and at what gasoline price a program alternative is no longer attractive. Using information from a sensitivity analysis, a program manager can take certain risk mitigation steps, such as assigning someone to monitor gasoline price changes, deploying more vehicles with smaller payloads, or decreasing the number of missions. It can provide important information for an analysis of alternatives that may result in the choice of a different alternative from the original recommendation. This can happen because, like a cost estimate, an analysis of alternatives is based on assumptions and constraints that may change. Thus, before choosing an alternative, it is essential to test how sensitive the ranking of alternatives is to changes in factors. In an analysis of alternatives, sensitivity is determined by how much a parameter or assumption must change to result in an alternative that differs from the one recommended in the original analysis. For example, a parameter is considered sensitive if a change of 10 percent to 50 percent results in a different alternative; it is considered very sensitive if the change is less than 10 percent.27
A sensitivity analysis provides a range of costs that span a best and worst case spread. In general, it is better for decision-makers to make a decision based on a range of potential costs that surround a point estimate—with the reasons behind what drives that range—than a point estimate alone. Sensitivity analysis can provide a clear picture of both the high and low costs that can be expected, with discrete reasons for what drives them. Figure 11 shows how identifying sensitivity can provide decision-makers with valuable insight.
Figure 11: A Sensitivity Analysis that Creates a Range around a Point Estimate
Figure 11 illustrates how certain assumptions affect the estimate. For example, increasing the quality of materials in the aircraft has the biggest effect on the cost estimate—adding $1.668 million to the point estimate while using a learning curve of 88 percent instead of 91 percent reduces the estimate by $60 million. Using similar visuals can quickly explain what-if analyses that can help management make informed decisions.
As shown in figure 11, sensitivity analysis makes for a more traceable estimate by providing ranges around the point estimate, accompanied by specific reasons for why the estimate could vary. This insight allows the cost estimator and program manager to further examine specific factors that could be potential sources of risk and to develop ways to mitigate them early. Sensitivity analysis permits decisions that influence the design, production, and operation of a system to focus on the elements that have the greatest effect on cost.
The following case study demonstrates how the lack of a sensitivity analysis affects a cost estimate for the Joint Intelligence Analysis Complex.
DOD’s Joint Intelligence Analysis Complex (JIAC), which provides critical intelligence support for the U.S., European, and Africa Commands and U.S. allies, is located in what DOD has described as inadequate and inefficient facilities at RAF Molesworth in the United Kingdom. To address costly sustainment challenges and instances of degraded theater intelligence capabilities associated with the current JIAC facilities, the Air Force planned to spend almost $240 million to consolidate and relocate the JIAC at RAF Croughton in the United Kingdom. GAO was asked to review the analysis associated with consolidating and relocating the JIAC and assess the extent to which DOD’s cost estimate for the JIAC consolidation at RAF Croughton aligned with best practices.
GAO assessed the cost estimate for the military construction project to consolidate and relocate the JIAC and found that it partially met three and minimally met one of the four characteristics of a reliable cost estimate defined by GAO best practices. For example, it minimally met the credibility standard because it did not contain a sensitivity analysis; such analyses reveal how the cost estimate is affected by a change in a single assumption. Without a sensitivity analysis to reveal how a cost estimate is affected by a change in a single assumption, the cost estimator cannot fully understand which variable most affects the cost estimate.
The use of a sensitivity analysis was not specified in cost estimation guidance for MILCON projects by either DOD or the Air Force. According to Office of the Secretary of Defense and Air Force officials, a sensitivity analysis is part of the underlying unit cost development, because costs are developed through the use of both historical data and industry averages. Officials further stated that the Office of the Secretary of Defense used actual data underpinned by relevant sensitivity and range analyses to develop its cost estimates. For example, Office of the Secretary of Defense and Air Force officials said that the Office of the Secretary of Defense used the DOD Selling Price Index—which averaged three commonly accepted national indexes for construction price escalation—to calculate actual project award cost data. However, for sensitivity analysis to be useful in informing decisions, careful assessment of the underlying risks and supporting data related to a specific MILCON project is also necessary. In addition, the sources of the variation should be well documented and traceable. Without conducting a sensitivity analysis to identify the effect of uncertainties associated with different assumptions, DOD and the Air Force increased the risk that decisions would be made without a clear understanding of the effects of these assumptions on costs.See appendix XI; best practice 17, perform sensitivity analysis.↩︎