Collecting Risk and Uncertainty Data
High-quality data are central to a successful risk and uncertainty analysis. The validity of the results of the analysis depends on analysts’ knowledge of and experience with a program’s risks. If the risk and uncertainty analysis has been poorly executed or is based upon low-quality data, management may get a false sense of security that all risks have been accounted for and that the analysis is based on sound data. When this happens, program decisions will be based on bad information. Similarly, if cost estimators focus on only the risks that most concern the program office or contractor rather than a broad range of potential risks, program decisions may be based on poor quality information. Case study 18 illustrates the effects of a limited risk and uncertainty analysis and its subsequent improvement.
For the 2020 Census, the Census Bureau intended to limit its per-household cost to not more than that of the 2010 Census, adjusted for inflation. To achieve this goal, the Bureau significantly changed how it conducted the census, in part by re-engineering key census-taking methods and infrastructure. In October 2015, the Bureau estimated that with its new approach, it could conduct the 2020 Census for a life cycle cost of $12.5 billion, in contrast to its estimate of $17.8 billion to repeat the design and methods of the 2010 Census (both in constant 2020 dollars). In 2016, Congress asked GAO to evaluate the reliability of the life cycle cost estimate the Bureau submitted to Congress in October 2015. GAO reviewed (1) the extent to which the Bureau’s life cycle cost estimate met best practices for cost estimation; (2) the extent to which the Bureau’s key cost assumptions were supported by field tests, prior studies, and other evidence-based analysis; and (3) the extent to which the Bureau identified and accounted for key risks facing the 2020 Census within its risk and uncertainty analyses of its life cycle cost estimate.
GAO found that Census Bureau carried out a risk and uncertainty analysis for the 2020 Census life cycle cost estimate, but only for a portion of estimated costs for fiscal years 2018 to 2020. According to Bureau officials, they scoped the analysis narrowly to those 3 years when most of the census costs occur. GAO found that, as a result, the Bureau’s risk and uncertainty analysis covered $4.6 billion, only about 37 percent of the $12.5 billion total estimated life cycle cost, and less than one-half of the total estimated cost of the census during future fiscal years.
The Bureau used management discretion to determine how much contingency to add on top of the remaining costs. An additional 10 percent was added for fiscal years 2018 through 2020, for a total additional contingency of $829 million. However, officials were not able to justify the 10 percent factor and there was no Bureau documentation justifying the additional contingency. Because the Bureau only carried out its uncertainty analysis on a portion of the cost estimate, GAO could not determine if it fully identified the level of risk associated with the estimate. Nor could GAO validate the Bureau’s reported confidence level of the total life cycle cost estimate or how it related to the Bureau’s total contingency.
In 2018, GAO evaluated the reliability of the Bureau’s revised life cycle cost estimate for the 2020 Census and the extent to which the Bureau was using it as a management tool. GAO found that the Bureau had made significant progress in improving its ability to develop a reliable cost estimate. In particular, the Bureau improved its risk and uncertainty analysis methodology for the 2017 life cycle cost estimate. Bureau analysts used a combination of modeling based on Monte Carlo simulation and other methods to develop the contingency estimates. GAO found that the Bureau substantially met the best practice of risk and uncertainty analysis for the 2017 estimate. In addition, in 2018 the Bureau established roles and responsibilities for oversight and approval of cost estimation processes, created a detailed description of the steps that should be taken to produce a high-quality cost estimate, and clarified the process for updating the cost estimate and associated documents over the life of a project.
Because collecting data to support the risk and uncertainty analysis can be a formidable effort, it should be done when the data are collected to develop the point estimate, if possible. Potential sources of risk and uncertainty can include:
- Economic - labor rate assumptions and inflation indexes
- Cost estimating - learning curve assumptions, cost estimating relationship error, and optimistic savings from new processes
- Programmatic -lack of resources
- Requirements - changes in specifications, procurement quantities, and system architecture
- Schedule - testing failures, optimistic task assumptions, and procurement delays
- Technology - success of unproven technologies, optimistic reuse assumptions, and design changes
Historical data should be used to derive risk data when possible. However, risk data must often be derived from in-depth interviews or in risk workshops. When expert opinion is used for risk and uncertainty data, it is essential that subject matter experts (SMEs) who are directly responsible for or involved in the workflow activities be interviewed. Estimates derived from interviews should be formulated with a consensus of knowledgeable technical experts and should be coordinated with the same people who manage the program and its risk mitigation watch list. Employees involved in the program from across the entire organization should be considered for interviews. Lower-level employees have valuable information on day-to-day tasks in specific areas of the program, including insight into how individual risks might affect their workflow responsibilities. Managers and senior decision-makers have insight into all or many areas of the program and can provide a sense of how risks might affect the program as a whole.
The starting point for the risk interviews is the program’s existing risk register. Interviewees are asked to provide their opinions on threats and opportunities and should be encouraged to introduce additional potential risk events that are not on the risk register. If unbiased data are to be collected, interviewees must be assured that their opinions on threats and opportunities will remain anonymous. They should also be guaranteed non-attribution and should be provided with an environment in which they are free to brainstorm on worst and best case scenarios. It is particularly important to interview SMEs without an authoritative figure in the room to avoid motivational bias. Motivational bias arises when interviewees feel (whether justifiably or unjustifiably) uncomfortable giving their honest thoughts about a program. This typically results from fear of being penalized by someone in authority. Most commonly, interviewees are labeled trouble makers or are ostracized from the team if their worst case scenario is worse than management’s opinion. Risk workshops may exhibit social and institutional pressures to conform, perhaps to get consensus or to shorten the interview session. The organization may greatly discourage introducing a risk that has not been previously considered, particularly if the risk is sensitive or may negatively affect the program. If an interviewee is accompanied by someone, risk analysts cannot guarantee that the interviewee’s responses are unbiased.
One way to avoid the potential for experts to be success oriented when choosing the upper and lower extremes of the distribution is to look for historical data that back up the distribution range. If historical data are not available, it may be necessary to adjust the extremes to account for the fact that being overly optimistic usually results in programs costing more and taking longer than planned. Studies have shown that, at best, subject matter experts identify 70 percent of the possible uncertainty range. Thus, it is necessary to skew the tails of the probability distributions to account for this possibility in order to more accurately represent the overall risk. One method of accounting for this aspect of expert input involves assuming that the experts’ minimum and maximum represent the 15th percentile and the 85th percentile of the distribution and adjusting the distribution accordingly.
Organizations should develop and publish default distribution bounds that cost estimators can use when risk data are not objective or available. Furthermore, to ensure that best practices have been followed and to prevent errors, some experts suggest vetting the risk and uncertainty analysis through a core group of experts to ensure that results are robust and valid.