Collecting Anonymous and Unbiased Risk Data
A schedule risk analysis requires the collection of program risk data. Risk data should be derived from a quantitative risk assessment and should not be based on arbitrary percentages or factors. A risk assessment is a part of the program’s overall risk management process in which risks are identified and analyzed and the program’s risk exposure is determined. As risks are identified, risk-handling plans are developed and incorporated into the program’s cost estimate and schedule, as necessary.33
Risk data can be collected in the form of three-point durations or by using the risk driver approach, to be described in the next section. The three-point estimates represent inherent uncertainty, estimating error, and perhaps estimating bias, while risk drivers represent identified risk events with probabilities as well as the likely effect if they occur. Regardless of which type of risk analysis is performed, it is essential that subject matter experts (SME) be interviewed who are directly responsible for or involved in the workflow activities. Estimates derived from interviews should be formulated in a consensus of knowledgeable technical experts and should be coordinated with the same people who manage the program and its risk mitigation watch list. Employees involved in the program from across the entire organization should be considered for interviewing. Lower-level employees have valuable information on day-to-day tasks in specific areas of the program, including their insight into how individual risks might affect their workflow responsibilities. Managers and senior decision makers have insight into all or many areas of the program and can provide a sense of how risks might affect the program as a whole.
The starting point for the risk interviews is the program’s existing risk register. Interviewees are asked to provide their opinion on threats and opportunities and should be encouraged to introduce additional risk events that are not on the risk register. If unbiased data are to be collected, interviewees must be assured that their opinions on threats and opportunities will remain anonymous. They should also be guaranteed nonattribution and should be provided with an environment in which they are free to brainstorm on worst and best case scenarios. It is particularly important to interview SMEs without an authoritative figure in the room to avoid motivational bias.
Motivational bias is a source of bias that arises when interviewees feel threatened (whether justifiably or nonjustifiably) if they give their true thoughts about a program. This threat is typically from fear of being punished by someone in authority. Most commonly, interviewees are labeled trouble makers or are ostracized from the team if their worst case scenario is worse than management’s opinion. Risk workshops may exhibit social and institutional pressures to conform, perhaps to get consensus or to shorten the interview session. The organization may greatly discourage introducing a risk that has not been previously considered, particularly if the risk is sensitive or may negatively affect the program. If an interviewee is accompanied by someone, risk analysts cannot guarantee that the interviewee’s responses are unbiased.
For more information on formal risk assessments and their relation to cost, schedule, and EVM, see GAO-09-3SP.↩︎