Phase: Analyze alternatives
12. Identify significant risks and mitigation strategies
Definition: The AOA team identifies and documents the significant risks and mitigation strategies for each analyzed alternative. Risks are ranked in terms of significance to the mission need and functional requirements. All risks are documented for each alternative along with any overarching or alternative specific mitigation strategies. Schedule risk, cost risk, technical feasibility, risk of technical obsolescence, dependencies between a new program and other projects or systems, procurement and contract risk, resource risks, and other risks are examined.
Effect: Not documenting the risks and related mitigation strategies for each alternative prevents decision-makers from performing a meaningful trade-off analysis necessary to select a preferred alternative.
13. Determine and quantify benefits/effectiveness
Definition: The AOA team uses a standard process to identify and document the benefits and effectiveness of each analyzed alternative. The AOA team drafts a metric framework that details the methods used to evaluate and quantify the measures of effectiveness and measures of performance for the whole mission need. The AOA team quantifies the benefits and effectiveness of each alternative over the alternative’s full life cycle, if possible. Just as costs cover the entire life cycle for each alternative, the benefits and effectiveness measures cover each alternative’s life cycle, if possible, in order to determine each alternative’s net present value (NPV), defined as the discounted value of expected benefits minus the discounted value of expected costs. In cases where the means to monetize a benefit are too vague (for example, intangibles like scientific knowledge), the AOA team treats those benefits as strategic technical benefits and uses scalability assessments to quantify those benefits so that they are compared across all viable alternatives. In situations where benefits cannot be quantified, the AOA team explains why this is the case as part of their analysis and documentation.
Effect: If the AOA team does not determine a standard process to quantify benefits and clearly establish criteria against which to measure all alternatives, bias is introduced to the study. Additionally, if the AOA team does not examine effectiveness over the entire life cycle, decision-makers cannot see the complete picture and are prevented from making an informed decision.
14. Tie benefits/effectiveness to mission need and functional requirements
Definition: The AOA team explains and documents how each measure of effectiveness supports the mission need and functional requirements. The AOA team explains how the measures of effectiveness describe the way the current environment is expected to evolve to meet the desired environment; the team also explains how the measures are tied to the specific mission need and functional requirements. This is the hierarchy that connects the overarching requirements to the data that are needed.
Effect: Unless the AOA team thoroughly explains and documents how the measures of effectiveness relate to the specific mission need and functional requirements, decision-makers will not have proper insight into the impact of each alternative.
15. Develop life cycle cost estimates (LCCEs)
Definition: The AOA team develops a LCCE for each analyzed alternative, including all costs from inception of the program through design, development, deployment, operation, maintenance, and disposal. The AOA team includes a cost expert who is responsible for development of a comprehensive, well-documented, accurate, and credible cost estimate for each viable alternative in the study. The LCCE for each alternative follows the cost estimating process described in the GAO Cost Estimating and Assessment Guide, as appropriate for an early acquisition cost estimate, and uses a common cost element structure for all alternatives and includes all costs for each alternative. Costs that are the same across the alternatives (for example, training costs) are included so that decision-makers can compare the total cost rather than just the portion of costs that varies across all viable alternatives. The level of detail included in the LCCE should be consistent with the maturity of the alternatives. The AOA team expresses the LCCE in present value terms and explains why it chose the specific discount rate used. The AOA team ensures that economic changes, such as inflation and the discount rate, are properly applied, realistically reflected, and documented in the LCCE for all alternatives.76
Effect: An LCCE that is incomplete (e.g. does not include cost estimates for all phases of an alternative’s life cycle) does not provide an accurate and complete view of the alternatives’ costs. Without a full accounting of life cycle costs, decision-makers will not have a comprehensive picture of the costs for each alternative and will have difficulty comparing the alternatives because comparisons may not be based on accurate information. Additionally, applying a discount rate is an important step in cost estimating because all cost data for each analyzed alternative must be expressed in like terms for comparison. Unless the AOA team properly normalizes costs to a common standard, any comparison would not be accurate, and any recommendations resulting from the flawed analysis would be negated. Properly normalizing costs is particularly important if various alternatives have different life cycle durations.
16. Include a confidence level or range for LCCEs
Definition: The AOA team presents the LCCE for each alternative with a confidence level or range, and not solely as a point estimate. Having a range of costs around a point estimate is useful because it conveys a level of confidence for each alternative to achieve a most likely cost. To document the level of risk associated with the point estimate for each analyzed alternative, the confidence level is included as part of the LCCE as part of the cost estimating Step 9, risk and uncertainty analysis. Decision-makers must have access to the confidence level associated with the point estimates for all viable alternatives in order to make informed decisions. Additionally, the AOA team uses a consistent method of comparing alternatives in order to present a comparable view of the risk associated with each alternative. For example, the comparison can be based on an established dollar value across alternatives (in order to observe the confidence level for each alternative at that dollar value). Alternatively, the comparison can be based on a predetermined confidence level across alternatives (in order to observe the dollar value associated with that confidence level for each alternative).
Effect: For decision-makers to make informed decisions, the alternatives’ LCCEs must reflect the degree of uncertainty. Without cost risk and uncertainty analysis the LCCEs for the viable alternatives are not credible.
17. Perform sensitivity analysis
Definition: The AOA team tests and documents the sensitivity of the cost and benefit and effectiveness estimates for each analyzed alternative to risks and changes in key assumptions. Major outcomes and assumptions are varied in order to determine each alternative’s sensitivity to changes in key assumptions. This analysis is performed in order to rank the key drivers that could influence the cost and benefit estimates based on how they affect the final results for each alternative. Each alternative includes both a sensitivity analysis and a risk and uncertainty analysis that identifies a range of possible costs based on varying key assumptions, parameters, and data inputs. As explained in best practice 16 (include a confidence level or range for LCCEs), life cycle cost estimates are adjusted to account for risk and sensitivity analyses.
Effect: Failing to conduct a sensitivity analysis to identify the uncertainties associated with different assumptions negatively impacts the credibility of the AOA process by increasing the chance the AOA team will recommend an alternative without an understanding of the full impacts on life cycle costs, which could lead to cost and schedule overruns.
The present value of the estimate reflects the time value of money— the concept that a dollar today can be invested and earn interest.↩︎