Phase: Document and review the AOA process

18. Document AOA process in a single document

Definition: The AOA team documents in a single document all steps taken to initialize, identify, analyze, and select alternatives. This document, which usually is a final report, describes all actions taken for all best practices of the AOA process. For example, the document clearly describes the preferred alternative and provides the detailed rationale for the recommendation based on analytic results. This document also includes, among all other things, the overall selection criteria and rational for their weighting; the rationale for nonviable or viable ratings for alternatives; a through description of alternatives; the ground rules, assumptions, and constraints for each alternative; the risk drivers and mitigation techniques; an analysis of the costs and benefits associated with each alternative; the trade-offs between costs, benefits, and risks; a description of the sensitivity analysis conducted and its results; the final rationale supporting the alternative selected by the AOA team or decision-makers, and the results and recommendations of the final independent review and any other reviews that took place throughout the AOA process.

Effect: Without a clear document that compiles all information, including standards used to rate and perform the analysis, it will not be apparent that the study is comprehensive, unbiased, and credible because the documentation does not explain the rationale for methodology or the calculations underlying the analysis. Having all the information related to all best practices of the AOA process in a single document also makes it easier for an independent reviewer to assess the AOA process.

19. Document ground rules, assumptions and constraints

Definition: The AOA team documents and justifies all ground rules, assumptions, and constraints used in the AOA process. Assumptions and constraints help to scope the AOA. Ground rules represent a common set of agreed upon standards that provide guidance and minimize conflicts in definitions. Assumptions are explicit statements used to specify precisely the environment to which the analysis applies, while constraints are requirements or other factors that cannot be changed to achieve a more beneficial approach. Ground rules, assumptions and constraints are detailed and justified for each alternative in the AOA plan.

Effect: Without documented and justified ground rules, assumptions and constraints it will be difficult for decision-makers to evaluate the alternatives.

20. Ensure AOA process is impartial

Definition: The AOA team conducts the analysis without having a predetermined solution in mind. The AOA process is an unbiased inquiry into the costs, benefits, and capabilities of all alternatives which informs the decision-making process rather than reflecting the validation of a predetermined solution.

Effect: An AOA process is not considered valid if it is biased. Performing a study with a predetermined solution distorts the results. The validity of the analysis is affected if bias is introduced to the inputs.

21. Perform independent review

Definition: An entity independent of the AOA process reviews the extent to which all best practices are followed. An independent review is one of the most reliable means to validate an AOA process. The AOA process is completed and documented with enough thoroughness to ensure that an independent organization outside of the program’s chain of command can review the AOA documentation and clearly understand the process and rationale that led to the selection of the preferred alternative. Part of the documentation includes approval and review from an office outside of the one that asked for or performed the AOA process. Recommendations provided by the review(s) throughout the AOA process should be followed by the AOA team. In the exceptional case that the AOA team does not follow a recommendation, the AOA team documents the reasons why those recommendations were not adopted. For certain projects, in addition to an independent review at the end of the AOA process, additional reviews are necessary at earlier stages of the process. Such reviews may be conducted after key steps are performed in the AOA process, for example the selection of the AOA team chapter 7, step 4, the development of the AOA process plan chapter 10, step 7, or the identification of viable alternatives chapter 14, step 11. While early reviews are not a substitute for the independent review conducted at the end of the AOA process, they help ensure that bias is not added throughout the course of the AOA process. Reviews throughout the AOA process can also keep the customer and the decision-maker informed of the process. Any issues with the AOA work conducted prior to the review can be corrected immediately, if necessary, rather than wait until the independent review at the end and redoing the work then.

Effect: Without independent reviews, the results are more likely to include organizational bias or lack the thoroughness needed to ensure that a preferred solution is chosen and not a favored solution, calling into question the credibility of the AOA process.