AOA Process Reliability Assessment
An important best practice is conducting an independent review of the AOA process. It is important that the AOA process and its results be validated by an organization independent of the customer and decision-maker to ensure that a high-quality AOA is developed, presented, and defended to management. As shown in the figure, this review can occur throughout the AOA process life-cycle and helps verify that the AOA adequately reflects the program’s mission need and provides a reasonable assessment of the cost and benefits associated with the alternatives.
Independent reviewers typically rely less on assumptions alone and, therefore, tend to provide more realistic analyses. Moreover, independent reviewers are less likely to automatically accept unproven assumptions associated with anticipated savings. That is, they bring more objectivity to their analyses, resulting in a reality check of the AOA process that reduces the odds that management will invest in an unreasonable alternative. After the AOA is complete, an AOA reliability assessment can be performed.to help improve an organization’s AOA development process.
To that end, we established four characteristics that identify a high-quality, reliable AOA process. These characteristics are useful in evaluating if the AOA process is well documented, comprehensive, unbiased, and credible.
“Well-documented” - the AOA process is thoroughly described in a single document, including all source data, clearly detailed methodologies, calculations and results, and that selection criterion are explained. A well-documented AOA process is considered a key characteristic for a high-quality AOA. Without good documentation, the customer, the decision-maker, or independent reviewers will not be convinced that the AOA results are comprehensive, unbiased, and credible; questions about the approach or data used to create the AOA cannot be answered; and the scope of the analysis cannot be thoroughly defined. Furthermore, without adequate documentation, an entity unfamiliar with the program will not be able to understand the rationale surrounding the selection of the preferred alternative.
“Comprehensive” - the AOA process ensures that the mission need is defined in a way to allow for a robust set of alternatives, that all analyzed alternatives have been considered, and that each alternative is analyzed thoroughly over the program’s entire life cycle. Without a clearly defined mission need and comprehensive list of alternatives, the AOA process could overlook the alternative that best meets the mission need. Furthermore, without considering the complete life cycle of each alternative, decision-makers will not have a complete picture of the alternatives analyzed.
“Unbiased” - the AOA process does not have a predisposition toward one alternative over another; it is based on traceable and verifiable information. If an AOA process is biased, the validity of the analysis is called into question. Furthermore, if the AOA process has the appearance of being biased, the customer, decision-maker, or independent reviewers may not act on the results of the AOA report and may request additional information, extending the time before the preferred alternative is selected or enacted.
“Credible” - the AOA process thoroughly discusses the limitations of the analyses resulting from the uncertainty that surrounds both the data and the assumptions for each alternative. If the AOA process is not credible, there is an increased chance that the AOA team will recommend an alternative without understanding the full impact of the life cycle costs, potential benefits, or how the alternatives relate to the status quo, which could result in the selection of a less than optimal alternative.
Table 49 shows the four characteristics and their associated AOA best practices.
Table 49: The Four Characteristics of the AOA Process and Their Corresponding Best Practices
| Characteristics | AOA process best practice |
|---|---|
Well-documented: The Analysis of Alternatives (AOA) process is thoroughly described, including all source data, methodologies, calculations and results, and selection criteria are explained.
|
9. Describe alternatives 12. Identify significant risks and mitigation strategies 14. Tie benefits/effectiveness to mission need and functional requirements 18. Document AOA process in a single document 19. Document ground rules, assumptions, and constraints |
Comprehensive: The level of detail for the AOA process ensures no alternatives are omitted and that each alternative is examined thoroughly for the program’s entire life cycle.
|
1. Define mission need 2. Define functional requirements 3. Develop AOA time frame 8. Develop list of alternatives 11. Assess alternatives’ viability 15. Develop Life cycle cost estimates (LCCE) |
Unbiased: The AOA process does not have a predisposition towards one alternative over another but is based on traceable and verified information
|
4. Establish AOA team 6. Weight selection criteria 7. Develop AOA process plan 13. Determine and quantify benefits and effectiveness 20. Ensure AOA process is impartial 22. Compare alternatives |
Credible: The AOA process discusses from any limitations of the analysis resulting from the uncertainty surrounding the data to assumptions made for each alternative
|
5. Define selection criteria 10. Include baseline alternative 16. Include a confidence level or range for LCCEs 17. Perform sensitivity analysis 21. Perform independent review |
Source: GAO. | GAO-20-195G
As stated above, the AOA is intended to compare the operational effectiveness, cost, and risks of a number of potential alternatives to address valid needs and shortfalls in operational capability. The best practices that GAO identified in the AOA process ensure that the best alternative that satisfies the mission need is chosen on the basis of the selection criteria. Case study 28 discusses why the AOA process was important for a Marine Corps program.
Since 1972, the primary platform for transporting Marines from ship to shore under hostile and hazardous conditions has been the Assault Amphibious Vehicle (AAV). According to DOD, the need to modernize the United States Marine Corps’ (USMC) ability to move personnel and equipment from ship to shore is essential. Since 1995, USMC has undertaken a number of efforts to do this. In 2011, USMC subsequently began the acquisition process for the Amphibious Combat Vehicle (ACV), a potential replacement vehicle for all or a portion of the AAV fleet. The ACV is intended to transport Marines from ship to shore and provide armored protection once on land.
In late 2014, the USMC completed an AOA update to support the release of the request for proposal for ACV Increment 1. Over the years, other AOAs have been completed for related acquisitions, including the Expeditionary Fighting Vehicle, the Marine Personnel Carrier and a previous version of the ACV considered in 2012. These previous AOAs and other supporting studies comprise a body of work that informed the ACV AOA update as well as the ACV acquisition as a whole.
GAO’s assessment of the 2014 AOA found that overall it met best practices for AOAs and was, therefore, considered reliable. Considered in the context of the related body of work, the Amphibious Combat Vehicle (ACV) met 15 of the 22 AOA best practices, including ensuring that the AOA process was impartial and developing an AOA process plan, among others. Further, four of the remaining best practices were substantially met, two were partially met, and one was minimally met. For example, best practices call for the documentation of all assumptions and constraints used in the analysis. GAO found that the 2014 AOA did not include a full list of assumptions and constraints and any assumptions or constraints from previous analysis, if relevant, were not updated or referenced in the new analysis. As a result, it could have been difficult for decision-makers to make comparisons and trade-offs between alternatives. DOD’s Cost Assessment and Program Evaluation staff also reviewed the 2014 AOA and found that it was sufficient. However, they identified a few areas of caution, including recommending additional testing of land mobility to further verify USMC assertions that the wheeled ACV 1.1 would have the same mobility in soft soil as tracked vehicles.