Appendix IV: Data Collection instrument and Rationale

Table 32 lists key data items that are generally required for auditors to thoroughly assess the quality and reliability of a program cost estimate.

Table 32: Common Data Collection Items

Scroll to the right to view full table.

Item	Rationale
Program management review briefings or similar documentation	This information tells the auditor what senior management was told and when the presentations were made’what problems were revealed, what alternative actions were discussed.
Budget documentation, including projected budget and OMB 300 reports	Budget documentation assures the auditor that agencies are properly employing capital programming to integrate the planning, acquisition, and management of capital assets into the budget decision-making process. Agencies are required to establish cost, schedule, and measurable performance goals for all major acquisition programs and should achieve, on average, 90 percent of those goals.
Technical baseline description	The technical baseline description provides the auditor with the program’s technical and program baseline. Besides defining the system, it provides complete information on testing plans, procurement schedules, acquisition strategy, and logistics plans. This is the document on which cost analysts base their estimates and is therefore essential to the auditor’s understanding of the program.
Work breakdown structure (WBS) and dictionary	The WBS and associated dictionary represent a hierarchy of product-oriented elements that provide a detailed understanding of what the contractor was required to develop and produce.
Program cost estimate supporting documentation, showing the basis of the estimates, including methodologies, data sources, risk simulation inputs and results, sensitivity analyses, and the like	Only by assessing the estimate’s underlying data and methodology can the auditor determine its quality. This information will address important concepts such as applicability of data, normalization of data, estimating methodologies, and parametric statistics.
Electronic cost estimate model	Access to the original electronic model will help auditors assess the estimate’s underlying data and methodologies. It also allows auditors to confirm there are few, if any, mistakes in the estimate.
Independent cost estimate (ICE) and related documentation, including a discussion of how cost estimates were reconciled	The ICE cost model and related documentation allow auditors to confirm that decision-makers were provided with additional insight into the program’s potential costs.
A risk management plan and a copy of the current risk register	Access to risk information helps auditors determine whether threats and opportunities have been properly quantified in the cost estimate.

Source: GAO. | GAO-20-195G

Table 33 lists key data items that may be applicable to an agency or a contract type, or are otherwise specific to a particular program’s cost estimate. Requests for these data should be tailored as needed.

Table 33: Supplementary Common Data Collection Items

Scroll to the right to view full table.

Item	Rationale
Nunn-McCurdy Amendment documentation	For DOD major defense acquisition programs, it is important that the auditor know the nature of the breach, when it occurred, when it was reported, and what action was taken.
Statement of work	The SOW conveys what the contractor was required to deliver at a given time. It also provides price or cost information, including the negotiated price or cost, as well as the type of contract (such as fixed-price, cost-plus-fixed-fee, cost-plus-award, or incentive fee). A statement of objective (SOO) and performance work statement (PWS) may also be used in the contractual process to establish desired service outcomes and performance standards.
Technology readiness assessments	A technology readiness assessment provides an evaluation of a system’s technological maturity by major WBS elements. It is useful in identifying over-optimism in technology assumptions. For those elements associated with technologies that have not achieved the required maturity level, the auditor can assess whether satisfactory mitigation plans have been developed to ensure that acceptable maturity will be achieved before milestone decision dates.
Design review reports, preliminary and critical	Design review reports provide the technical information needed to ensure that the system is satisfactorily meeting its requirements. The preliminary design review ensures that the system can proceed into detailed design, while meeting its stated performance requirements within cost (program budget), schedule (program schedule), risk, and other system constraints. The critical design review ensures that the system can proceed into system fabrication, demonstration, and test, while meeting its stated performance requirements within cost, schedule, risk, and other system constraints. It also assesses the system’s final design as captured in product specifications for each configuration item in the system (product baseline) and ensures that each product in the product baseline has been captured in the detailed design documentation.
Acquisition decision memorandum	This provides the documented rationale for the milestone decision authority’s (or investment review board’s) approving a program to advance to the next stage of the acquisition process.
Selected acquisition reports (SARs)	For major defense acquisition programs, the SAR provides the history and current status of total program cost, schedule, and performance, as well as program unit cost and unit cost breach information. For joint programs, SARs provide information by participant. Each SAR includes a full, life cycle cost analysis for the reporting program; an analysis of each of its evolutionary increments, as available; and analysis of its antecedent program, if applicable.
EVM contract performance reports for the past 12 months and monthly thereafter during the audit	Contract performance reports are management reports essential to an auditor’s ability to develop a comprehensive analysis. They are timely summary data from which to assess current and projected contract or program performance. The auditor can use them to reasonably project future program performance.
Cost and software data reporting	CSDRs provide the auditor with actual contractor development or procurement costs by WBS. Especially useful is the fact that recurring and nonrecurring costs are differentiated.
Integrated baseline review reports	The purpose of an IBR is to verify the technical content and realism of the interrelated performance budges, resources, and schedules. It helps the auditor understand the inherent risks in offerors’ or contractors’ performance plans and the underlying management control systems, and it should contain a plan to handles these risks.
EVM surveillance reports for the past 12 months	EVM surveillance reports assure the auditor that contractors are using effective internal cost and schedule control systems that provide contractor and government managers with timely and auditable data to effectively monitor programs, provide timely indications of actual and potential problems, meet requirements, and control contract performance. Surveillance ensures that a supplier’s EVM implementation of processes and procedures is being maintained over time and on all applicable programs and is in compliance with the 32 EVM guidelines.
Integrated master schedule	The IMS contains the detailed tasks or work packages necessary to ensure program execution. The auditor can use the IMS to verify the attainability of contract objectives, evaluate progress toward program objectives, and integrate the program schedule activities with the program components.

Source: GAO. | GAO-20-195G