Assessing the Extent to Which Cost Estimating Best Practices Are Met

Auditors should collect data produced from both a program’s cost estimate and, if applicable, its EVM system. They can collect these data via questionnaires, structured interviews, direct observations, or computations, among other methods. Appendix IV is a sample data collection instrument with reasons why auditors need the information.

After auditors have collected the data, they must evaluate the data for integrity and quality. For cost estimates, auditors must confirm that, at a minimum, internal quality control checks show that the data are reliable and valid. To do this, they must have source data and must evaluate the rationale for each cost element. Examples of these techniques include verifying that:

• the parameters (or input data) used to create the estimate are valid and applicable;
• labor costs include a time-phased breakdown of labor hours and rates, if applicable;
• the calculations for each cost element are correct and the results make sense;
• the program cost estimate is an accurate total of sub-element costs; and
• inflation adjustments are properly made to account for differences in the price of goods and services over time.

Auditors should clarify issues about data and methodology with agency cost estimators. For example, auditors may ask what adjustments were made to account for differences between the new and existing systems with respect to design, manufacturing processes, and types of materials. In addition, auditors should look for multiple sources of data that converge toward the same number in order to gain confidence in the data used to create the estimate.

It is particularly important that auditors understand problems associated with the historical data—such as program redesign, schedule slips, and budget cuts—and whether the cost estimators normalized the data to remove their effects. According to experts in the cost community, program inefficiencies should not be removed from historical data because the development of most complex systems usually encounters problems. The experts stress that removing data associated with past problems introduces unnecessary risk. This topic is discussed in greater detail in chapter 9.

With regard to EVM, auditors should request a copy of the system compliance or certification letter that shows the contractor’s ability to satisfy the 32 EVM guidelines. The guidelines are test points to determine the quality of a contractor’s EVM system. Contract performance reports (CPR) formally submitted to the agency should be examined for reasonableness, accuracy, and consistency with other program status reports as a continuous measure of EVM system quality and robustness. Auditors should also request a copy of the integrated baseline review (IBR) results (discussed in chapter 19) to see what risks and treatment plans were identified. Auditors should request copies of internal management documents or reports that use EVM data to ensure that EVM is being used for management, not only for external reporting. Finally, to ensure that EVM data are valid and accurate, auditors should look for evidence that EVM analysis and surveillance are performed regularly by staff trained in this specialty.