Validating the EVM System

If EVM is to be used to manage a program, the contractor’s (and subcontractors’) EVM system should be validated to ensure that it complies with the agency’s implementation of the EIA-748 guidelines, provides reliable data for managing the program and reporting its status to the government, and is actively used to manage the program. This validation process is commonly referred to as system acceptance. The steps involved in the system acceptance process are shown in figure 35. Sometimes these steps may overlap rather than go in sequence because of resource or capability constraints between the EVM system owner, the government customer, or both. However, all steps leading up to actual acceptance must be addressed for an EVM system owner or agency program to implement a compliant EVM system.58

Figure 35: The Earned Value Management System Acceptance Process
Tip: Click the figure to view a larger version in a new browser tab.

The system acceptance process has four phases. In system design and implementation, establishing the EVM policy (which includes documented processes and procedures) is followed by implementing the EVM system. Once complete, reviews can begin. These reviews are the Self-Assessment Review (SAR), the Progress Assessment Review (PAR), and the Compliance Evaluation Review (CER). The purpose of these reviews is to assess EVM system compliance with the EIA-748 guidelines and identify areas of non-compliance. The PAR is optional and is conducted after the SAR. It is usually conducted by personnel from the CER team in preparation for a CER.

The CER is an independent review conducted by an individual or organization that:

  • has no stake in the EVM system, program, or contract being reviewed;
  • has the knowledge, skills, and abilities to fairly evaluate the fitness of the EVM systems implementation or surveillance; and
  • relies on the NDIA EVMS intent guide to determine whether the EVM system is compliant with EIA-748 guidelines.

The purposes of the CER include:

  • ensuring senior management actively participates and accepts ownership of the EVM process;
  • verifying that the EVM system is compliant with the EIA-748 guidelines;
  • demonstrating the use of the EVM system and EVM system outputs in making management decisions;
  • ensuring that the data and reports produced by the EVM system are reliable and capable of being used for planning, risk mitigation, corrective actions, forecasting schedule completion dates, and estimating at completion costs; and
  • verifying that the EVM system produces data that is consistent with the program technical, schedule, and cost status.59

Data traces are necessary for verifying that lower-level reporting aligns with higher levels, and that the data provide accurate management information. Interviews verify that the EVM system is fully implemented and actively used to manage the program. Additionally, the compliance review process and its results should be documented.

Upon successful completion of EVM system acceptance, an acceptance recognition document should be prepared and released. When cross-agency acceptance occurs, this is best accomplished by mutual agreements between agencies and organizations to recognize EVM system EIA-748 compliant acceptance or recognition documents. An agency can accept another organization’s EVMS acceptance with the understanding that they will need to instill a rigorous surveillance process (later in this chapter) to ensure that the written system description meets the intent of the 32 guidelines and is actively being followed. An alternative acceptance procedure is for a partner agency (or cross-agency) to review the documentation from the EVM system owner’s compliance evaluation review.

When no independent entity exists to perform EVM acceptance, the assessment may be performed by a qualified source that is independent from the program’s development, implementation, and direct supervision—for example, an agency’s inspector general. Moreover, civilian agencies may negotiate an interagency agreement to conduct acceptance reviews to satisfy the criteria for independence. For this arrangement to succeed, staff trained in EVM system reviews are required.

Best practices call for centers of excellence that include staff who are experienced in EVM system design, implementation, and validation and have a strong knowledge of EIA-748 guidelines. In addition, these staff should have good evaluation skills, including the ability to review and understand EVM data and processes and the ability to interview personnel responsible for the EVM system implementation to determine how well they understand their own system description and processes.

Case study 26 showcases programs which used an EVM system that had not been certified as being compliant with the EIA-748 guidelines.

Case Study 26: Certified EVM Systems, from NASA, GAO-13-22

The National Aeronautics and Space Administration (NASA) historically has experienced cost growth and schedule slippage in its portfolio of major projects and has taken actions to improve in this area, including adopting the use of EVM. In 2012, GAO was asked to examine, among other things, the extent to which NASA was using EVM to manage its major space flight acquisitions.

We found that 10 major spaceflight projects had not yet fully implemented EVM. As a result, NASA was not taking full advantage of opportunities to use an important tool that could help reduce acquisition risk. We assessed the 10 projects against three fundamental EVM practices that are necessary for maintaining a reliable EVM system and found shortfalls in two of three fundamental practices. Specifically, we found that more than half of the projects did not use an EVM system that was fully certified as compliant with the industry EVM standard.

Four projects had a certified EVM system, three did not, and three had a mixture in which some contractors and subcontractors had certified systems and some did not. When an EVM system is certified, the agency has assurance that the implemented system was validated for compliance with the EIA-748 standard by independent and qualified staff and therefore can be considered to provide reliable and valid data from which to manage a project. The Global Precipitation Measurement, Tracking and Data Relay Satellite System, Landsat Data Continuity Mission, and James Webb Space Telescope were the only projects that provided evidence that the contract performance reports provided came from EVM systems that were certified as compliant with the EIA-748 standard. The Lunar Atmosphere and Dust Environment Explorer, Magnetospheric Multiscale and Radiation Belt Storm Probes projects did not have EVM systems that were certified to be compliant with the EIA-748 standard. Finally, the Jet Propulsion Laboratory, a federally funded research and development center that the California Institute of Technology manages under a contract with NASA, was the only NASA Center with a certified EVM system. The Jet Propulsion Laboratory was responsible for managing the Orbiting Carbon Observatory 2 project. The Mars Atmosphere and Volatile Evolution Mission and Stratospheric Observatory for Infrared Astronomy prime contractors also had certified systems; however, their project offices did not.

GAO, NASA: Earned Value Management Implementation across Major Spaceflight Projects is Uneven, GAO-13-22 (Washington, D.C.: November 19, 2012).

  1. More information on EVM system acceptance is in National Defense Industrial Association (NDIA), Integrated Program Management Division, Earned Value Management System Acceptance Guide (Arlington, VA.: March 24, 2013).↩︎

  2. National Defense industrial Association (NDIA) Integrated Program Management Division (IPMD), Earned Value Management System Acceptance Guide (Arlington, VA.: March 24, 2013).↩︎