Analyze Performance

Analyze the Data

The basic steps for analyzing EVM data are

  1. Analyze performance:

    • validate the data,

    • determine what variances exist,

    • probe schedule variances to see if activities are on the critical path,

    • develop historical performance data indexes,

    • graph the data to identify any trends, and

    • review the format 5 variance analysis for explanations and corrective actions.

  2. Project future performance:

    • identify the work that remains,

    • calculate a range of EACs and compare the results to available funding,

    • determine if the contractor’s EAC is feasible, and

    • calculate an independent date for program completion.

  3. Formulate a plan of action and provide analysis to management.

These steps should be taken in sequence because each step builds on findings from the previous one. Developing independent EACs without first validating the EVM data is not recommended. It is important to understand what is causing problems before making projections about final program status. For example, if a program is experiencing a negative schedule variance, it may not affect the final completion date if the variance is not associated with an activity on the critical path or if the schedule baseline represents an early “challenge” date. Therefore, it is a best practice to follow the analysis steps so that all information is known before making independent projections of costs at completion.

Validate the Data

It is important to make sure that the CPR data make sense and do not contain anomalies that would make them invalid. If existing errors are not detected, then the data will be skewed, resulting in erroneous metrics and poor decision making. To determine if the data are valid, analysts should check all levels of the WBS, focusing on whether there are errors or data anomalies such as:

  • negative values for ACWP, BAC, BCWP, BCWS, or EAC;

  • unusually large performance swings (BCWP) from month to month;

  • BCWP and BCWS data with no corresponding ACWP;

  • BCWP with no BCWS;

  • BCWP with no ACWP;

  • ACWP with no BCWP;

  • ACWP that is far greater or less than the planned value;

  • inconsistency between EAC and BAC—for example, no BAC but an EAC or a BAC with no EAC;

  • ACWP exceeds EAC; and

  • BCWP or BCWS exceed BAC.

If the CPR data contain anomalies, the performance measurement data may be inaccurate. For example, a CPR reporting actual costs (ACWP) with no corresponding earned value (BCWP) could indicate that unbudgeted work is being performed but not captured in the CPR. Or, it could mean that an accounting error occurred in a previous reporting period that is now being reconciled. Another reason could be work that was behind schedule is finally being done; in this case there would be BCWP without BCWS because the work is occurring later than planned. Case study 25 highlights CPR data with these anomalies.

Case Study 25: Data Anomalies, from James Webb Space Telescope, GAO-16-112

Based on analysis of James Webb Space Telescope (JWST) contractor EVM data over 17 months, GAO found that some of the data used to conduct the analyses were unreliable. First, GAO found that both Northrop Grumman and Harris were reporting optimistic EACs that did not align with their historical EVM performance and fell outside the low end of our independent EAC range. Second, GAO found various anomalies in contractor EVM data for both contractors that they had not identified throughout the 17-month period we examined. The anomalies included unexplained entries for negative values of work performed (meaning that work was unaccomplished or taken away rather than accomplished during the reporting period), work tasks performed but not scheduled, or actual costs incurred with no work performed. For Northrop Grumman, many were relatively small in value ranging from a few thousand to tens of thousands of dollars. These anomalies are problematic because they distort the EVM data, which affects the projection of realistic EACs. GAO found that these anomalies occurred consistently within the data over a 17-month period, which brought into question the reliability of the EAC analysis built upon this information. NASA did not provide explanations into the anomalies for either contractor. While the contractors were able to provide explanations for the anomalies upon request, their explanations or corrections were not always documented within EVM records. Some of the reasons the contractors cited that were not in the EVM records included tasks completed later than planned, schedule recovered on behind schedule tasks, and replanning of customer-driven tasks. Without reconciling and documenting data anomalies, and utilizing reliable data for the risk-adjusted EAC, the JWST project did not have a reliable method to assess its cost reserve status going forward. This meant that some of the cost information the project officials used to inform their decision making may have failed to indicate true program performance, and as result, project management may not have had a solid basis for decision making.

GAO recommended that to resolve contractor data reliability issues and ensure that the project obtained reliable data to inform its analyses and overall cost position, the NASA Administrator direct JWST project officials to require the contractors to identify, explain, and document all anomalies in contractor-delivered monthly earned value management reports. In February 2016, NASA issued letters to the contractors requiring them to explain all anomalies in the contractor earned value management reports.

GAO, James Webb Space Telescope: Project on Track but May Benefit from Improved Contractor Data to Better Understand Costs, GAO-16-112 (Washington, D.C.: December 17, 2015).

In addition to checking the data for anomalies, the analyst should verify that the CPR data are consistent across formats. For example, the analyst should review whether the data reported on the bottom line of format 1 matches the data on the bottom line of format 2. The analyst should also assess whether program cost is consistent with the authorized budget.

Determine Variances

Cost and schedule variances from the baseline plan give management at all levels information about where corrective actions are needed to bring the program back on track or to update completion dates and EACs. While variances are often perceived negatively, they provide valuable insight into program risk and its causes. Variances empower management to make decisions about how best to handle risks. For example, management may decide to allocate additional resources or hire technical experts, depending on the nature of the variance.

Because negative cost variances are predictive of a final cost overrun if performance does not change, management needs to focus on containing them as soon as possible.

Probe Schedule Variances for Activities on the Critical Path

Analysts should determine whether schedule variances are from activities on the critical path. If they are, then the program will be delayed, resulting in additional cost unless other measures are taken. The following methods are often used to mitigate schedule problems:

  • consuming schedule reserve if it is available,

  • diverting staff to work on other tasks while dealing with unforeseen delays,

  • preparing for follow-on activities early so that transition time can be reduced,

  • consulting with experts to determine whether process improvements can reduce task time,

  • adding more people to speed up the effort, and

  • working overtime.

Caution should be taken with adding more people or working overtime because these options cost money. In addition, when too many people work on the same thing, communication tends to break down. Similarly, working excessive overtime can make staff less efficient.

A reliable network schedule that is kept current is a critical tool for monitoring program performance. Carefully monitoring the contractor’s network schedule will allow for determining when forecasted completion dates differ from the planned dates. Activities may be re-sequenced or resources realigned to reduce the schedule delay. It is also important to determine whether schedule variances are affecting downstream work. For example, a schedule variance may compress the durations of remaining activities or cause “stacking” of activities toward the end of the program, to the point at which success may no longer be realistic. If this happens, then an overtarget schedule may be necessary (discussed in chapter 20).

Various schedule measures should be analyzed to better understand the impact of schedule variances. For example, the amount of total float, as well as the number of activities with lags, date constraints, or lack of progress should be examined each month.54 Some indicators of poor schedule health:

  • Excess total float usually indicates that the schedule logic is flawed, broken, or absent. Large total float values should be checked to determine if they are real or a consequence of incomplete scheduling.

  • Date constraints typically are substitutes for logic and can mean that the schedule is not well planned.

  • Lags are typically reserved for time that is unchanging, does not require resources, and cannot be avoided (as in waiting for concrete to cure), but lags are often inappropriately used instead of logic to force activities to start or finish on a specified date.

  • If open work packages are not being statused regularly, it may be that the schedule and EVM are not really being used to manage the program. Analyzing these issues can help assess the schedule’s accuracy.

In addition to monitoring tasks on the critical path, close attention should be paid to near-critical tasks, as these may alert management to potential schedule problems. If an activity is not on the critical path but is experiencing a schedule variance, it may be turning critical. Therefore, schedule variances should be examined for their causes. For instance, if material is arriving late and the variance will disappear once the material is delivered, its effect is minimal. But, if the late material is causing activities to slip, then its effect is much more significant.

A negative schedule variance eventually disappears when the full scope of work is completed because at this point the amount of work accomplished is equal to the amount of work planned. However, a negative cost variance is not corrected unless work that has been overrunning begins to underrun—a highly unlikely occurrence. Schedule variances are usually followed by cost variances, because as schedule increases various costs such as labor, rented tools, and facilities increase. The amount of the estimate due to inflation typically increases also. Additionally, management tends to respond to schedule delays by adding more resources or authorizing overtime.

Develop Historical Performance Data Indexes

Performance indexes are measures of program efficiency that indicate how a program is performing. Performance indexes determine the effect a cost or schedule variance has on a program. For example, a $1 million cost variance in a $500 million program is not as significant as it is in a $10 million program. Table 23 provides three performance indexes and describes what each indicates about program status.

Table 23: EVM Performance Indexes
Scroll to the right to view full table.
Index Formula Indicator
Cost performance index (CPI) CPI = BCWP / ACWP The CPI metric is a measure of cost expended for the work completed. A CPI value greater than 1.0 indicates the work accomplished cost less than planned, while a value less than 1.0 indicates the work accomplished cost more than planned.a
Schedule performance index (SPI) SPI = BCWP / BCWS The SPI metric is a measure of the amount of work accomplished versus the amount of work planned. An SPI value greater than 1.0 indicates more work was accomplished than planned, while an SPI value less than 1.0 indicates less work was accomplished than planned.a
To complete performance index (TCPI) TCPI = BCWR / (EAC - ACWP)b The TCPI is a comparison of the amount of work remaining to the budget remaining. It is the calculated projection of cost efficiency that must be achieved on the remaining work to meet a specified goal, such as BAC or EAC. The performance efficiency need to complete the project is often more than the previous level of performance achieved. The TCPI can be compared to a CPI to test the EAC’s reasonableness and used as the basis for discussion of whether the performance required is realistic.c

Source: DOD and PMI. | GAO-20-195G

aDOD, OUSD A&S (AE/AAP), Earned Value Management Implementation Guide, (Washington, D.C.: January 2019).

bBCWR = budgeted cost of work remaining, or BAC - BCWP.

cProject Management Institute, Inc. Practice Standard for Earned Value Management, Second Edition, 2011.

The cost performance index (CPI) and schedule performance index (SPI) can be used independently or together to forecast a range of cost estimates at completion. They also give managers early warning of potential problems that need correcting to avoid adverse results.

Like variances, performance indexes should be investigated. An unfavorable CPI—one less than 1.0—may indicate that work is being performed less efficiently or that material is costing more than planned. Or it could mean that more expensive labor is being employed, unanticipated travel was necessary, or technical problems were encountered. Similarly, a mistake in how earned value was taken or improper accounting could cause performance to appear to be less efficient. More analysis is needed to know what is causing an unfavorable condition. Likewise, favorable cost or schedule performance indexes may stem from errors in the EVM system, not necessarily from work taking less time than planned or underrunning its budget. Thus, failure to assess the full meaning behind the indexes runs the risk of basing estimates at completion on unreliable data.

An SPI different from 1.0 warrants more investigation to determine what effort is behind or ahead of schedule. Analysts should examine the WBS to identify issues at the activity level associated with completing the work. Using this information, management could decide to reallocate resources, where possible, from activities that might be ahead of schedule (SPI greater than 1.10) to help activities that are struggling (SPI less than 0.90) to get back on track. There should also be an analysis of the available float of activities that are slipping to see if proactive steps should be taken so resources are allocated more efficiently to future activities.

If the TCPI is much greater than the current or cumulative CPI, then the analyst should discover whether this gain in productivity is even possible. If not, then the contractor is most likely being overly optimistic. A rule of thumb is that if the TCPI is more than 5 percentage points higher than the CPI, the EAC is too optimistic. For example, if a program’s TCPI is 1.2 and the cumulative CPI is 0.9, it is not expected that the contractor can improve its performance that much through the remainder of the program. To meet the EAC, the contractor must produce $1.20 worth of work for every $1.00 spent. Given the contractor’s actual performance of $0.90 worth of work for every $1.00 spent, it is unlikely that it can improve its performance that much. One could conclude that the contractor’s EAC is unrealistic and that it underestimates the final cost.

Performance reported early in a program tends to be a good predictor of how the program will perform later, because early control account budgets tend to have a greater probability of being achieved than those scheduled to be executed later. DOD’s contract analysis experience suggests that all contracts are front-loaded to some degree, simply because more is known about near-term work than far-term.

In addition to the performance indexes, three other useful calculations for assessing program performance are:

  • percent planned = BCWS/BAC,

  • percent complete = BCWP/BAC, and

  • percent spent = ACWP/BAC.

Taken together, these formulas measure how well a program is performing. For example, if percent planned is much greater than percent complete, the program is significantly behind schedule. Similarly, if percent spent is much greater than percent complete, the program is significantly overrunning its budget.

Review the Format 5 Variance Analysis

After determining which WBS elements are causing cost or schedule variances, examining the format 5 variance analysis can help determine the technical reasons for variances, what corrective action plans are in place, and whether or not the variances are recoverable. Corrective action plans for cost and schedule variances should be tracked through the risk mitigation process. In addition, favorable cost variances should be evaluated to see if they are positive as a result of performance without actual cost having been recorded. This can happen when accounting accruals lag behind invoice payments. Finally, the variance analysis report should discuss any contract rebaselines, and whether any authorized unpriced work exists and what it covers.

  1. Total float is the amount of time an activity can be delayed or extended before delay affects the program’s finish date. A lag is used in a schedule to denote the passing of time between two activities. Lags cannot represent work and cannot be assigned resources. Date constraints can be placed on an activity’s start or finish date to override network logic. They can limit the movement of an activity to the past or future or both. See GAO. Schedule Assessment Guide: Best Practices for Project Schedules, GAO-16-89G. (Washington, D.C.: December 22, 2015) for more information.↩︎

  2. A waterfall chart is made up of floating columns that show how an initial value increases and decreases by a series of intermediate values leading to a final value.↩︎