Correlation between Cost Elements
Positive correlation occurs when two WBS elements are both influenced by the same factor and can be expected to vary in the same direction within their own probability distributions in any consistent scenario. Correlation might be positive and fairly strong if, for instance, the contractor’s productivity is expected to be similar for multiple elements that have been bid. Unless correlation is specified between these element costs in a simulation, certain iterations or scenarios would have some elements that cost more and others that cost less in their respective ranges during an iteration. This would be inconsistent with the idea that they all react to the same assumptions about the contractor’s productivity. Specifying correlations between cost elements ensures that each iteration represents a scenario in which program costs are consistently higher or lower in their ranges together. Correlation prevents the simulation from inappropriately drawing a low value for one element and a high value for another element, causing a cancellation of risk when both elements are positively correlated. Because the program cost estimate is the sum of the cost elements, if the costs are higher together or lower together, there is a chance that total program cost will be very high or very low. Therefore, correlation affects the low and high values in the simulation results because correlated elements tend to reinforce one another. In practice, if an organization decides to focus on the 80th percentile, correlation matters; correlation does not matter as much around the 50th percentile.
Figure 17 shows the effect of including correlation between WBS elements in the three-point risk simulation for the airframe production. In this example, 90 percent correlation was added between related elements. While the 90 percent correlation is high (correlation is measured between -1.0 and 1.0), there are often no actual data on correlation, so expert judgment can be used to set the correlation coefficients. Assuming this degree of correlation, we get the result shown in figure 17.
Figure 17: Cumulative Probability Distributions with and without Correlation
Notice that the correlation has widened the overall distribution. The 50th percentile is nearly the same in both cases, $2.611 billion without correlation and $2.612 billion with correlation. However, the 80th percentile increases by $24 million when correlation is added.
To properly capture correlation, the cost model should be structured with all dependencies intact. For instance, if the cost of training is modeled as a factor of hardware cost, then any uncertainty in the hardware cost will be inherently positively correlated to the risk in training cost. Thus, when the simulation is run, risks fluctuating within main cost element probability distributions will accurately flow down to dependent WBS elements.
In cases where dependencies are not modeled, it may be necessary to assign correlation to elements to account for correlated risk. These elements are typically level-of-effort support activities, like systems engineering and program management. In addition, correlation may have to be assigned to some elements of the cost model to account for effects that the model may not capture. For example, a program risk may be that the length of an aircraft wing increases. A larger wing would likely require a larger engine than was originally estimated. If this risk effect is not accounted for in the cost model, it must be inserted into the risk scenario.
One of the advantages of a CER-based cost model is the manner in which the statistical analysis used to derive the CERs can also be drawn on to identify, and in some cases quantify, the correlations between various cost risk elements.
To determine correlation factors, estimators may examine the correlation coefficients from the simulation model to determine the amount of correlation that already exists in the cost model. As a rule of thumb, it is better to insert a nominal correlation between elements than to have no correlation input at all.36
Assigning a risk to multiple WBS elements with the risk driver method causes the elements to be correlated, negating the need for correlation factor estimates. If the risk occurs on one assigned element during the simulation, it occurs on all the assigned elements. If there are some risks assigned to one element but not another, correlation will be less than 100 percent. Modeling correlation with risk drivers avoids the difficult task of estimating a number of pair-wise correlations.
Correlation should never be ignored. Doing so can significantly affect the cost risk analysis by understating the probability distribution, resulting in a false sense of confidence in the estimate. In particular, high-risk programs should be expected to have a wide range of possible costs.
Department of Defense and the National Aeronautics and Space Administration, The Joint Agency Cost and Schedule Risk Handbook (Washington, D.C.: March 12, 2014) says that several academic papers have suggested a default correlation of 0.25 while others have recommended 0.45 or 0.63. The handbook recommends using 0.3 as a default.↩︎