The Four Characteristics of a Reliable Cost Estimate and their Best Practices

The four characteristics of a high-quality, reliable cost estimate are that it is comprehensive, well documented, accurate, and credible. A comprehensive estimate includes all possible costs, ensures that no costs were omitted or double-counted, and explains and documents key assumptions. A well-documented estimate can easily be repeated or updated and traced to original sources by auditors. An accurate estimate is developed by estimating each cost element using the best methodology from the data collected, adjusted properly for inflation, and contains few, if any, minor mistakes. A credible estimate incorporates results from sensitivity and risk and uncertainty analysis, is reconciled with an independent cost estimate, and is based on results that are cross-checked with alternate methodologies.

Comprehensive

Analysts should ensure that the cost estimate is complete and accounts for all possible costs. Comprehensive cost estimates completely define the program and reflect the current schedule and technical baseline. The reviewer should check that the estimate captures the complete technical scope of the work to be performed using a product-oriented WBS. Cost estimates should be structured with sufficient detail to ensure that cost elements are neither omitted nor double-counted. The reviewer should determine whether all assumptions and exclusions on which the estimate is based are reasonable and clearly identified and explained. Where information is limited and judgments must be made, the cost estimate should document all cost-influencing ground rules and assumptions.

The cost estimate includes all life cycle costs.

A life cycle cost estimate should encompass all past, present, and future costs for every aspect of the program, regardless of funding source. The cost estimate should include both government and contractor costs of the program over its full life cycle, from inception of the program through development, production, operations and maintenance, and disposal of the program. Any items excluded from the cost estimate should be documented and justified.

Unless the cost estimate accounts for all costs, the estimate cannot enhance decision making by allowing for design trade off studies to be evaluated on the basis of cost as well as on a technical and performance basis. Without fully accounting for life cycle costs, management will have difficulty successfully planning program resource requirements and making wise decisions.

A detailed discussion of life cycle cost estimates can be found in chapter 2 and chapter 4.

The technical baseline description completely defines the program, reflects the current schedule, and is technically reasonable.

There should be a documented technical baseline description that provides a common definition of the program, including a detailed technical, programmatic, and schedule description of the system, from which cost estimates are derived. The technical baseline should be developed by qualified personnel such as system engineers, approved by management, and should reside in one central location. The technical baseline should include sufficient detail of technical, program, and schedule characteristics based on the best available information known at the time, and this information should be updated as changes occur.

Key to developing a comprehensive estimate is having an adequate understanding of the acquisition program—the acquisition strategy, technical definition, characteristics, system design features, and technologies to be included in its design. Without these data, the cost estimator will not be able to identify the technical and program parameters that support the cost estimate and the quality of the cost estimate will be compromised.

A detailed discussion of a technical baseline can be found in chapter 6, step 3.

The cost estimate is based on a WBS that is product-oriented, traceable to the statement of work, and at an appropriate level of detail to ensure that cost elements are neither omitted nor double-counted.

The WBS should clearly outline the end product and major work of the program. This outline should be traceable to the schedule and, if applicable, the earned value management (EVM) system. In addition to hardware and software elements, the WBS should contain program management and other common elements such as testing, training, and data to ensure that all work is covered. The WBS should be standardized and there should be at least three levels of indenture that break larger products down into progressive levels of detail. Calculations should be performed to ensure that the sum of the lower level elements equal the higher level element. In addition, the WBS needs to be updated as the program becomes better defined and changes occur. There should be a WBS dictionary that defines what is included in each element and how it relates to others in the hierarchy.

A WBS provides a basic framework for a variety of related activities like estimating costs, developing schedules, identifying resources, determining where risks may occur, and providing the means for measuring program status using EVM. Without a WBS, the program lacks a framework to develop a schedule and cost plan that can easily track technical accomplishments—in terms of resources spent in relation to the plan as well as completion of activities. If a cost estimate does not specifically break out common costs, such as government furnished equipment costs, or does not include an associated WBS dictionary, one cannot ensure that the estimate includes all relevant costs. A standardized WBS is essential to ensure cost data can be collected and shared among programs and organizations, allowing for common cost measures. Without a standard, product-oriented WBS to facilitate the tracking of resource allocations and expenditures, the agency may not have the proper insight to reliably estimate the cost of future similar programs.

A detailed discussion of the WBS can be found in chapter 7, step 4.

The cost estimate documents all cost-influencing ground rules and assumptions.

The cost estimate documentation should include all defined ground rules and assumptions. The rationale and historical data needed to support the assumptions should be Included with the list of ground rules and assumptions. Risks associated with assumptions need to be identified and traced to specific WBS elements, and cost influencing assumptions should be used as inputs to the sensitivity and uncertainty analyses. Examples of potential program threats include budget constraints, delayed program content, dependency on other agencies, and technology maturity. The ground rules and assumptions should be developed by cost estimators with input from the technical community.

Cost estimates are typically based on incomplete or uncertain information and therefore need to be bound by the constraints that make estimating possible. These constraints are usually made in the form of ground rules and assumptions. However, because these assumptions are best guesses, the risks associated with any of these assumptions changing need to be identified and assessed. Many assumptions profoundly influence cost; the subsequent rejection of even a single assumption by management could invalidate many aspects of the cost estimate. Unless ground rules and assumptions are clearly documented, the cost estimate will not provide a basis for areas of potential risk to be identified and treated. Furthermore, the estimate will not be able to be reconstructed when the original estimators are no longer available.

A detailed discussion of ground rules and assumptions can be found in chapter 8, step 5.

Well-documented

Cost estimates are considered valid if they are well documented to the point at which they can be easily repeated or updated and can be traced to original sources through auditing. Thorough documentation also increases an estimate’s credibility and supports an organization’s decision-makers. The documentation should explicitly identify the primary methods, calculations, results, rationales or assumptions, and sources of the data used to generate each cost element.

Cost estimate documentation should be detailed enough to provide an accurate assessment of the cost estimate’s quality. For example, it should identify the data sources, justify all assumptions, and describe each estimating method for every WBS cost element. Estimating methods used to develop each WBS cost element, including any cost estimating relationships, should be thoroughly documented so that their derivation can be traced to all sources.

The documentation should show the source data used, the reliability of the data, and the estimating methodology used to derive each element’s cost.

The documentation should identify what methods were used such as analogy, expert opinion, engineering build up, parametric, or extrapolation from actual cost data. The supporting data are documented as well. For example, sources, content, time, and units are documented, along with an assessment of the accuracy of the data and reliability and circumstances affecting the data. It includes descriptions of how the data were normalized. Inflation indexes used to convert constant year dollars to budget year dollars are documented.

Data are the foundation of every cost estimate. The quality of the data affects the estimate’s overall reliability. Depending on the data quality, an estimate can range anywhere from a mere guess to a highly defensible cost position. Without sufficient background knowledge about the source and reliability of the data, the cost estimator cannot know with any confidence whether the data collected can be used directly or need to be modified. Because data can be gathered from a variety of sources, they are often in many different forms and need to be adjusted before being used. As a result, data normalization is often necessary so that the data are consistent and allow for valid comparisons and projections to occur.

For more information on data and documentation, see chapter 9, step 6 and chapter 13, step 10.

The documentation describes how the estimate was developed so that a cost analyst unfamiliar with the program could understand what was done and replicate it.

The data contained in the documentation should be adequate for easily updating the estimate to reflect actual costs or program changes so that they can be used for future estimates. The estimate documentation should include narrative and cost tables, and contain an executive summary and an introduction. A description of methods and related data should be broken out by WBS cost elements. In addition, sensitivity analysis, risk and uncertainty analysis, and updates that reflect actual costs and changes should be included. The documentation should completely describe the risk and uncertainty analysis. For example, the documentation discusses contingency and how it was derived, the cumulative probability of the point estimate, correlation, and the derivation of risk distributions. The documentation needs to include access to an electronic copy of the cost model, and both the documentation and the cost model are stored so that authorized personnel can easily find and use them for other cost estimates. Finally, analysts should ensure that guidance was used to govern the creation, maintenance, structure, and status of the cost estimate.

Without good documentation, management and oversight will not be convinced that the estimate is credible; supporting data will not be available for creating a historical database; questions about the approach or data used to create the estimate cannot be answered; lessons learned and a history for tracking why costs changed cannot be recorded; and the scope of the analysis cannot be thoroughly defined. Furthermore, without adequate documentation, analysts unfamiliar with the program will not be able to replicate the estimate because they will not understand the logic behind it.

For more information on data and documentation, see chapter 9, step 6 and chapter 13, step 10.

The documentation discusses the technical baseline description and the data in the technical baseline are consistent with the cost estimate.

The technical data and assumptions in the cost estimate documentation are consistent with the technical baseline description.

Because the technical baseline is intended to serve as the basis for developing a cost estimate, it should be discussed in the cost estimate documentation. Without a technical baseline, the cost estimate will not be based on a comprehensive program description and will lack specific information regarding technical and program risks.

For more information on technical baseline descriptions and documentation, see chapter 6, step 3 and chapter 13, step 10.

The documentation provides evidence that the cost estimate is reviewed and accepted by management.

The documentation should provide evidence that management was presented with a clear explanation of the cost estimate so as to convey its level of competence. For instance, management should be presented with an overview of the program’s technical foundation, time-phased dollars of the life cycle cost estimate, ground rules and assumptions, the estimating method and data sources for each WBS cost element. The cost estimating team should also convey the results of sensitivity analysis and cost drivers; the results of risk and uncertainty analysis including S curve cumulative probabilities and risk distributions; and the point estimate compared to an independent cost estimate along with any differences. A comparison to the current budget, conclusions and recommendations, and any other concerns or challenges that need to be addressed should also be presented. Management’s acceptance of the cost estimate, including recommendations for changes, feedback, and the level of contingency decided upon to reach a desired level of confidence, should be documented.

A cost estimate is not considered valid until management has approved it. Thus, it is imperative that management understands how the estimate was developed, including the risks associated with the underlying data and methods. Providing a briefing or documentation package to management about how the estimate was constructed—including the specific details about the program’s technical characteristics, assumptions, cost estimating methodologies, data sources, sensitivity, and risk and uncertainty—is necessary for management to gain confidence that the estimate is accurate, complete, and high in quality.

For more information on management approval and documentation, see chapter 13, step 10 and chapter 14, step 11.

Accurate

Validating that a cost estimate is accurate requires thoroughly understanding and investigating how the cost model was constructed. For example, all WBS cost elements should be checked to verify that calculations are accurate and account for all costs, including indirect costs. Moreover, appropriate inflation rates should be used so that costs are expressed consistently and accurately. Checking spreadsheet formulas, databases, and cost model data inputs is imperative to validate cost model accuracy. Besides the basic checks for accuracy, the estimating technique used for each cost element should be reviewed. In addition, cost estimators should update the cost estimate regularly to reflect significant changes in the program—such as when schedules or other assumptions change—and actual costs, so that it is always reflecting current status. During the update process, variances between planned and actual costs should be documented, explained, and reviewed.

The cost model was developed by estimating each WBS element using the best methodology from the data collected.

The estimating technique used for each cost element should be reviewed. Depending on the analytical method chosen, several questions should be answered to ensure accuracy. For example, if analogy is used, adjustments should be reasonable and based on program information, physical and performance characteristics, and the like. If the build-up method is used, the work scope should be well-defined, the WBS sufficiently detailed, a detailed and accurate materials and parts list available, specific quantities available, and an auditable source for labor rates provided. If the parametric method is used, the data set should be homogeneous and of a sufficient size for developing the cost estimating relationship (CER). Parametric models should be calibrated and validated using historical data. If CERs are used, statistics should be reasonable and documented, and the inputs are within the valid dataset range. If learning curves are used, they should represent manual, complex, and repetitive labor effort; production breaks are incorporated if production is not continuous. Finally, expert opinion is used sparingly and the estimates account for the possibility that bias influenced the results.

Understanding the methodology used to calculate each WBS element is essential to ensuring that the appropriate method was chosen and applied correctly. Each selected cost estimating method has strengths and weaknesses depending on, in part, where the program is in its life cycle and the availability of data. A program estimate’s credibility will suffer because the rationales of the methodology or the calculations underlying the cost elements are not accurate or based on sound estimating practices.

For more information on selecting a cost methodology, see chapter 10, step 7.

The estimate has been adjusted properly for inflation.

The cost data should be adjusted for inflation so that they can be described in like terms and to ensure that comparisons and projections are valid. The final estimate should be converted to budget year dollars.

Adjusting for inflation is an important step in cost estimating because in the development of an estimate, cost data must be expressed in like terms. If the inflation index used is not correct, the resulting estimate could overstate or understate the cost of the program. It is imperative that inflation assumptions be well documented and that the cost estimator always performs uncertainty and sensitivity analysis to study the effects of changes on the assumed inflation rates. Moreover, without access to the estimate details, analysts cannot verify that appropriate rates were used to adjust costs for inflation so that they are expressed consistently.

For more information on inflation adjustments, see chapter 9, step 6.

The cost estimate contains few, if any, minor mistakes.

The cost estimate should be mathematically sound. In other words, it should not contain mistakes, such as numbers that do not sum properly or costs that do not match between documents, among others. The program should employ a quality control process to ensure the cost estimates contains few, if any, minor mistakes.

Validating that a cost estimate is accurate requires thoroughly understanding and investigating how the cost model was constructed. Cost models with limited details complicate the ability to determine if all WBS cost estimate calculations are accurate and account for all costs.

For more information on mathematically sound cost estimates, see chapter 10, step 7.

The cost estimate is regularly updated to ensure it reflects program changes and actual costs.

The estimate should be updated to reflect changes in technical or programmatic assumptions, and how these changes affect the cost estimate should be documented. Estimates for cost elements should be replaced with actual costs from valid sources as they become available.

If the estimate is not updated, it will be difficult to analyze changes in program costs, and collecting cost and technical data to support future estimates will be hindered. The cost estimate should be updated when the technical baseline changes to maintain credibility. Unless properly updated on a regular basis, the cost estimate cannot provide decision-makers with accurate information.

For more information on updating the estimate, see chapter 6, step 3 and chapter 15, step 12.

Variances between planned and actual costs are documented, explained, and reviewed.

The estimate should document any variances for elements whose actual costs or schedules differ from the estimate. Any lessons learned from the variances should be documented.

Without a documented comparison between the current estimate, updated with actual costs, and the baseline estimate, the cost estimator cannot determine how well they are estimating and how the program is changing over time.

For more information on updating the estimate, see chapter 15, step 12.

The estimate is based on a historical record of cost estimating and actual experiences from other comparable programs.

The estimate is based on reliable and historical data and the data are applicable to the program. There is enough knowledge about the data source to determine if the data can be used to estimate accurate costs for the new program. If EVM data are available, they should be derived from an EVM system validated against the EIA-748 guidelines.

Accurate cost estimates are rooted in historical data. These data provide insight into actual costs of similar programs, including any cost growth from the original baseline estimate. Historical data can be used to challenge optimistic assumptions and bring more realism to the estimate. Thus, collecting valid and useful historical data is a key step in developing a sound cost estimate.

For more information on basing the estimate on actual data from comparable programs, see chapter 9, step 6.

Credible

Figure 21: Estimating Steps and Best Practices Related to a Credible Estimate
Tip: Click the figure to view a larger version in a new browser tab.

Because uncertainty cannot be avoided, it is necessary for cost estimators to identify the cost elements that represent the most risk and, if possible, quantify the risk. Credible cost estimates clearly identify limitations resulting from uncertainty or bias surrounding the data or assumptions. Major assumptions are varied and alternative outcomes recomputed to determine how sensitive outcomes are to changes in the assumptions and parameters.

A sensitivity analysis identifies key elements that drive cost and permits what-if analysis, often used to develop cost ranges and risk reserves. Along with a sensitivity analysis, a risk and uncertainty analysis adds to the credibility of the cost estimate because it identifies, among other things, the level of confidence associated with achieving the cost estimate. Another way to strengthen the credibility of the cost estimate is to perform cross-checks on cost estimating methodologies. That is, cost analysts should apply different methods to estimate high-value cost elements and determine if they produce similar results. Finally, an independent cost estimate (ICE) is considered one of the best and most reliable validation methods. An ICE provides an independent view of expected program costs that tests the program office’s estimate for reasonableness.

The cost estimate included a sensitivity analysis that identifies a range of possible costs based on varying major assumptions, parameters, and data inputs.

As a best practice, a sensitivity analysis should be included in all cost estimates because it examines the effects of changing assumptions and ground rules. Key cost drivers and ground rules and assumptions should be identified as factors, particularly those supporting cost elements that represent the highest percentage of cost. The total cost should be re-estimated by varying each factor. Results are documented and outcomes evaluated for those factors most sensitive to change.

Without conducting a sensitivity analysis to determine how the cost estimate is affected by a change in a single factor, the cost estimator will not fully understand which variable most affects the cost estimate. Simply varying factors by applying a subjective plus or minus percentage that does not have a valid basis does not constitute a valid sensitivity analysis. For management to make informed decisions, there should be a clear link between the technical baseline parameters, assumptions, and cost model inputs examined by cost estimators in the sensitivity analysis. Carefully assessing the underlying risks and supporting data, and documenting the sources of variation is necessary for a sensitivity analysis to inform decisions. An agency that fails to conduct sensitivity analyses to identify the effects of uncertainties associated with different assumptions increases the risk that decisions will be made without a clear understanding of these impacts on costs.

For more information on sensitivity analysis, see chapter 11, step 8.

A risk and uncertainty analysis was conducted that quantified the imperfectly understood risks and identified the effects of changing key cost driver assumptions and factors.

For management to make informed decisions, the program estimate must reflect the degree of uncertainty so that a level of confidence can be given about the estimate. Cost analysts should conduct a risk and uncertainty analysis that models a probability distribution for each cost element’s uncertainty based on data availability, reliability, and variability; accounts for correlation between cost elements; uses an uncertainty modeling technique such as Monte Carlo simulation to develop a distribution of total possible costs and an S curve to show alternative cost estimate probabilities; associates the point estimate with a cumulative probability; recommends contingency for achieving a desired confidence level; allocates, phases, and converts the risk-adjusted cost estimate to budget year dollars for budgeting as necessary.

Having a range of costs around a point estimate is more useful to decision-makers because it conveys the level of confidence in achieving the most likely cost and also informs them on cost, schedule, and technical risks. Without risk and uncertainty analyses, the estimate will lose credibility. Management will not be able to make good decisions because the estimate will lack the level of confidence associated with achieving the cost estimate. An estimate without risk and uncertainty analysis is unrealistic because it does not assess the variability in the cost estimate from such effects as schedules slipping, missions changing, and proposed solutions not meeting users’ needs. Lacking risk and uncertainty analysis, management cannot determine a defensible level of contingency that is necessary to cover increased costs resulting from unexpected design complexity, incomplete requirements, technology uncertainty, and other uncertainties.

For more information on risk and uncertainty analysis, see chapter 12, step 9.

Major cost elements were cross-checked to see if results were similar.

Major cost elements should be cross-checked with results from alternative methodologies to determine if results are similar.

One way to reinforce the credibility of the cost estimate is to see whether applying a different method produces similar results. Unless an estimate employs cross-checks, the estimate will have less credibility because stakeholders will have no assurance that alternative estimating methodologies produce similar results.

For more information on using alternative methodologies to cross-check results, see chapter 10, step 7.

An independent cost estimate (ICE) was conducted by a group outside the acquiring organization to determine whether other estimating methods produce similar results.

An ICE should be performed by an organization outside of the program office’s influence. The depth of the ICE should be sufficient to allow for reconciliation between the ICE and the program office estimate, and differences between the two should be documented and justified. The ICE should be based on the same technical baseline and ground rules as the program office estimate.

Without an ICE, decisions-makers will lack certain insights into a program’s potential costs because ICEs frequently use different methods and are less burdened with organizational bias. Moreover, ICEs typically incorporate adequate risk and, therefore, tend to be more conservative by forecasting higher costs than the program office. A program estimate that has not been reconciled with an ICE has an increased risk of proceeding underfunded because an ICE provides an objective and unbiased assessment of whether the program estimate can be achieved.

For more information on comparing the cost estimate to an independent cost estimate, see chapter 10, step 7.