Sources of Data
Because all cost estimating methods are data-driven, analysts must know the best data sources. Table 7 lists some basic sources. Analysts should use primary data sources whenever possible. Primary data are obtained from the original source, can usually be traced to an audited document, are considered the best in quality, and are ultimately the most useful. Secondary data are derived rather than obtained directly from a primary source. Because they are derived, and thus changed, from the original data, their overall quality is lower and less useful. In many cases, secondary data are actual data that have been “sanitized” to obscure their proprietary nature. Without knowing the details, analysts will likely find such data of little use.
Table 7: Basic Primary and Secondary Data Sources
| Data type | Primary | Secondary |
|---|---|---|
| Basic accounting records | x | |
| Data collection input forms | x | |
| Cost reports | x | x |
| Historical databases | x | x |
| Interviews | x | x |
| Program briefs | x | x |
| Subject matter experts | x | x |
| Technical databases | x | x |
| Other organizations | x | x |
| Contracts or contractor estimates | x | |
| Cost proposals | x | |
| Cost studies | x | |
| Focus groups | x | |
| Research papers | x | |
| Surveys | x | |
Source: DOD and NASA. | GAO-20-195G
Cost estimators should understand whether and how data were changed before deciding whether they will be useful. For this reason, it is always better to use actual costs rather than estimates as data sources because actual costs represent the most accurate data available. While secondary data should not be the first choice, they may be all that is available. In these cases, the cost estimator should seek to understand what the data represent, how old they are, and whether they are complete. If these questions can be answered, the secondary data may be useful for estimating and would certainly be helpful for cross-checking the estimate for reasonableness.
Sources of historical data include business plans, catalog prices, contract performance reports, contract funds status reports, cost and software data reports, forward pricing rate agreements, historical cost databases, market research, program budget and accounting data from prior programs, supplier cost information, historical or current vendor quotes, and weight reports. Cost estimators should collect actual cost data from a list of similar and legacy programs. Because most new programs are improvements over existing ones, data should be available that share common characteristics with the new program.
Historical data provide the cost estimator insight into actual costs on similar programs, including any cost growth since the original estimate. As a result, historical data can be used to challenge optimistic assumptions and bring more realism to the estimate. For example, a review of the average labor rates for similar tasks on other programs can be a powerful cross-check against assumptions of skill mixes and overall effort. In addition, historical data from a variety of contractors can be used to establish generic program costs, or they can be used to establish cost trends for a specific contractor across a variety of programs. Contractor cost trends allow the cost estimator to establish adjustment factors if they are relying on proposal data for cost estimates. Additionally, insights can be obtained on cost accounting structures to allow for an understanding of how a certain contractor charges items such as other direct costs and overhead.
However, historical cost data also contain information from past technologies. It is essential that appropriate adjustments are made to account for differences between the new system and the existing system with respect to design characteristics, manufacturing processes (automation versus hands-on labor), and types of material used, among others. Statistical methods like regression, that analyze cost against time and performance characteristics, can reveal the appropriate technology-based adjustment.
Contract performance reports (CPRs) and cost and software data reports are excellent sources of historical cost data for DOD programs. More information on CPRs can be found in chapter 19.
Cost data reports are often used in estimating analogous programs, with the assumption that it is reasonable to expect similar programs at similar contractors’ plants to incur similar costs. This analogy may not hold for the costs of hardware or software, but may hold in other WBS areas such as data, program management, or systems engineering. If the cost estimator can establish costs for the major deliverables, such as hardware or software, a factor may be applied for other areas of the WBS based on historical data available from cost reports. Rate and factor agreements contain rates and factors negotiated between the contractor and the government. Because the contractor’s business base may be fluid, with direct effect on these rates and factors, such agreements do not always exist. Information in the agreements represents negotiated direct labor, overhead, general and administrative data, and facilities capital cost of money. These agreements may cover myriad factors, depending on each contractor’s accounting and cost estimating structure. Typical factors include material scrap, material handling, quality control, sustaining tooling, and miscellaneous engineering support factors.
The scope of the estimate often dictates the need to consult with other organizations for raw data. Once government test facilities have been identified, for example, those organizations can be contacted for current cost data, support cost data, and the like. Other government agencies may also be involved with the development of similar programs and can be potential sources of data. Additionally, a number of government agencies and industry trade associations publish cost data that are useful in cost estimating.
Contractor proposals also provide cost data. However, a contractor proposal is a document that represents the contractor’s best estimate of cost, and proposals tend to be influenced by the amount the customer can spend. Therefore, proposal data should be viewed with caution, and care should be taken to determine if the proposal data are supportable. During source selection in a competitive environment, for instance, lower proposed costs may increase the chances of receiving a contract award. In this situation, cost estimators should analyze the cost data for realism. While often overly optimistic, a proposal can nonetheless provide useful information, such as:
structure and content of the contractor’s WBS;
contractor’s actual cost history on the same or other programs;
negotiated bills of material;
subcontracted items;
government-furnished equipment compared to contractor-furnished equipment lists;
contractor rate and factor data, based on geography and makeup of workforce;
technological state-of-the-art assumptions; and
estimates of management reserve and level of risk.
Because of the potential for bias in proposal data, the estimator must test the data to see whether they deviate from other similar data before deciding whether they are useful for estimating. This can be done through a plant visit where the cost estimator visits the contractor to discuss the basis for the proposal data. As with any potential source of data, it is critical to ensure that the data apply to the estimating task and are valid for use.