Software Errors Cost U.S. Economy $59.5 Billion Annually

NIST Assesses Technical Needs of Industry to Improve Software-Testing

FOR IMMEDIATE RELEASE: June 28, 2002

Software bugs, or errors, are so prevalent and so detrimental that they cost the U.S. economy an estimated $59.5 billion annually, or about 0.6 percent of the gross domestic product, according to a newly released study commissioned by the Department of Commerce's National Institute of Standards and Technology (NIST). At the national level, over half of the costs are borne by software users and the remainder by software developers/vendors. The study also found that, although all errors cannot be removed, more than a third of these costs, or an estimated $22.2 billion, could be eliminated by an improved testing infrastructure that enables earlier and more effective identification and removal of software defects. These are the savings associated with finding an increased percentage (but not 100 percent) of errors closer to the development stages in which they are introduced. Currently, over half of all errors are not found until "downstream" in the development process or during post-sale software use.

NIST funded the study, which was conducted by the Research Triangle Institute (RTI) in North Carolina, as part of a joint planning process with industry to help identify and assess technical needs that would improve software-testing capabilities. Findings of the 309-page report are intended to identify the infrastructure needs that NIST can meet through its research programs.

"The impact of software errors is enormous because virtually every business in the United States now depends on software for the development, production, distribution, and after-sales support of products and services," said NIST Director Arden Bement. "Innovations in fields ranging from robotic manufacturing to nanotechnology and human genetics research have been enabled by low-cost computational and control capabilities supplied by computers and software."

In 2000, total sales of software reached approximately $180 billion, supported by a large workforce encompassing 697,000 software engineers and 585,000 computer programmers.

Software is error-ridden in part because of its growing complexity. The size of software products is no longer measured in thousands of lines of code, but in millions. Software developers already spend approximately 80 percent of development costs on identifying and correcting defects, and yet few products of any type other than software are shipped with such high levels of errors. Other factors contributing to quality problems include marketing strategies, limited liability by software vendors, and decreasing returns on testing and debugging, according to the study. At the core of these issues is difficulty in defining and measuring software quality.

The increasing complexity of software, along with a decreasing average product life expectancy, has increased the economic costs of errors. The catastrophic impacts of some failures are well-known. For example, a software failure interrupted the New York Mercantile Exchange and telephone service to several East Coast cities in February 1998. But high-profile incidents are only the tip of a pervasive pattern that software developers and users agree is causing substantial economic losses.

Study Design and Background Facts

In the study, RTI identified a set of quality attributes and used them to construct metrics for estimating the cost of an inadequate testing infrastructure. Two in-depth case studies were conducted, one in the manufacturing sector (transportation equipment) and one in the service sector (financial services).

For the analysis of transportation equipment industries, data were collected from 10 vendors of computer-aided design/manufacturing/engineering (CAD/CAM/CAE) and product data management (PDM) software, and from 179 users, primarily automotive and aerospace companies. Approximately 60 percent of the automotive and aerospace manufacturers surveyed reported significant software errors in the previous year. Respondents who experienced errors reported an average of 40 major and 70 minor software bugs per year in their CAD/CAM/CAE or PDM software systems.

The total cost impact on these manufacturing sectors from an inadequate software-testing infrastructure is estimated to be $1.8 billion, and the potential cost reduction from feasible infrastructure improvements is $0.6 billion. Users of CAD/CAM/CAE and PDM software absorb approximately three-fourths of the total impact, with the automotive industry representing about 65 percent and the aerospace industry representing 10 percent. Software developers experience the remaining one-fourth of the costs.

For the analysis of financial services, data were collected from four developers of financial electronic data interchange (FEDI) and clearinghouse software as well as the software embedded in routers and switches that support electronic data exchange, and from 98 software users, primarily banks and credit unions. Approximately two-thirds of the software users surveyed reported experiencing major software errors in the previous year. Respondents that did have major errors reported an average of 40 major and 49 minor software bugs per year in their FEDI or clearinghouse software systems. Approximately 16 percent of those bugs were attributed to router and switch problems, and 48 percent were attributed to transaction software problems. The source of the remaining 36 percent of errors was unknown. Typical problems encountered due to bugs were increased person-hours used to correct posting errors, temporary shut down leading to lost transactions, and delay of transaction processing.

The total cost impact on the financial services sector from an inadequate software-testing infrastructure is estimated to be $3.3 billion. Potential cost reduction from feasible infrastructure improvements is $1.5 billion. Software developers absorb about 75 percent of the economic impacts. Users experience the remaining 25 percent of costs, with banks accounting for the majority of user costs.

The annual cost to these two major industry groups from inadequate software infrastructure is estimated to be $5.18 billion. Based on similarities across industries with respect to software development and use and, in particular, software-testing labor costs, RTI projected the cost to the entire U.S. economy. Using the per-employee impacts for the two case studies, an extrapolation to other manufacturing and service industries yields an approximate estimate of $59.5 billion as the annual cost to the nation of inadequate software testing infrastructure.

Thus, if all software bugs could be identified and removed instantly (in real time), the combined economic benefits to the two industry groups and to the economy would be $5.85 billion and $59.5 billion, respectively. Realizing that such a "perfect infrastructure" is not attainable, industry experts were asked for estimates of a plausible reduction in delayed identification and removal of software errors. Based on this information, a "feasible improved infrastructure" scenario was constructed. For this scenario, software developers were asked to estimate the potential cost savings associated with enhanced testing tools, and users were asked to estimate cost savings if the software they purchase had 50 percent fewer bugs and errors. This improved infrastructure scenario is estimated to result in a combined annual benefit of $2.10 billion to the two industry groups studied, and $22.2 billion to the U.S. economy.

Next Steps

The path to higher software quality is significantly improved software testing. Standardized testing tools, suites, scripts, reference data, reference implementations and metrics that have undergone a rigorous certification process would have a large impact on the inadequacies currently plaguing software markets. For example, the availability of standardized test data, metrics and automated test suites for performance testing would make benchmarking tests less costly to perform. Standardized automated testing scripts, along with standard metrics, also would provide a more consistent method for determining when to stop testing.

Electronic copies of NIST Planning Report 02-3, The Economic Impacts of Inadequate Infrastructure for Software Testing, can be obtained from http://www.nist.gov/director/prog-ofc/report02-3.pdf. (To read these files, you can download Adobe Acrobat Reader free.) Paper copies can be requested by e-mail from dherbert@nist.gov (refer to the title or Planning Report 02-3).

A non-regulatory agency of the U.S. Department of Commerce's Technology Administration, NIST develops and promotes measurement, standards, and technology to enhance productivity, facilitate trade and improve the quality of life.