Dovel Technologies (Dovel) has been at the forefront of implementing information technology systems for the United States Government for over 12 years. Periodically, we take stock of our performance as part of our Dovel World Class Quality Initiative and we compare results to the previous years. As part of our commitment to continual improvement, we analyze our past performance focusing on key quality metrics from software deliveries between 2008 and 2017. This report summarizes Dovel’s 2017 achievements that will serve as the benchmark for future goals.
Dovel has been a Capability Maturity Model Integration® (CMMI) maturity level 3 appraised company since 2008. This achievement is a testament to our corporate commitment and attention to quality, which provides significant benefits to our clients and our employees. In 2017 Dovel conducted a fourth CMMI maturity level 3 appraisal and the results confirmed Dovel’s strong performance.
Since 2011, our appraisals have included Dovel’s agile software development methodology. This report includes metrics captured for projects using incremental and agile methods. Dovel’s success with multiple software approaches allows clients the flexibility to choose the methodology that best fits their strategy.
In order to ensure an objective view of Dovel’s performance, we requested information for this report from Mr. Capers Jones, a well-known software engineering industry expert. Additionally, we have shared in the past the results of our annual survey with Mr. Jones who publishes research on companies’ quality metrics. We hereby thank him for the wealth of advice he provided us.
What are we measuring and why?
Strong companies collect metrics to help operational performance and decision making, and Dovel is no exception. We operate a robust Measurement & Analysis (M&A) program aimed at continual improvement. The Dovel metrics program measures specific aspects of software quality. Defect potentials and defect removal efficiency are two high-level metrics.
Capers Jones describes these metrics as follows:
The term ‘defect potentials’ refers to the total quantity of bugs or defects that will be found in five software artifacts: requirements, design, code, documents, and ‘bad fixes’ or secondary defects.
The term ‘defect removal efficiency’ refers to the percentage of total defects found and removed before software applications are delivered to customers.
As of 2007, the U.S. average for defect potentials was about five (5) defects per function point. The U.S. average for defect removal efficiency was only about 85%. The U.S. average for delivered defects was about 0.75 defects per function point.
Defect potential and defect removal efficiency are clear indicators of quality software, a major goal of Dovel Technologies. We compare each release to the previous set of releases in order to measure and report improvement. Collecting and analyzing these metrics, and vowing to improve on them, ensures that Dovel’s performance remains consistent year over year.
Dovel’s 2017 Performance
The figure below shows Dovel’s defect potentials and defect removal efficiency from 2008 through 2017. We measure defect potential in terms of function points. Function points measure the logical size of functionality delivered to an end user and are an industry standard for measuring the size of a software system. We measure defect removal efficiency as the percentage of defects that were found and removed before the software release was deployed to the client’s production system.
Figure 1 – Dovel Software Release Quality Compared to Industry
Figure 1 plots the observed defects per function point and defect removal efficiency for each release in our sample compared to major software quality ranges described by Capers Jones research. Dovel’s performance improved since measurement began in 2008 and continues in a range that meets or exceeds industry benchmarks. In 2014, we took over a large software system with a struggling methodology, and in 2015 we transitioned this system to an incremental development approach aligned to the client’s business requirements process and we applied our measurement program. Throughout 2016, we continued our focus on defect detection and drove improvements in defect removal efficiency. In 2017, as the system grew in size and active users, we have scaled our processes from three to seven development teams while maintaining the same predictable quality levels. Our 2017 numbers reflect both the continued high performance of our existing programs and demonstrate how our development approach is able to deliver best in class defects per function point and defect removal efficiency on very large software systems.
After reviewing Dovel’s 2017 data, Mr. Capers Jones commented:
“The 2018 edition of the Dovel Quality Initiative continues a long trend of generating useful quantitative data on the critical topic of software engineering quality. My clients all find the report to be of use. Software quality is poorly understood and the Dovel report is a good source of current data.”
What is the benefit to our clients?
Low defect potentials and high defect removal efficiency rates are substantially beneficial to a client. Defects are a major contributor to schedule slippage and, therefore, a major contributor to the total cost of ownership of IT systems. Defects are the main source of user dissatisfaction with IT solutions and their decision to cancel systems efforts. Thus a high defect removal rate—which measures the number of defects reported by users in the first 90 days of system implementation—directly correlates to high client satisfaction. After all, who wants to use a system with a high number of defects?
Dovel has the experience and flexibility to employ incremental and agile methods to achieve results with minimum staffing levels and shorter schedules, reducing cost while increasing responsiveness and alignment to user needs. When comparing the Dovel data to industry, we have achieved a high quality level with 0.3 defects per function point (compared to an industry average of 5.0) as well as a 93% defect removal rate (compared to an industry average of 85%).
Less defects and higher defect removal leads to higher quality solutions with greater user adoption and less cost to the customer.
This is why Dovel can deliver a high-quality solution to clients, and increase user satisfaction.
1 Jones, C. (2008, June). Measuring Defect Potentials and Defect Removal Efficiency. Crosstalk, 11-13.