Smarter quality management

Organizations that create and deliver software — whether for their own IT operations, for the packaged applications market, or as the core of their final product, as in the systems space — must grapple not only with today’s competitive landscape, but also with increased complexity in their processes and supply chains. Many factors serve to complicate software delivery, but differentiation lies at the heart of managing this complexity.

Here are a few examples. In the products arena, customers demand more from the software components designed for the latest hardware, often with requirements that change rapidly, even as software projects are under way. Keeping track of changes while meeting aggressive (and unaltered) deadlines is difficult, if not impossible. In the IT space, more businesses are focused on their operational software for capturing and providing value to their customers and lines of businesses. Ecommerce websites compete to improve customer relations and simplify online business; businesses that create highly optimized supply chains supporting fast, efficient ecosystem of partners quickly rise in the marketplace.

Effective quality management creates opportunities to deliver key business benefits, such as improved market share, higher customer satisfaction, and increased brand equity. But top quality in the completed product cannot serve as the single guiding principle by which products are produced and delivered. Time to market is also key; costs and risk factors must also be part of the balancing act. Get these things wrong, and you may face unsustainable costs, missed windows of opportunity, unhappy customers, even a massive recall or the complete failure of a system at a critical moment. Get these things right, and you can achieve a positive operational return on investment from efficiencies gained in development activities.

This paper introduces a practical approach to test management (TM) that helps reduce cycle time without sacrificing quality. The underlying concepts presented here will be familiar to software project managers, especially those with TM experience. Read on to understand the fundamentals and essential processes of test and quality management.

Here’s one way to understand the “soft” in software: it is relatively easy to change. But for software designed for the commercial space, where the competitive pressures described above govern a software project’s success or failure, the “soft” feature happens to be one of its riskiest attributes.

That’s because software projects are seldom designed and manufactured as in traditional engineering projects — a bridge, for example. While a bridge is engineered through traditional planning and architecture based on the laws of physics, then produced according to an organized plan with a division of labor, software on the other hand is, at its essence, simply information.

Many industry leaders compare software production to movie production: a collaboration involving a team of craftsmen and emerging from the naturally creative process of artistic yet technical people.

Over the past two decades, this unique feature of software has been understood and embraced by iterative development practitioners, who now tend to develop software in stages called “iterations.” Each iteration delivers a working, functional version of the software under development, so it can be reviewed, tested, and vetted by stakeholders and other teams seeking adherence to the original project requirements.

This allows project managers to make smaller, incremental course corrections during the project lifecycle — thus ensuring the final deliverable is close to expectations — as opposed to having separate teams work according to a plan, assemble various components near project end, and discover major failures due to integration or deployment complexities.

For testing teams, the iterative development process integrates quality management and continuous testing across all stages of the project work flow, as opposed to relegating test activity to the end of the project. We will describe the role of iterative development-based quality management and continuous testing more fully in future sections.

What is the role of the test management team during the iterative lifecycle? What do they test for, and how do they know what is supposed to change from one iteration to the next?

As previously noted, traditional software testing only occurs late in the lifecycle, after multiple coding teams have spent much time and effort to deliver their components toward the complete project. Because these traditionally managed projects proceed according to strictly described requirement sets, and various component teams focus on their portions alone, it is up to the testers to discover the discontinuities and malfunctions as these components are assembled — then it’s the testers who must deliver the bad news that much rework has to be done in order to get the project back on track.

A relatively modest, first iteration may only address 15 percent of the full set of project requirements, but as a functional module of working code, the completed iteration can be demonstrated, and tested. So, any defects discovered by test teams at this early stage have a proportionally small impact on the larger development team, who make the fixes, then proceed to the next iteration where more of the requirements can be incorporated into the working version (iteration) of the software.

The number of iterations required by any software project depends on many factors, of course, such as the complexities of the team’s supply chain, the complexity of the software under development, the physical location of team members (are they geographically distributed, perhaps internationally? or are they co-located under a single roof?), and the competitive demands that determine optimum time to market.

During any software delivery process, “When do we release?” is a key question with no simple answer. You must consider project-specific variables, such as the cost of delays, the opportunity value of early delivery, marketplace quality expectations, and the costs associated with defects. Ultimately, the delivery strategy will be based on the actual or perceived importance of each variable.

The optimal time to release is when the total risk exposure is minimal, typically around the time where the risk associated with competitive threats starts to outweigh the risk reduction associated with further quality improvements, as illustrated in Figure 1.

The best time to release varies widely based on your software delivery strategy and your target market. Software delivery for both IT (internal business systems) and smart products (products using embedded software, including “system-of-systems” design) is typically dominated by one of two motivators, depending on the organization’s target market: time to market (schedule-driven), or quality impact (quality-driven).

For the remainder of this paper, we will assume that the software delivery lifecycle is based on a risk-driven approach. We will explore how to bend the green curve shown in Figure 4 downward and to the left, for reduced time to market without compromising the risk profile.

If a faster reduction in risk is the goal, how do you achieve it? The answer is not through testing, which is focused simply on discovering defects. In traditional testing practices, testing is considered a late stage activity, squeezed between an often-late development handoff date and an immovable ship date. Not only does this practice fail to yield the benefits of incremental, iterative development techniques explained earlier; it also minimizes, or at best reduces, the amount of time spent on quality assurance, and makes fixes all the more difficult unless you’re willing to compromise the release date.

As noted earlier, iterative development techniques greatly improves on this scenario, by having functional units tested incrementally, in stages, throughout the lifecycle, rather than leaving the testing phase until project completion. And test management takes this improvement a step further. ²

By choosing test and quality management practices with the potential to deliver a positive ROI within a relative short amount of time, you can justify risk reduction measures from not only a quality standpoint but also a financial standpoint.

Approaching quality from a full lifecycle perspective — as opposed to assuming that testing activity will suffice — should not seem like such a radical idea. After all, testing a product is an engineering task just like development: the requirements need to be analyzed by the test architect, its testing strategy has to be defined, the relevant test benches and test frameworks need to be built (designed and implemented), and more. These engineering development processes are very similar to the ones followed during product development. In fact, product development and testing can be viewed as two parallel development threads emanating from the same requirements, with many synchronization points between the threads, up to the point of the testing activity itself where a verdict is made based on meeting expectations or not (as expressed in the test cases). This applies to both the traditional development process as well as Agile methods (with variants such as Test-First Development and Test-Driven Design). The important point is that TM is a thread that must run in parallel to development, especially in Agile development.

Although a complete discussion of TM is beyond the scope of this paper, we can show how TM reduces risk — and thus allows earlier software delivery without compromising quality — by demonstrating how one of the test management best practices can improve iterative testing within the software development lifecycle.

These are some of the test management best practices, each of which contributes to risk reduction and therefore increased quality and reduced cost. Now, let’s consider the overall impact of test management on the defects density and the cost of fixing them.

Test management best practices center on quality synchronization points across the whole development lifecycle. We have discussed the benefits of traceability to requirements, and we have briefly noted collaboration between stakeholders and the test team, as well as automated reporting. Other best practices include: allowing quality professionals to contribute to the team effort from the very beginning of a project; the integration of practitioners testing as part of test management; and the use of consolidated quality dashboards.

Together, these test management best practices can benefit the overall business in measurable ways. Using CMMI³ as a proxy for the maturity of the development process, Figure 5 shows that the overall business impact of test management is quite compelling.

Figure 5: Graphing percentage of defects detected (Y axis) against an organization’s software development maturity level (X axis).

Let’s assume an organization at CMMI level 2, with 1,000 defects detected during functional testing. Figure 5 shows that on average, without QM practices, about 30 percent of the defects are being detected in functional testing (the left, blue bar), and therefore the total number of defects is 3300. However, by applying TM practices, the defect detection rate increases to 58 percent (the right, green bar), therefore detecting 1914 (58 percent of 3,300), or 914 more defects.

As fixing defects during User Acceptance Testing (UAT) is about 7 times more expensive than during Unit/Integration test, and assuming a fix cost of USD 120 per defect during unit/integration test, fixing 914 defects in UAT is already increasing the cost by over half a million dollars.

This also does not take into account less-tangible savings, such as increased quality, customer retention, and other implications of quality as a differentiating asset.

As most of the development teams are around CMMI levels 2 and 3, the benefits and the savings that were described above apply to most of the industry. But as development teams become more mature, apply TM practices and move up to CMMI levels 4 and 5, the focus shifts into less tangible benefits (but for some, even more important) such as reduction in the number of defects that are introduced in the first place, measured improvements around planning/execution of quality related activities, customer retention, and leveraging quality as a differentiating asset.

In this paper, we have described several improvements to methods used by software teams in the design, testing, and deployment of software for systems or IT. Teams may use these test management methods to deploy that software more quickly, while mitigating quality-related risks throughout the lifecycle. The multi-disciplined practice of test management is breaking the functional and organizational silos that are so common in today’s companies. It encourages an analytical process that’s closely integrated with the development lifecycle.

Analyzing the market and best practices shows that business outcomes can be optimized, and that smart improvements within the realm of proven best practices for requirements management and traceability, collaborative test planning and automated reporting — a combination of disciplines that defines test management — can help address the need for increased innovation with more competitive products and services to your customers.

With straightforward adjustments to your test management processes, deployment practices and tooling, we can help you realize significant savings that help improve the customer’s bottom line.

We look forward to working with you.