What about bugs?

Given enough eyeballs, all bugs are shallow. Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone.—Eric S. Raymond, programmer and advocate of open source software, from The Cathedral and the Bazaar.

Rating defects is often a nightmare. Bad sales people tell their customers to rate everything a Sev1 to ensure the feature gets delivered. Bad QA leads use bug ratings to prove a political point. Bad developers rate critical bugs as Sev4 so they don’t have to work on it.

One company instituted a new rating system: “We’re customer driven so let’s have customers rate the severity!” Seemed like a good idea at first. But within days, there was a new field in the trouble ticketing system: “real severity.” So many customers rated their feature requests as a Sev1 that the rating system broke down so customer support set up a new field for realistic severity ratings using an objective approach.

You’ll want to implement an objective rating system for defects. A common technique is to rate defects by severity using a 4-point scale, like this:

• Severity 1 (Sev1): system outage—the product is not working

• Sev2: critical functionality not working

• Sev3: non-critical functionality not working

• Sev4: not working as desired or enhancement request

The people in development and support should be clear on this rating system. One team spent 2 or 3 hours daily debating the severity of reported defects. This is really unproductive! Better to classify them as they come in and let the scale determine which will be addressed. This triage can typically be done by customer service and quality assurance without involving development or product management.

Major defects need to be fixed. These items go on top of the list, which means you’ll take a hit in team productivity. If a team has a history of finishing 10 stories each week, an influx of bugs will pull the team off new development and deploy them on bug fixes, so the team will finish fewer stories that week. If this continues, you’ll see your team’s historical productivity drop to 8 stories a week—or 5—because of defects and technical debt.

Generally a Sev1—system outage—must be fixed immediately. Sev2’s and Sev3’s can wait for a dev cycle; Sev4’s will be put into the queue for future consideration.

Consider a playbooking product. A customer wants to have a running total shown on the checkbook register. How would you rate this? It’s not a feature that is failing but a new request; therefore it’s a Sev4. Still, it’s a good idea so we’ll probably put it high up in our backlog but it’s not a showstopper.

Another example: a report doesn’t total correctly. Is that a Sev2 or a Sev3? In this case it depends on whether the report is critical or not. If it’s a critical daily report, then it’s a Sev2; if it’s a non-critical report than isn’t run often, it’s a Sev3.

Think of it another way. Do you need to stop development of new functionary to fix this? Does it affect your company operations? Does it affect the customer’s operations? Is it worth delaying the next release?

There will always be bugs. After all, your developers are human. They sometimes make mistakes. Our products are increasingly complex and it’s sometimes hard to see the impact a change might make elsewhere.

In the end, severity rating shouldn’t be a major weekly event. Use an objective system so the team is always focused on the most important work.