Testing isn’t just a phase of development, but rather a key component of good software development. Testing should be a way to catch problems and identify weaknesses your processes. It shouldn’t be used only as a crutch for poor development practices. Steve McConnell says it best in Code Complete…

Testing by itself does not improve software quality. Test results are an indicator of quality, but in and of themselves they don’t improve it. Trying to improve software quality by increasing the amount of testing is like trying to lose weight by weighing yourself more often. What you eat before you step on to the scale determines how much you will weigh, and the software-develompent techniques you use determine how many errors testing will find. If you want to lose weight, don’t buy a new scale; change your diet. If you want to improve your software, don’t just test more; develop better. ¹

The goal of testing is to discover errors and fix them, but you should never lose focus of your overall quality strategy. Fixing bugs is important, but recognizing opportunities to fix your process can be even more important. On those occasions where the team is overwhelmed with bug reports, it’s easy to fall into the trap of going heads down and just brute-forcing your way through fixing, but an abundance of problems is often a sign of deeper problems.

If you find during testing that a certain module or area of the application is particularly error-prone, don’t only focus on fixing the bugs. Instead, take the time to look into the development process and work with team members to understand why that area has more bugs. That way you can find the underlying causes and address those by improving your process. It could be as simple as a teaching opportunity for a junior developer or as complex as an underlying problem with your code review process. Remember, an ounce of prevention is worth a pound of cure.

Enough context, though. Let’s dive into the specific steps of a typical testing cycle. For smaller projects, you won’t necessarily need them all, but we’ll cover them all for thoroughness.

Steps of a Review Process

No matter how small an iteration may be, running it through these steps frequently will help keep bugs under control and ensure that you fix bugs earlier in your process when they’re the cheapest to find and fix. Depending on how many issues are uncovered during a given review process, you might even run through a few passes of testing and fixing before the current iteration is ready to move on, and that’s to be expected from time-to-time.

Remember, though, if you’re regularly using numerous passes through QA for each release, you might want to revisit aspects of your development process to see how you could minimize the number of issues remaining once you begin testing.

1. Code Freeze

Assuming that you have a team of people writing code, shortly before you begin a testing cycle, you’ll want to have a code freeze where nobody can add any new code. This is usually as simple as having a set time or sending out a team message for everyone to get their code committed and stop writing new code. It helps give you a consistent codebase to test and release and prevents any rushed code from breaking things right before testing. Otherwise, an ill-conceived or hurried patch could break the entire application and prevent the team from being able to test.

2. Release to QA & Smoke Test

Once the code freeze starts, and the build passes, you’ll want to get the application into your QA environment so that everyone is testing the exact same codebase. This consistency is key. If one person is testing and finds a bug in the wrong version of the codebase, it can be impossible to track down the source of the problem. So it’s always best to have everyone testing on the exact same version of your application.

Once you’ve released to QA you’ll want to do a quick smoke test to make sure that there aren’t any show stoppers that would prevent the team from being able to test a significant portion of your application. For instance, if you’re building an e-commerce store, but the shopping cart is broken, the amount of testing that you’d be able to accomplish would be severely constrained. Smoke tests help you make sure that everything is more or less in working order before starting testing.

3a. Test

Depending on the size of your team, you may have a dedicated team of testers, or your developers may simply be switching gears to test mode. Having dedicated and high quality testers that really know how to break things is ideal, but it’s not the end of the world if your developers are also doing the testing. By simply providing a focused testing period, they’re going to uncover more issues than they would otherwise.

Either way, at this point, you’d have everyone begin testing. If your developers are testing, it’s best if they’re not testing their own areas. This helps everyone look at things through fresh eyes rather than going easy testing their own areas. For example, if you build something, you’re constantly testing as you build. If you switch over to testing, you’re likely to fall into the same rut and test in the same way that you were while building it. If you’re testing someone else’s work, you’re more likely to approach it with a clear mind and find issues that the developer may have overlooked.

If you're testing someone else's work, you're more likely to approach it with a clear mind and find issues that the developer may have overlooked.

Finally, you might even want to utilize test scripts. With a test script, you write up the steps for the key interactions and have individuals walk through the process manually. While this can be done automatically with many of today’s modern testing frameworks, the benefits of having a human walk through the process and subjectively scan the interface for problems can greatly increase the number of issues found.

3b. Report

Testing and reporting generally go hand-in-hand, but the ability to write a great bug report is an incredibly valuable skill. Any developer would ideally want every detail needed to fix an issue with the initial bug report, but that’s not always possible. This can lead to a back-and-forth scenario of a developer requesting more information about a bug report and the tester providing that necessary information.

Depending on your technology, team, and experience levels, the information included in a perfect bug report will vary. Sometimes browser information is relevant, and other times it’s not. Sometimes, the authenticated user account matters, and others it doesn’t. It’s impractical for tester’s to report every possibly useful detail with every single bug report. In some cases if a tester provides too much information, it can actually be a red herring and send the developer on a wild goose chase. It is, however, entirely reasonable to expect testers to use their best judgement to provide as much information that think is likely to be relevant.

A problem well stated is a problem half solved.Charles F. Kettering

By encouraging testers to invest as much time and effort as is practical when reporting issues, both developer time and real-time turnaround of issues can be minimized. For instance, it’s clear that “Login is broken” is a poor bug report and immediately raises several questions. How is it broken? Was there an error message? Was it broken for multiple user accounts or one user account? (It could just as easily be a problem with user test data as with the login functionality.) So it’s important for testers to do their best to at a minimum identify the specific circumstances that they were testing.

4. Prioritize and Assign

After the testing period is complete, a project manager or team lead should sit down with the client/stakeholder to review, prioritize, de-duplicated, and assign all of the issues discovered. The collaboration between both internal and external team members is critical for this process to ensure agreement.

The first step is to scan all of the newly discovered issues and close any duplicates. Then, with a good overview of the number and severity of issues, each issue can be individually prioritized and assigned to the respective team member responsible for fixing it. Most of the discussion and need for agreement revolves around prioritization. Your internal team member is responsible for ensuring realistic priorities while the stakeholder will play a key role in determining what’s most important to the business. Finally, the project manager will be able to accurately determine who the best person is to fix the problem.

5. Fix

After prioritization is completed, it’s time to get to work fixing things. The only critical step here is that developers must work on fixing newly discovered bugs rather than starting new development. This ensures that all of the defects are cleaned out before beginning work on new code.

6. Release Fixes to QA

Once fixing is wrapped, you push the updated application back into your QA environment and do a quick smoke test to make sure nothing was broken during the fixing phase.

7. Retest

Arguably the most important phase of testing is retesting. After developers have fixed the outstanding issues and marked them as resolved, the testers can go through the list of their reported issues that are now marked as resolved and verify that the fix is correct. If it is, then the issue can be closed, but if it’s not correct, the issue needs to be reopened.

This is the step that separates issue tracking from a todo list, and this extra layer of process is what prevents bugs from slipping through the cracks. The best way to retest is to have the original reporter verify that it was fixed. This is the absolute last line of defense before shipping.

This is critical. You should never skip this step as this is the step that separates cursory testing from true testing. Anytime developers write new code, there’s a chance that it contains bugs. In fact, you should trust fixed code even less than original code as fixes are more likely to contain errors than the original code. ² Don’t skip retesting.

Anytime developers write new code, there's a chance that it contains bugs. In fact, you should trust fixed code even less than original code as fixes are more likely to contain errors than the original code. Don't skip retesting.

What’s next?

A publisher would never release a book without an editor reviewing it, and you shouldn’t ship software without testing it. Even with automated tests, continuous integration, static code analysis, and more, you still need human testing to uncover problems that can’t be discovered by machines. Even more important, though, is that you don’t view testing as the end of your quality assurance strategy.

Even after you ship, you're still going to have bugs, and having the processes in place to enable you to quickly respond to and address bugs that pop up after you ship is just as important as any other quality assurance steps. So next we'll go over the ways to help you catch and fix any problems that arise in production by setting up tools and processes that support listening not only to your customers but to the heartbeat of your application.

1. Steve McConnell, Code Complete, Second Edition (Microsoft Press, 2009), 501. ↩
2. Glenford J. Myers. The Art of Software Testing, Second Edition (John Wiley & Sons, 2004), 120. ↩