|
|
|
|
Are you tired of your projects ending up severely “challenged” and missing their commitments for schedule, budget and scope? TenStep has the full solution of products and services to help your organization successfully execute projects. Contact them today at info@TenStep.com. They will work on the best package of projects and services to meet YOUR organization’s needs.
Regression testing is a technique that can be used in a number of the acceptance tests and system tests. There are two main objectives to regression testing. The first is to make certain that a component with a defect that has been found and fixed actually does work and provides the expected outputs. Very often, programmers will “tweak” the code and report the defect fixed without truly resolving the cause of the defect. The second objective is to ensure that the fix of the defect does not break any other code and introduce new errors in the component. It is in fact this possibility that makes regression testing necessary. Regression testing validates the overall integrity of the system, not the fix of the initial defect itself.
Regression testing involves creating a complete set of test data, running the test, and then documenting the results. When changes are made to the solution, this same set of test cases is rerun and the results are compared to the prior standard to look for unexpected discrepancies. For example, you may have 500 test cases that have been built through the testing process. These test cases are all executed during the acceptance test. The results of the test may include a handful of errors. These errors are fixed and the same set of test data is run again. However, this time a handful of other problems emerge that did not show up the first time. This has pointed out two potential problems. One is that the correction of a prior error has led the logic down a path to another error in code that was not exercised previously. The other potential problem is that correcting one or more of the prior errors actually introduced another error that did not exist before.
There is a common rule in testing that says where there is one bug, there are usually more waiting to surface. Your regression test cases should be focused on casting a wider net of coverage around the location of the defect. This means setting up test cases that exercise as many portions of code as possible, as well as actually adding test cases that focus on areas of known defects. The test team should also add test cases to features and functions related to the known defects. For instance, if there is a defect in the customer lookup process, additional test cases should be added to any areas of the solution where that similar customer lookup process takes place.
Regression testing is especially valuable after a solution goes live and new releases of the solution are issued. After changes are made to the solution, new test cases are added to the pre-existing set of test cases. The full set of test cases is executed, and the results are compared with the results from prior tests. All discrepancies are either validated given the new changes or identified as errors that need to be fixed.
In selecting regression test cases, consider these best practices.
Automate, automate, automate. Regression tests that are automated are the least expensive to run since they simply take up computer time. If possible, the validation of results and comparison to prior baseline runs should be automated as well.
Reduce your test suite size. Some people think that the more test cases they build, the better the potential results. That is true to a point, but there is also a point where more test cases just become hard to maintain and require more work to determine which test cases are associated with particular errors. The better approach is to keep the fewest number of test cases that will thoroughly test the solution. This includes eliminating duplicate tests. You may have multiple test cases with differing sets of data; however, all of them test the same basic features and functions. You may not want to narrow these test cases down to one, but you probably don't want 50 similar test cases either.
Add narrow tests for specific errors, but then phase them out over time. It is a common practice, and a good practice, to add test cases in areas where there are known errors, especially if related errors keep occurring in the same area. In fact, some of these cases may overlap; however, you are trying to test as many nuances in the logic as possible. Once these areas of the solution become stable, the test cases can be safely reduced. Test cases that are too narrow in coverage are worth keeping for a short time and should be added into the mix in multiple cycles just to make sure the defect does not reappear. However, the tests can then be archived. They will be available if similar errors occur in the future, but they will not overwhelm the testers by continuing to add more and more test cases to the regression test.
Create smoke test suites. These regression test suites should include test cases that have a relatively low risk of finding defects but that validate the integrity of the system. These include test cases that are designed to work correctly the entire way through. Their purpose is not necessarily to detect errors, but to provide a valid set of working test data that can exercise the entire solution.
Pay special attention to user errors. Test cases should be drawn up from defects found by your clients. This type of defect found by a user suggests a gap in test coverage, and a new test case should be created to close the gap and protect the system integrity.
Running these types of regression test cases will help complete your testing effort on a timely basis, especially if they can be automated. They will also make certain that errors are identified and fixed quickly. While regression testing is tedious, it is helpful for resolving and preventing future defects as well as for creating many test cases that can be automated and reducing the overall testing effort costs.
Each month, Tom Mochal presents techniques and processes for IT development projects. Tom is the recent winner of the 2005 PMI Distinguished Contribution Award. His company, TenStep, Inc. develops business methodologies, including a project management process called TenStep (www.TenStep.com) and a project lifecycle process called LifecycleStep (www.LifecycleStep.com).
