I have seen a lot of project heartache that could have been avoided with better testing. However, testing is an art form and very few organizations do it well. Most people believe the extent of testing is to make sure that the system behaves as expected. But that is the easy part.
There are many aspects of testing, including unit testing (just testing the particular aspect of functionality), integration testing (making sure all of the components work end-to-end), interface testing, conversion testing, stress testing (simulating real world volumes), and exception testing (because users don’t always do what you expect). This is far from complete. The type and amount of testing will vary by project.
Most project plans that I see need to triple the amount of testing that they plan. The development of a comprehensive test plan is a key component of any IT project.
Also, testing should be auditable and signed off by the business. When I worked for SMS in the 80s we created folders for screen prints. The outside of the folder had the scenario stapled and the expected results were checked off as they were validated. Today we typically document the testing in a QuickBase application.
I feel so strongly about testing we have created a new position in our organization for a testing guru. This person will be the consultant to the various projects helping the project teams to develop effective test plans. This person will also develop our internal methodologies which we will improve over time.
Candid CIO 
Will,
I truly agree with your comments and opinion regarding testing. I also believe it is an art form but has faded over the past few years due to project deadlines and milestones. If a project is in jeopardy of not meeting its go live date, testing and documentation always seem to suffer. I am not sure how many Analysts out there (including vendors), understand the difference and necessity of the various testing stages, therefore, I believe, implementation quality has suffered.
I too worked for organizations in the 80’s where testing was regulated and managed by senior business leaders, and the end result was not turned over to the production user community until test results (which included issues logging and tracking) were documented, reviewed and signed off by business leaders, as well as IT. This included documentation of unit, interface, integration, stress, conversion, and backout procedure testing.
The idea of a Testing Guru is very appropriate to assist each project teat in the development of effective test plans. Caution must be made, however, that the Testing Guru Assists and does not get “saddled” with the development of test plans, as well as documenting and tracking of the issues log. All project teams should include appropriate resources to fulfill all requirements of the testing phase of the project, and the Testing Guru be utilized as a testing consultant, not the testing secretary.
Unfortunatley, many times the business decision is to cut out the neccesary hardware and software resources for these TEST systems. The project team ends up utilizing the systems that are about to go into production for testing, but this luxury is lost once the system is live.
On the flip side, many of these turn-key TEST systems would not be utilized on a regular basis, thus very hard to justify their existance. If these applications can utilize standard hardware components it’s much easier to set-up VMWARE occurences of these systems. However, many vendors require a proprietary environment and charge a premium for a true TEST system. “Dictaphone” comes to mind…..
How common is it to use these Test systems for data backup or recovery?
A test system should NEVER be used as the platform for disaster recovery or backup. The test system is just what it implies….a “Test System”. Though it is usually a mirror image of the production system it also has the updated version which is being tested as well as test patients/accounts, and is were fixes and updates are loaded and tested before being moved to production. A backup/disaster recovery environment should be a mirror image of the production system, including current/production (tested) versions, fixes and production data.
You are including user observation tests, right?
Steven, tell us more about user observation testing.
Testing is not so much an art form as it is a mind set. If it is viewed as important, it will be done well even if the tester is not an expert. Art is wonderful but by definition not reproducible.
User Observation testing is how well end users understand an application’s interface.