Do you make these 7 test planning mistakes?

There is a very good discussion on PMStudent.com about an article by Jennifer Bedell titled “Do testers goldplate too?”.

Jennifer says in her post:

Goldplating by a tester can occur when a tester goes beyond the stated requirements in an effort to produce a “quality” product.  A tester may feel that their suggestion would improve the customer experience so they log this in the defect log.  While their suggestion may do exactly what they envisioned, if it was not within the scope of the stated requirements, it becomes a form of goldplating or “feature creep”.   A tester’s job is to ensure that a quality product is delivered, but many testers rely on their own definition of quality rather than using the requirements to define quality.

I have managed projects where the testing phase got out of control. The test team kept coming up with more things to test and more reasons to defer the Go Live date. This situation was extremely demoralizing to the entire project team.

Having lived this nightmare more than once, I always come to the same conclusion. As Project Managers, we learned to put so much time and effort in getting the requirements right, but we neglect to invest enough effort in developing a robust Test Plan.

A robust test plan puts boundaries around testing and leads to a controlled scope. In my experience, I see these 7 weaknesses in test planning as the root cause for scope creep and gold plating:

1) No written test scope or test plan

Testers want to do a great job. However, without a written clear scope, they can end up spending valuable project hours testing unimportant stuff. Many hours can be wasted arguing with the QA team or customers about whether a test is in scope or not. Make sure to specify not only what will be tested but also what will not be tested and why. Once the plan is approved, it should be baselined and every subsequent change should go thru change management.

2) Unclear Roles & Responsibilities

Make sure it is clear from the start of the project what the tester(s) will do in the project. Sometimes testers get confused about their role. There is a big difference between testing, Quality Assurance, and Quality Control. The definition of these is not always clear or consistent. With an overzealous QA manager, getting agreement on these terms and what is applicable to your project can consume hours and lead to frustration. Make sure test team is clear on what their role is as early as possible in the project, before conflict starts and people raise their defenses. If there is a disagreement, you can at least address it at the start of the project and determine what testing or QA will cost your project.

3) No Change management process.

Once the test plan is baselined (approved), every change to the plan should go thru change management process. This will give you the ability to show the cost of every change and let the sponsor approve or deny. Quality has its price and to approve a change or reject it is a business decision for the sponsor to make. Force the decision to be about value and cost and remove the emotions out of the debate.

4) No clear or defined acceptance criteria

With no clear or defined acceptance criteria, too many hours can be spend debating with the test team about which defects are considered showstoppers and which ones can be fixed after the Go Live. Your Go Live date will be held hostage at the discretion of the test team, especially if they report to an uncompromising Test Manager.

Make sure you know what constitutes a showstopper as early as possible in the project. Definitely before the test phase begins.

Make sure to agree as early as possible on types of defects or issues priorities so that it is clear:

  • What showstoppers must be fixed before the Go Live. These bugs can be priority 1.
  • What can wait until post-Go Live, if we cannot resolve before the Go Live. These bugs can be priority 2.
  • What can be deferred to future phase/projects or handled as part of the ongoing maintenance and support. These bugs can be priority 3.

Only priority 1 can stop the Go Live. Everything else can wait until after the Go Live.

5) Missing or incomplete Requirements Traceability Matrix

Make sure each requirement is covered by a test case/scenario. Indicate the reasons why a requirement is not covered by a test, so there is no confusion later down the road. Make sure this is reviewed and approved by the all stakeholders, before testing starts. This way only bugs resulting from a valid and planned test case can be worked on. With a traceability matrix in your hand, you can have a meaningful discussion if there is a debate about new test cases that have not been planned. Again, these can be discussed as Change Orders with their budget and schedule impact taken into account.

6) Testing Tools, Techniques, and Methodologies

Will you be using a testing tool such as Inflectra on this project? The organization can suddenly require that all testing get performed using a new tool/platform, after you already baselined your schedule and budget. This can add to project timeline and budget. Your project test plan should indicate what tools will be used. What if the team requires training in the new tools. This all requires you to apply change management process to ask for more budget and/or more time. But you can only apply change management, if you have a baselined test plan.

7) Applicable Standards and guidelines

What applicable standards and guidelines were in effect when the project started? The baseline budget and schedule can be impacted when the test team decides to adopt new standards and guidelines for testing. For example, you project can suddenly be required by the test manager (at their discretion) to use a specific testing methodology. Or some standards or guidelines change and now you have to make your project compliant with the new standards. You should be able to treat this as a change request and apply change management process.

As Johanna Rothman said “Shipping the product is a business decision”. I agree and it should never be a quality decision. But without a robust test plan, the test effort on your project can take on a life of its own. Avoid these mistakes and set your project up for success by investing enough time and effort to build a robust test plan.

5 Responses to Do you make these 7 test planning mistakes?
  1. Don R. James
    February 21, 2010 | 4:55 pm

    Excellent!
    The ‘V’ Model gives us the locations for testing and the locations where test plans should be developed.
    Too many times, we forget that a comprehensive review of the baselined requirements v. the test plans (and scripts, and scenarios) is a System Test.
    Think about it.
    DR J

    • samad_aidane
      February 23, 2010 | 5:12 pm

      Don, thank you so much for taking the time to read the post and comment. Unfortunately, testing gets shortchanged and is the first to be cut when deadlines and/or budgets become priority. I think as project managers we need to support our test teams and make sure that we follow the principles as those promoted by the V model. Thank you again. I will be checking you website http://www.pmotogo.com

  2. Shim Marom
    March 13, 2010 | 6:55 pm

    Hi Samad, just read your post and it is spot on. Just about the sort of issues we would face if we did not establish the right process for managing our testing effort from start to end.

    Cheers, Shim
    http://www.quantmleap.com

    • samad_aidane
      March 16, 2010 | 2:30 am

      Shim, Thank you for stopping by and commenting.

      I agree. I think testing does not get the attention it deserves. Also, I feel bad for how test teams get treated on some projects as second class citizens. I always take care of my test team because they end up saving my projects from all the blind spots that surround them.

      I see the absence of establishing the right process for managing test effort as one of the big sources for scope creep. I worked on projects where we spend all the time making sure the requirements are tights but we kept the test scope wide open. In the middle of one my projects, the organization hired a zealous QA Manager who saw themselves above any budget or schedule constrains and kept prolonging the test phase and would never agree to a Go Live date. I learned a big lesson on that project and I vowed to never get burned that way again. Now I spend a lot of time with my test team on the test plan and acceptance criteria for each test case. I want to make sure we know what will be tested, where (environment), when, by whom and for how long. I also now have 3 categories of bugs. 1-showstopper, 2-can be fixed after the Go live, and 3-can be fixed sometime in the future after the go live. this way we only focus on the showstoppers as we get closer the Go Live date and not let this date be held hostage by the customer or QA team.

      Thank you again.

  3. […] Guerrilla Project Management: Do you make these 7 test planning mistakes? Tags: agile, communication, JIRA, JIRA Agile, leadership, Lean Processes, plan time, planning, product management, productivity, progress tracking, project, project management, project reporting, resource planning, Tempo Planner […]

Do you make these 7 test planning mistakes?

There is a very good discussion on PMStudent.com about an article by Jennifer Bedell titled “Do testers goldplate too?”.

Jennifer says in her post:

Goldplating by a tester can occur when a tester goes beyond the stated requirements in an effort to produce a “quality” product.  A tester may feel that their suggestion would improve the customer experience so they log this in the defect log.  While their suggestion may do exactly what they envisioned, if it was not within the scope of the stated requirements, it becomes a form of goldplating or “feature creep”.   A tester’s job is to ensure that a quality product is delivered, but many testers rely on their own definition of quality rather than using the requirements to define quality.

I have managed projects where the testing phase got out of control. The test team kept coming up with more things to test and more reasons to defer the Go Live date. This situation was extremely demoralizing to the entire project team.

Having lived this nightmare more than once, I always come to the same conclusion. As Project Managers, we learned to put so much time and effort in getting the requirements right, but we neglect to invest enough effort in developing a robust Test Plan.

A robust test plan puts boundaries around testing and leads to a controlled scope. In my experience, I see these 7 weaknesses in test planning as the root cause for scope creep and gold plating:

1) No written test scope or test plan

Testers want to do a great job. However, without a written clear scope, they can end up spending valuable project hours testing unimportant stuff. Many hours can be wasted arguing with the QA team or customers about whether a test is in scope or not. Make sure to specify not only what will be tested but also what will not be tested and why. Once the plan is approved, it should be baselined and every subsequent change should go thru change management.

2) Unclear Roles & Responsibilities

Make sure it is clear from the start of the project what the tester(s) will do in the project. Sometimes testers get confused about their role. There is a big difference between testing, Quality Assurance, and Quality Control. The definition of these is not always clear or consistent. With an overzealous QA manager, getting agreement on these terms and what is applicable to your project can consume hours and lead to frustration. Make sure test team is clear on what their role is as early as possible in the project, before conflict starts and people raise their defenses. If there is a disagreement, you can at least address it at the start of the project and determine what testing or QA will cost your project.

3) No Change management process.

Once the test plan is baselined (approved), every change to the plan should go thru change management process. This will give you the ability to show the cost of every change and let the sponsor approve or deny. Quality has its price and to approve a change or reject it is a business decision for the sponsor to make. Force the decision to be about value and cost and remove the emotions out of the debate.

4) No clear or defined acceptance criteria

With no clear or defined acceptance criteria, too many hours can be spend debating with the test team about which defects are considered showstoppers and which ones can be fixed after the Go Live. Your Go Live date will be held hostage at the discretion of the test team, especially if they report to an uncompromising Test Manager.

Make sure you know what constitutes a showstopper as early as possible in the project. Definitely before the test phase begins.

Make sure to agree as early as possible on types of defects or issues priorities so that it is clear:

  • What showstoppers must be fixed before the Go Live. These bugs can be priority 1.
  • What can wait until post-Go Live, if we cannot resolve before the Go Live. These bugs can be priority 2.
  • What can be deferred to future phase/projects or handled as part of the ongoing maintenance and support. These bugs can be priority 3.

Only priority 1 can stop the Go Live. Everything else can wait until after the Go Live.

5) Missing or incomplete Requirements Traceability Matrix

Make sure each requirement is covered by a test case/scenario. Indicate the reasons why a requirement is not covered by a test, so there is no confusion later down the road. Make sure this is reviewed and approved by the all stakeholders, before testing starts. This way only bugs resulting from a valid and planned test case can be worked on. With a traceability matrix in your hand, you can have a meaningful discussion if there is a debate about new test cases that have not been planned. Again, these can be discussed as Change Orders with their budget and schedule impact taken into account.

6) Testing Tools, Techniques, and Methodologies

Will you be using a testing tool such as Inflectra on this project? The organization can suddenly require that all testing get performed using a new tool/platform, after you already baselined your schedule and budget. This can add to project timeline and budget. Your project test plan should indicate what tools will be used. What if the team requires training in the new tools. This all requires you to apply change management process to ask for more budget and/or more time. But you can only apply change management, if you have a baselined test plan.

7) Applicable Standards and guidelines

What applicable standards and guidelines were in effect when the project started? The baseline budget and schedule can be impacted when the test team decides to adopt new standards and guidelines for testing. For example, you project can suddenly be required by the test manager (at their discretion) to use a specific testing methodology. Or some standards or guidelines change and now you have to make your project compliant with the new standards. You should be able to treat this as a change request and apply change management process.

As Johanna Rothman said “Shipping the product is a business decision”. I agree and it should never be a quality decision. But without a robust test plan, the test effort on your project can take on a life of its own. Avoid these mistakes and set your project up for success by investing enough time and effort to build a robust test plan.

5 Responses to Do you make these 7 test planning mistakes?
  1. Don R. James
    February 21, 2010 | 4:55 pm

    Excellent!
    The ‘V’ Model gives us the locations for testing and the locations where test plans should be developed.
    Too many times, we forget that a comprehensive review of the baselined requirements v. the test plans (and scripts, and scenarios) is a System Test.
    Think about it.
    DR J

    • samad_aidane
      February 23, 2010 | 5:12 pm

      Don, thank you so much for taking the time to read the post and comment. Unfortunately, testing gets shortchanged and is the first to be cut when deadlines and/or budgets become priority. I think as project managers we need to support our test teams and make sure that we follow the principles as those promoted by the V model. Thank you again. I will be checking you website http://www.pmotogo.com

  2. Shim Marom
    March 13, 2010 | 6:55 pm

    Hi Samad, just read your post and it is spot on. Just about the sort of issues we would face if we did not establish the right process for managing our testing effort from start to end.

    Cheers, Shim
    http://www.quantmleap.com

    • samad_aidane
      March 16, 2010 | 2:30 am

      Shim, Thank you for stopping by and commenting.

      I agree. I think testing does not get the attention it deserves. Also, I feel bad for how test teams get treated on some projects as second class citizens. I always take care of my test team because they end up saving my projects from all the blind spots that surround them.

      I see the absence of establishing the right process for managing test effort as one of the big sources for scope creep. I worked on projects where we spend all the time making sure the requirements are tights but we kept the test scope wide open. In the middle of one my projects, the organization hired a zealous QA Manager who saw themselves above any budget or schedule constrains and kept prolonging the test phase and would never agree to a Go Live date. I learned a big lesson on that project and I vowed to never get burned that way again. Now I spend a lot of time with my test team on the test plan and acceptance criteria for each test case. I want to make sure we know what will be tested, where (environment), when, by whom and for how long. I also now have 3 categories of bugs. 1-showstopper, 2-can be fixed after the Go live, and 3-can be fixed sometime in the future after the go live. this way we only focus on the showstoppers as we get closer the Go Live date and not let this date be held hostage by the customer or QA team.

      Thank you again.

  3. […] Guerrilla Project Management: Do you make these 7 test planning mistakes? Tags: agile, communication, JIRA, JIRA Agile, leadership, Lean Processes, plan time, planning, product management, productivity, progress tracking, project, project management, project reporting, resource planning, Tempo Planner […]

Do you make these 7 test planning mistakes?

There is a very good discussion on PMStudent.com about an article by Jennifer Bedell titled “Do testers goldplate too?”.

Jennifer says in her post:

Goldplating by a tester can occur when a tester goes beyond the stated requirements in an effort to produce a “quality” product.  A tester may feel that their suggestion would improve the customer experience so they log this in the defect log.  While their suggestion may do exactly what they envisioned, if it was not within the scope of the stated requirements, it becomes a form of goldplating or “feature creep”.   A tester’s job is to ensure that a quality product is delivered, but many testers rely on their own definition of quality rather than using the requirements to define quality.

I have managed projects where the testing phase got out of control. The test team kept coming up with more things to test and more reasons to defer the Go Live date. This situation was extremely demoralizing to the entire project team.

Having lived this nightmare more than once, I always come to the same conclusion. As Project Managers, we learned to put so much time and effort in getting the requirements right, but we neglect to invest enough effort in developing a robust Test Plan.

A robust test plan puts boundaries around testing and leads to a controlled scope. In my experience, I see these 7 weaknesses in test planning as the root cause for scope creep and gold plating:

1) No written test scope or test plan

Testers want to do a great job. However, without a written clear scope, they can end up spending valuable project hours testing unimportant stuff. Many hours can be wasted arguing with the QA team or customers about whether a test is in scope or not. Make sure to specify not only what will be tested but also what will not be tested and why. Once the plan is approved, it should be baselined and every subsequent change should go thru change management.

2) Unclear Roles & Responsibilities

Make sure it is clear from the start of the project what the tester(s) will do in the project. Sometimes testers get confused about their role. There is a big difference between testing, Quality Assurance, and Quality Control. The definition of these is not always clear or consistent. With an overzealous QA manager, getting agreement on these terms and what is applicable to your project can consume hours and lead to frustration. Make sure test team is clear on what their role is as early as possible in the project, before conflict starts and people raise their defenses. If there is a disagreement, you can at least address it at the start of the project and determine what testing or QA will cost your project.

3) No Change management process.

Once the test plan is baselined (approved), every change to the plan should go thru change management process. This will give you the ability to show the cost of every change and let the sponsor approve or deny. Quality has its price and to approve a change or reject it is a business decision for the sponsor to make. Force the decision to be about value and cost and remove the emotions out of the debate.

4) No clear or defined acceptance criteria

With no clear or defined acceptance criteria, too many hours can be spend debating with the test team about which defects are considered showstoppers and which ones can be fixed after the Go Live. Your Go Live date will be held hostage at the discretion of the test team, especially if they report to an uncompromising Test Manager.

Make sure you know what constitutes a showstopper as early as possible in the project. Definitely before the test phase begins.

Make sure to agree as early as possible on types of defects or issues priorities so that it is clear:

  • What showstoppers must be fixed before the Go Live. These bugs can be priority 1.
  • What can wait until post-Go Live, if we cannot resolve before the Go Live. These bugs can be priority 2.
  • What can be deferred to future phase/projects or handled as part of the ongoing maintenance and support. These bugs can be priority 3.

Only priority 1 can stop the Go Live. Everything else can wait until after the Go Live.

5) Missing or incomplete Requirements Traceability Matrix

Make sure each requirement is covered by a test case/scenario. Indicate the reasons why a requirement is not covered by a test, so there is no confusion later down the road. Make sure this is reviewed and approved by the all stakeholders, before testing starts. This way only bugs resulting from a valid and planned test case can be worked on. With a traceability matrix in your hand, you can have a meaningful discussion if there is a debate about new test cases that have not been planned. Again, these can be discussed as Change Orders with their budget and schedule impact taken into account.

6) Testing Tools, Techniques, and Methodologies

Will you be using a testing tool such as Inflectra on this project? The organization can suddenly require that all testing get performed using a new tool/platform, after you already baselined your schedule and budget. This can add to project timeline and budget. Your project test plan should indicate what tools will be used. What if the team requires training in the new tools. This all requires you to apply change management process to ask for more budget and/or more time. But you can only apply change management, if you have a baselined test plan.

7) Applicable Standards and guidelines

What applicable standards and guidelines were in effect when the project started? The baseline budget and schedule can be impacted when the test team decides to adopt new standards and guidelines for testing. For example, you project can suddenly be required by the test manager (at their discretion) to use a specific testing methodology. Or some standards or guidelines change and now you have to make your project compliant with the new standards. You should be able to treat this as a change request and apply change management process.

As Johanna Rothman said “Shipping the product is a business decision”. I agree and it should never be a quality decision. But without a robust test plan, the test effort on your project can take on a life of its own. Avoid these mistakes and set your project up for success by investing enough time and effort to build a robust test plan.

5 Responses to Do you make these 7 test planning mistakes?
  1. Don R. James
    February 21, 2010 | 4:55 pm

    Excellent!
    The ‘V’ Model gives us the locations for testing and the locations where test plans should be developed.
    Too many times, we forget that a comprehensive review of the baselined requirements v. the test plans (and scripts, and scenarios) is a System Test.
    Think about it.
    DR J

    • samad_aidane
      February 23, 2010 | 5:12 pm

      Don, thank you so much for taking the time to read the post and comment. Unfortunately, testing gets shortchanged and is the first to be cut when deadlines and/or budgets become priority. I think as project managers we need to support our test teams and make sure that we follow the principles as those promoted by the V model. Thank you again. I will be checking you website http://www.pmotogo.com

  2. Shim Marom
    March 13, 2010 | 6:55 pm

    Hi Samad, just read your post and it is spot on. Just about the sort of issues we would face if we did not establish the right process for managing our testing effort from start to end.

    Cheers, Shim
    http://www.quantmleap.com

    • samad_aidane
      March 16, 2010 | 2:30 am

      Shim, Thank you for stopping by and commenting.

      I agree. I think testing does not get the attention it deserves. Also, I feel bad for how test teams get treated on some projects as second class citizens. I always take care of my test team because they end up saving my projects from all the blind spots that surround them.

      I see the absence of establishing the right process for managing test effort as one of the big sources for scope creep. I worked on projects where we spend all the time making sure the requirements are tights but we kept the test scope wide open. In the middle of one my projects, the organization hired a zealous QA Manager who saw themselves above any budget or schedule constrains and kept prolonging the test phase and would never agree to a Go Live date. I learned a big lesson on that project and I vowed to never get burned that way again. Now I spend a lot of time with my test team on the test plan and acceptance criteria for each test case. I want to make sure we know what will be tested, where (environment), when, by whom and for how long. I also now have 3 categories of bugs. 1-showstopper, 2-can be fixed after the Go live, and 3-can be fixed sometime in the future after the go live. this way we only focus on the showstoppers as we get closer the Go Live date and not let this date be held hostage by the customer or QA team.

      Thank you again.

  3. […] Guerrilla Project Management: Do you make these 7 test planning mistakes? Tags: agile, communication, JIRA, JIRA Agile, leadership, Lean Processes, plan time, planning, product management, productivity, progress tracking, project, project management, project reporting, resource planning, Tempo Planner […]