Let’s say “No” to groupthink and stop quoting the Chaos Report

A project manager asked on LinkedIn:

  • We know that IT projects are almost always failures in the sense that we never hit our dates nor budget. Are there metrics that prove PMPs do a better job?

Here is my response to this question:

We need to be able to examine the underlying data and measurements methods used as the basis for any report or study on IT project failures. Without examining the data, continuing to quote such reports is simply engaging in groupthink.

I have long felt that the people behind the Chaos Report, which is frequently quoted on IT failures, have the most successful marketing machine I have see in long time. It is amazing how the report managed to become an authority on IT project management, just by continuously showing how much IT is lousy at project management.

I don’t blame them. There is apparently a lot of money to be made from this report. I believe the report costs $1000. The report’s underlying data and measurement methods are not included in this price nor are they available for sale.

The 2009 Chaos report says:

– 32% Successful (On Time, On Budget, Fully Functional)

– 44% Challenged (Late, Over Budget, And/Or Less than Promised Functionality)

– 24% Failed (Canceled or never used)

Let’s look for a moment at the definition of term “challenged” above.

The question I have is: which baseline schedule, budget, or requirements are we talking about? Is it the initial one that is determined in the business case phase, before the PM was even assigned? Or is it the baseline after the requirements are known or the solution design is approved?

These are important questions to ask, as the answer determines how we interpret the term “challenged”.

Projects are a series of negotiations. As long as changes to the original budget, timeline, and requirements have been negotiated along the way and the customer is satisfied with the outcome, then who really cares if the project completed with a different estimates from the original ones? That project should be considered successful.

A Change Management process, when used properly in a project, ensures that the customer is in charge and has final say on whether changes to the original estimates are approved or not. If a change to the original estimates is not acceptable, the customer can always refuse to approve it. The customer, at the end of the day, can always pull the plug on the project. However, if they approve the change and the project continues to completion, is that a failed project?

So before quoting statistics about IT project failure, let’s avoid groupthink and ask about the actual data behind the statistics.

8 Responses to Let’s say “No” to groupthink and stop quoting the Chaos Report
  1. PM Hut
    January 27, 2010 | 11:32 pm

    Good post.

    I have no idea btw why the CHAOS report is so expensive. I did publish an article about the CHAOS Report covering the results from 1994 to 2009. The success rate has doubled btw between 1994 and 2009, so it’s not all doom and gloom.

    I don’t think the report is far from reality, success (according to CHAOS) is about project management success which is different from project success, which means projects have to be on time, on scope, and on budget. How many projects have you see that were successful with that respect?

    • samad_aidane
      January 28, 2010 | 11:25 am

      Thank you Jorge for your comment.

      As you said in your wonderful article “My Theory on Why IT Projects Fail”, a project may be classified as failed when “one of the required features is not working according to the specifications, but all the other 20 features work well. Seriously, many times it is very badly subjective”

      I totally agree. My experience has been that for every project that is considered a success, there is at least one person in the organization who thinks it is a failure. And no amount of evidence will convince them otherwise. Think of major ERP projects that bring so much organizational change management issues.

      Going back to the Chaos Report, what if we apply the same criteria and metrics, used to measure IT “Challenged” projects, to non-IT projects and find out that in comparison IT projects are not doing so bad?

      That would be very interesting.

      But we will never know because we don’t know the metrics. And that is where I am having a real issue with the statistics we quote.

  2. Maud Schlich
    February 2, 2010 | 7:47 am

    Thanks for you post. Whenever people cite this report to motivate to change something in project management I point them to the article of Robert Glass: http://portal.acm.org/citation.cfm?id=1145287.1145301&coll=ACM&dl=ACM&idx=1145287%E2%88%82=periodical&WantType=periodical&title=Communications%20of%20the%20ACM&CFID=735452&CFTOKEN=99658899
    His thesis in the article are really worthwhile reading and still true.
    And yes, we really do need metrics to be able to improve something. But I also believe that these cannot be universal but have to be found out in every company by themselves (using GQM).

  3. […] a previous blog post titled, Let’s say “No” to groupthink and stop quoting the Chaos Report, I wrote that: “We need to be able to examine the underlying data and measurement methods […]

  4. […] [3] Let’s say “No” to groupthink and stop quoting the Chaos Report […]

  5. […] [3] Let’s say “No” to groupthink and stop quoting the Chaos Report […]

  6. […] with a different estimates from the original ones? That project should be considered successful. Let’s say “No” to groupthink and stop quoting the Chaos Report by Samad […]

Let’s say “No” to groupthink and stop quoting the Chaos Report

A project manager asked on LinkedIn:

  • We know that IT projects are almost always failures in the sense that we never hit our dates nor budget. Are there metrics that prove PMPs do a better job?

Here is my response to this question:

We need to be able to examine the underlying data and measurements methods used as the basis for any report or study on IT project failures. Without examining the data, continuing to quote such reports is simply engaging in groupthink.

I have long felt that the people behind the Chaos Report, which is frequently quoted on IT failures, have the most successful marketing machine I have see in long time. It is amazing how the report managed to become an authority on IT project management, just by continuously showing how much IT is lousy at project management.

I don’t blame them. There is apparently a lot of money to be made from this report. I believe the report costs $1000. The report’s underlying data and measurement methods are not included in this price nor are they available for sale.

The 2009 Chaos report says:

– 32% Successful (On Time, On Budget, Fully Functional)

– 44% Challenged (Late, Over Budget, And/Or Less than Promised Functionality)

– 24% Failed (Canceled or never used)

Let’s look for a moment at the definition of term “challenged” above.

The question I have is: which baseline schedule, budget, or requirements are we talking about? Is it the initial one that is determined in the business case phase, before the PM was even assigned? Or is it the baseline after the requirements are known or the solution design is approved?

These are important questions to ask, as the answer determines how we interpret the term “challenged”.

Projects are a series of negotiations. As long as changes to the original budget, timeline, and requirements have been negotiated along the way and the customer is satisfied with the outcome, then who really cares if the project completed with a different estimates from the original ones? That project should be considered successful.

A Change Management process, when used properly in a project, ensures that the customer is in charge and has final say on whether changes to the original estimates are approved or not. If a change to the original estimates is not acceptable, the customer can always refuse to approve it. The customer, at the end of the day, can always pull the plug on the project. However, if they approve the change and the project continues to completion, is that a failed project?

So before quoting statistics about IT project failure, let’s avoid groupthink and ask about the actual data behind the statistics.

8 Responses to Let’s say “No” to groupthink and stop quoting the Chaos Report
  1. PM Hut
    January 27, 2010 | 11:32 pm

    Good post.

    I have no idea btw why the CHAOS report is so expensive. I did publish an article about the CHAOS Report covering the results from 1994 to 2009. The success rate has doubled btw between 1994 and 2009, so it’s not all doom and gloom.

    I don’t think the report is far from reality, success (according to CHAOS) is about project management success which is different from project success, which means projects have to be on time, on scope, and on budget. How many projects have you see that were successful with that respect?

    • samad_aidane
      January 28, 2010 | 11:25 am

      Thank you Jorge for your comment.

      As you said in your wonderful article “My Theory on Why IT Projects Fail”, a project may be classified as failed when “one of the required features is not working according to the specifications, but all the other 20 features work well. Seriously, many times it is very badly subjective”

      I totally agree. My experience has been that for every project that is considered a success, there is at least one person in the organization who thinks it is a failure. And no amount of evidence will convince them otherwise. Think of major ERP projects that bring so much organizational change management issues.

      Going back to the Chaos Report, what if we apply the same criteria and metrics, used to measure IT “Challenged” projects, to non-IT projects and find out that in comparison IT projects are not doing so bad?

      That would be very interesting.

      But we will never know because we don’t know the metrics. And that is where I am having a real issue with the statistics we quote.

  2. Maud Schlich
    February 2, 2010 | 7:47 am

    Thanks for you post. Whenever people cite this report to motivate to change something in project management I point them to the article of Robert Glass: http://portal.acm.org/citation.cfm?id=1145287.1145301&coll=ACM&dl=ACM&idx=1145287%E2%88%82=periodical&WantType=periodical&title=Communications%20of%20the%20ACM&CFID=735452&CFTOKEN=99658899
    His thesis in the article are really worthwhile reading and still true.
    And yes, we really do need metrics to be able to improve something. But I also believe that these cannot be universal but have to be found out in every company by themselves (using GQM).

  3. […] a previous blog post titled, Let’s say “No” to groupthink and stop quoting the Chaos Report, I wrote that: “We need to be able to examine the underlying data and measurement methods […]

  4. […] [3] Let’s say “No” to groupthink and stop quoting the Chaos Report […]

  5. […] [3] Let’s say “No” to groupthink and stop quoting the Chaos Report […]

  6. […] with a different estimates from the original ones? That project should be considered successful. Let’s say “No” to groupthink and stop quoting the Chaos Report by Samad […]

Let’s say “No” to groupthink and stop quoting the Chaos Report

A project manager asked on LinkedIn:

  • We know that IT projects are almost always failures in the sense that we never hit our dates nor budget. Are there metrics that prove PMPs do a better job?

Here is my response to this question:

We need to be able to examine the underlying data and measurements methods used as the basis for any report or study on IT project failures. Without examining the data, continuing to quote such reports is simply engaging in groupthink.

I have long felt that the people behind the Chaos Report, which is frequently quoted on IT failures, have the most successful marketing machine I have see in long time. It is amazing how the report managed to become an authority on IT project management, just by continuously showing how much IT is lousy at project management.

I don’t blame them. There is apparently a lot of money to be made from this report. I believe the report costs $1000. The report’s underlying data and measurement methods are not included in this price nor are they available for sale.

The 2009 Chaos report says:

– 32% Successful (On Time, On Budget, Fully Functional)

– 44% Challenged (Late, Over Budget, And/Or Less than Promised Functionality)

– 24% Failed (Canceled or never used)

Let’s look for a moment at the definition of term “challenged” above.

The question I have is: which baseline schedule, budget, or requirements are we talking about? Is it the initial one that is determined in the business case phase, before the PM was even assigned? Or is it the baseline after the requirements are known or the solution design is approved?

These are important questions to ask, as the answer determines how we interpret the term “challenged”.

Projects are a series of negotiations. As long as changes to the original budget, timeline, and requirements have been negotiated along the way and the customer is satisfied with the outcome, then who really cares if the project completed with a different estimates from the original ones? That project should be considered successful.

A Change Management process, when used properly in a project, ensures that the customer is in charge and has final say on whether changes to the original estimates are approved or not. If a change to the original estimates is not acceptable, the customer can always refuse to approve it. The customer, at the end of the day, can always pull the plug on the project. However, if they approve the change and the project continues to completion, is that a failed project?

So before quoting statistics about IT project failure, let’s avoid groupthink and ask about the actual data behind the statistics.

8 Responses to Let’s say “No” to groupthink and stop quoting the Chaos Report
  1. PM Hut
    January 27, 2010 | 11:32 pm

    Good post.

    I have no idea btw why the CHAOS report is so expensive. I did publish an article about the CHAOS Report covering the results from 1994 to 2009. The success rate has doubled btw between 1994 and 2009, so it’s not all doom and gloom.

    I don’t think the report is far from reality, success (according to CHAOS) is about project management success which is different from project success, which means projects have to be on time, on scope, and on budget. How many projects have you see that were successful with that respect?

    • samad_aidane
      January 28, 2010 | 11:25 am

      Thank you Jorge for your comment.

      As you said in your wonderful article “My Theory on Why IT Projects Fail”, a project may be classified as failed when “one of the required features is not working according to the specifications, but all the other 20 features work well. Seriously, many times it is very badly subjective”

      I totally agree. My experience has been that for every project that is considered a success, there is at least one person in the organization who thinks it is a failure. And no amount of evidence will convince them otherwise. Think of major ERP projects that bring so much organizational change management issues.

      Going back to the Chaos Report, what if we apply the same criteria and metrics, used to measure IT “Challenged” projects, to non-IT projects and find out that in comparison IT projects are not doing so bad?

      That would be very interesting.

      But we will never know because we don’t know the metrics. And that is where I am having a real issue with the statistics we quote.

  2. Maud Schlich
    February 2, 2010 | 7:47 am

    Thanks for you post. Whenever people cite this report to motivate to change something in project management I point them to the article of Robert Glass: http://portal.acm.org/citation.cfm?id=1145287.1145301&coll=ACM&dl=ACM&idx=1145287%E2%88%82=periodical&WantType=periodical&title=Communications%20of%20the%20ACM&CFID=735452&CFTOKEN=99658899
    His thesis in the article are really worthwhile reading and still true.
    And yes, we really do need metrics to be able to improve something. But I also believe that these cannot be universal but have to be found out in every company by themselves (using GQM).

  3. […] a previous blog post titled, Let’s say “No” to groupthink and stop quoting the Chaos Report, I wrote that: “We need to be able to examine the underlying data and measurement methods […]

  4. […] [3] Let’s say “No” to groupthink and stop quoting the Chaos Report […]

  5. […] [3] Let’s say “No” to groupthink and stop quoting the Chaos Report […]

  6. […] with a different estimates from the original ones? That project should be considered successful. Let’s say “No” to groupthink and stop quoting the Chaos Report by Samad […]