If you think about it for a minute you will see that your question doesn’t really make much sense at all.

Just because the Chaos report is packaged in a nice and easily digestible way does not absolve it of the need to also be accurate. Are you suggesting that incorrect evidence is better than no evidence at all?

Cheers, Shim

Thank you for the comments.

So sorry for the late reply. I just saw your comments this afternoon.

To answer your question, there has not been any response from Chaos Report authors.

Thanks for the link to Fran’s article. Loved it and left a comment. It is is great to see that the interview contributed to the conversation about the Chaos Report.

See a continuation of this discussion with comments at http://blogs.innerworkings.com/fmckeagney/2010/10/01/meaningless-the-rise-fall-of-the-chaos-report-figures/.

Have we heard any response from the Chaos report authors as yet?

Thank you for taking the time to comment.

Unfortunately I am not aware of any other reliable report at this point. I don’t think PMI uses the figures from the Chaos Report anymore but I am not 100% sure. I would love to know if there are other reliable studies.

Thank you again.

Where are reliable datas??? As i said, the PMI was using these datas a few years ago as a reference for project succes rate. Is it still the point?
regards
Tannguy

Thank you so much for your comment. I am glad you found this information helpful. I have been concerned for a long time about the negative impact the Chaos Report has on the perception of IT projects. Especially the perception of the business sponsors. I was thrilled when Mr. Chris Verhoef agreed to do the interview. I am also grateful to Mr. Glen Alleman for mentioning this post on his great blog: http://herdingcats.typepad.com.

Thank you again Mike.

PS: Thanks too, to Glen Alleman at http://herdingcats.typepad.com/ for signposting me here.

Thank you so much for your comment. I found http://www.ambysoft.com/surveys to be a great resource when I was researching scaling agile. Your surveys were extremely helpful to me about what specific agile practices people are using on their project.

I plan to use your site more and recommend it as a resource as I continue the blog post series I started on tailoring agile for large system integration projects.

Thank you again Scott.

Cheers, Shim Marom
http://www.quantmleap.com

Project failure does not only occur when the solution doesn’t work. Yes, missing performance objectives is something we see with IT projects, but I have found these types of failures to be in the minority, and seldom does the technology fail outright.

Project failure occurs when:
- Organizations invest in the wrong projects (not those most essential to meeting Enterprise Strategy)
- The speed of the project does not meet the needs of the business (projects are approved predicated on when value is expected to be realized)
- The value of the project is not commensurate to the investment in the project (and I find most organizations are not even able to accurately determine investment value – and sometimes, the “actual” cost of the project, which includes indirect, business process change, and full-lifecycle costs)

I highly suggest you start following @mkrigsman. Michael Krigsman writes a post entirely dedicated to project failures. For us project success advocates, his blog is sort of our “Project Failure Maypole.”

My last comment is in regard to you assertion, “It is not possible to consistently achieve 100% success rate.” Here is where I completely agree with you. In fact, I want to always see a certain level of failure because it means we’re pushing the envelope and not playing it too safe. So I hope you weren’t making the assertion based on the assumption that I was advocating something that is not only unrealistic, but contrary to businesses accepting a healthy amount of risk in their investments.

For those of us who find IT Project failure rates to be unbearable, I doubt there is anyone who has ever suggested the target is a 100% success rate. What I do want is for Enterprises (not just IT) to do what they say they are going to do, and deliver what they say they are going to deliver. And once again, I find few that are able when it comes to their IT projects.

Steve Romero, IT Governance Evangelist

The current level of project failures represents an acceptable level based on the the fact that:
a. It is not possible to consistently achieve 100% success rate (that’s almost a physical law); and
b. Where 100% success rate is required (for instance in space exploration missions), exuberant amounts of money are required to make it happen. So, in principle, the current level (more or less) of success rate represents a socially acceptable level based on the opportunity cost involved with changing it either way (up or down).

My conclusion: Let’s stop talking about a problem that doesn’t really exists.

Cheers, Shim.

Totally agree with you. The end should not justify the means.

My point, which I didn’t do a good job articulating in my previous comment, is that the report figures are astounding and shocking. I can see how, when used as a persuasion tool, their shock and awe impact can be very effective with executives. I am not saying that this is justified. I am just saying that as a persuasion tool, fear is much more effective than reward. And this is what I think makes the Chaos Report very widely quoted.

With research, such as the one that Chris and his team produced, we can begin to show that the Chaos Report is not the right persuasion tool. We will then have the opportunity to think of more effective ways to help our organizations increase their awareness about how to reduce project failure and increase success.

Steve

And thanks again for a great post.

Steve

First, I cite numerous studies in my presentations. Upon citing the Standish Report, I tell audiences I believe the report is flawed and I explain the basis of my views. The reasons I even bother to take audiences through the exercise of citing flawed research are as follows:
- Many people are aware of the Standish Group
- The study shows a high degree of failure which ignites discussion
- Their flawed (in my view) approach to project failure characterization underscores the need for enterprises to define project failure in terms that result in their acquiring the data essential to making good project investment decisions

I came to much higher level conclusions than those cited in this post. I did not have near the insights I have now, which is why I have to provide even more explanation when citing the Standish Chaos report.

Steve Romero, IT Governance Evangelist

I seriously fail to understand how using a “flawed and misleading” (to quote Steve’s comment) can be used as the basis for “findings can be a powerful tool to motivate executives to establish proper governance structure in their organizations” (to quote your comment).

I don’t think the end justifies the means, which means that just to convince organizations to adopt proper project management disciplines should not be used as a justification for using the wrong data. If this was the case we could all write our own imaginary reports and present them to executives, every-time we wanted to convince them of that point or another. Clearly it is wrong and as professionals we should shy away from using shoddy data to strengthen our arguments (irrespective of how valid our arguments are).

I love that and your passion shows in your wonderful posts on the IT Governance Evangelist blog.

Steve, first of all, I want to thank you for your candid comments about your experience with the Chaos Report.

It is very helpful to me to understand how others have used the Chaos Report figures. I can appreciate how its findings can be a powerful tool to motivate executives to establish proper governance structure in their organizations.

This conversation we are having was exactly my hope from interviewing Chris and getting the word out about his team’s findings. It is through these conversations that we gain a better understanding of how we can help our organizations better understand failure and increase project success.

It is funny that I got introduced to your blog just the day before by a tweet from Rick Morris. I enjoyed your recent post: “IT Governance Stops the Hate between the Business and IT”.

Thank you again Steve.

Do I really need to elaborate on the ethical and professional issues associated with such an admission?

The Chaos report is incorrect – full stop! To make use of such report just to advance an agenda, whatever the agenda is, is incorrect, to say the least.

Shim Marom
http://www.quantmleap.com

Thank you so much my friend for reading and commenting,

You are so right. You really can’t dumb down this type of information and expect to deliver the same points. A lot of work went into the research that Chris and his team did and I am just grateful to Chris for taking the time, from his busy schedule, to answer my questions and make this information available to us.

I have felt the same way you did about the 30% success rate that is frequently quoted. Like I mentioned in my follow-up to Chim’s comments, I have never worked at an organization (or heard of one) that would tolerate this low success rate. If the low success rate was true, I would have left IT a long time ago.

I think studies such as the Chaos Report are very powerful as they shape the perception that the business stakeholders develop of IT Projects and IT project managers. The perception is often negative and can range from skepticism to outright hostility. In some organizations, it takes a lot of hard work and many years of solid track record before the perception is corrected. My hope, from getting the word out about this research, is to equip project managers with information they can use to educate their stakeholders about IT Failure myths. Ultimately, I want IT project managers to expect that they will success, to feel confident that failure is not the norm, and to believe in themselves that they have the capacity to deliver successful projects. Dammit!!! we deserve success!!!

Cheers my friend.

I don’t lament the non-availability of Standish data simply because I have become accustomed to their practice of not sharing it. This terrible research practice matters little because I use their flawed results to convince organizations to aggressively address project failure rates.

Every study I have seen in the past two decades has shown at least half of all IT projects fail. And yes, Enterprises have managed to take us into the information age despite these high failure rates. This is simply explained when project failure is defined as a project that does not meet its intended and stated commitments. Using this definition, a project failure does not necessarily mean the effort should never have been sanctioned. It simply means the mechanisms used to make project decisions (from ideation to completion) did not meet their stated objectives. These mechanisms constitute the project’s failings, even in those instances where the technology indeed brings us into the information age. Side note: I contend the majority of project failures are caused by poor Project and Portfolio Management (PPM) practices, as opposed to poor Project Management practices.

After almost 30 years working in almost every area of IT, I became an IT Governance Evangelist. I have been traveling the world touting IT Governance and its essential processes and mechanisms for over 3 years now. I have spoken to thousands of people in over 200 forums in which I have presented (100 of these to individual companies). I rarely encounter Enterprises with established specific definitions of project success and failure and the associated ability to make decisions based on applying those definitions. Most people I meet don’t even have the word “failure” in their corporate vernacular (given its incredibly negative connotation and the pervasive human aversion to the word).

Vrije Universitei does an incredible job noting the many variables and moving parts associated with understanding and determining project success and failure. I just don’t believe I could use their impressive research and analysis to influence the necessary changes to improve project success in Enterprises today. I would bet most of my audiences would be asleep before I could get halfway through just explaining your blog post, let alone taking a deep dive into the actual research. I use the Standish Report and its suspect failure rates because it is an effective means to motivate organizations to take action. I use the studies from the Standish Group, Garter, Forrester and MIT CISR (my favorite) to urge folks to define project success and failure in their Enterprises and to use those definitions to understand and subsequently improve their success rates.

Your great post and the incredible work done by Vrije Universitei will now put an end to my practice of citing the Standish Chaos report, even if it was under the veil of good intentions. Better still, I will still cite their results, explain my misgivings regarding their approach, and then cite your blog post and the findings of Vrije Universitei. Frankly, I care little about the nuances and resulting disparities between one research approach and another. What I do care about is inciting Enterprises to take action regarding their ability to understand project failure and increase project success.

Let me close by noting the serendipitous nature of the timing of your post. I received a call from Standish Group Customer Service today asking about my use of the Standish website. I told them I only used it to obtain summaries of their research (because it is free) to quote in my presentations. I said I had little insight or interest in their paid services. I agreed to schedule some time with them next week, to give them the opportunity to talk to me about their services. After reading your post, I am sure it will be a very interesting conversation.

Steve Romero, IT Governance Evangelist
http://community.ca.com/blogs/theitgovernanceevangelist/

Without examining data objectively, you get nothing more than subjective conjecture. I just kept asking myself, 30% success rate? I thought only the weatherman could have a 30% success rate and keep his job. If we, as project managers, were only successfully delivering 30% of the time, we’d be out on our rears.

Regards,
Derek
http://twitter.com/derekhuether

Thank you so much for taking the time to read and comment.

I appreciate the work that Chris and his team have done and I wanted to make sure that I do everything I can so others can read it and benefit from it. I think it is valuable work and it need to be read by all those who are interested in IT and software project failure.

On a personal note, I appreciate this research because I am passionate about the topic of IT Failure.

I lived all my professional life (during the last 15 years) in IT departments and IT consulting companies. It has been painful to me, since the first chaos report came out, to experience firsthand the negative perception of IT in the business community. The perception is that we in IT are unable to increase the success rate of projects, despite all the progress and great work that has been done over the last 15 years.

The negative impact of the Chaos Report findings on the perception of IT by the business community is real. It undermines the trust in IT project management and self confidence of IT project managers.

I never believed the Chaos Report because I knew that no organization I worked with in the past can tolerate 30% success rate of its projects. From my own personal experience, I knew that this number is incorrect. Only thru research, such as this one from Chris and his team, can we begin to correct this perception with real data.

By the way, working with Chris on this interview was a wonderful experience for me and I am grateful for his time, efforts, and patience.

Cheers, Shim Marom
http://www.quantmleap.com