Working In Uncertainty

Results of a survey on 'project risk management'

Introduction

Many thanks to all who participated in this survey. Without your donations of time and thought this would not have been possible.

The results of this survey indicate an opportunity to rethink guidance on project management to make it more consistent with the basic beliefs shared by most people.

The survey results

The survey was completed by 45 people. Here are the questions that were asked, excluding the first two, which asked for name and first language, and the last, which was an opportunity to comment. The percentages are the percentages of respondents selecting each answer. (More detail about respondents and the implications of these results is given later.)

3. Do you consider yourself to be a professional in any of these? (click all that apply)

4. A discipline called ‘project risk management’ should apply to:

(If you selected ‘full lifecycle’ then please interpret references to ‘project management’ in the next questions as referring to project management in a ‘full lifecycle’ sense.)

5. The scope of ‘project risk management’ should include:

6. ‘Project risk management’ should be about:

7. Guidance on how to do ‘project risk management’ should recommend that for big, important decisions, where there is significant uncertainty:

8. It is acceptable for ‘project risk management’ guidance to recommend probability calculations:

9. It is acceptable for ‘project risk management’ guidance to recommend judgement unsupported by calculations:

10. Where several decisions within the scope of ‘project risk management’ are to be taken at the same point in a project, guidance on how to do ‘project risk management’ should advise that:

Implications

Projects involve uncertainty and humans have been developing ways to cope with uncertainty for hundreds of years. The longest established approach is that of science and mathematics, founded on probability theory and using techniques for decision making and modelling. Quantification is common, but not essential within this 'management science' tradition.

During the late 20th century an approach based on making lists of 'risks' and thinking of things to do to 'manage' those 'risks' arose and several influential guidance documents on 'risk management' have been published promoting this approach, including COSO's ERM framework and more recently an international standard, ISO 31000:2009.

Indeed, so consistently has guidance on 'risk management' pursued this risk-list approach that if you search the internet for 'risk management guidance' virtually all the material is written on this basis. It's not that guidance on dealing with uncertainty the scientific way doesn't exist; it's more that few people have thought to try to summarise the vast literature into a 30 page document, and much of it does not use the word 'risk'.

Risk-listing has also been applied to projects and an effect of this guidance has been to create a new activity on major projects where lists of 'risks' are made and 'managed'. A person employed to create and operate such a system is often called a 'project risk manager'.

Does this risk-list approach agree with the basic beliefs that people have about how to deal with uncertainty on projects and their expectations about what 'project risk management' should be like? No. The basic beliefs of most people, as revealed by this new survey, contradict the practice of 'project risk management' based on lists of 'risks'. However, they are compatible with the scientific approach to dealing with uncertainty on projects. Here are the most relevant points, one by one:

• Decision types: The overwhelming majority of respondents (91%) thought that the scope of project risk management should encompass all decisions on projects, not just those perceived as responses to risks. In the risk-list approach the 'risks' are identified first, and then actions are considered to manage those 'risks'. This approach does not naturally encompass all decisions. In contrast, the traditional management science approach has typically focused on decisions and there is no special difficulty in applying it to all decisions on projects.

• Integration: The overwhelming majority of respondents in this study (82%) preferred project risk management to be about ways to do core management activities so that alternative outcomes were kept in mind, rather than have a separate activity feeding into project management. The long-established management science approach has always been about trying to do core things, such as making decisions, in a better way, often in conditions of uncertainty. In contrast, the risk-list approach involves doing something that does not often arise naturally, to take a set of decisions that are not integrated with most decisions on projects.

• Linked decisions: The overwhelming majority of respondents (83%) preferred the idea of taking decisions together when they are being taken at the same point in a project and are causally linked, or might be. Very few respondents thought it appropriate to take the decisions separately despite causal links. The format of risk-list based risk management is geared to taking decisions separately in respect of each 'risk' on the list. Separate decisions are not obligatory but the technique strongly encourages it. In contrast, the traditional scientific approach is to take linked decisions together, usually with some kind of model to make evaluating possible outcomes easier.

Common practice on projects today is to apply 'project risk management' to projects once the big decisions have been made, restricting its role to the execution of projects that create some kind of asset (e.g. build a bridge or computer system). There is a tendency to focus on tweaks to the project plan at this stage and to make decisions on the basis of judgement, described as 'qualitative' assessment and justified as convenient and accessible.

The results of this new survey show that these tendencies also contradict the basic beliefs and preferences of most people. Generally people would prefer to expand the scope of project risk management to include earlier, bigger decisions, and to use appropriate methods that involve calculation where reasonable. Here are the most relevant results:

• Lifecycle scope: An overwhelming majority of respondents (82%) thought project risk management should apply to the whole lifecycle of an asset, not just to the construction project once it has been shaped and begun.

• Use of calculations: Almost all respondents (91%) thought that guidance on project risk management should recommend using calculations to help forecast possible outcomes for big decisions with significant uncertainty, or should recommend calculations for all situations.

• Use of probability calculations: Similarly, the vast majority of respondents (87%) thought that probability calculations should be recommended for big decisions with significant uncertainty, or for all decisions.

• Use of judgement unsupported by calculation: Most respondents (65%) thought that guidance on project risk management should recommend judgement unsupported by calculations only for small decisions, or not at all. Just 18% thought judgement unsupported by calculation should be recommended for all decisions.

The opportunity exists to write new guidance for project risk management that summarises and applies the familiar principles and techniques of 'management science' to uncertainty on projects. This would agree with the basic beliefs of most people and make project risk management suitable for application to all decisions on projects, including the biggest.

The achievements of project risk managers who already work in this way suggest that the practical implications for people on projects would include:

• more valued roles for those project risk managers who can support the traditional 'management science' methods (going beyond facilitation of risk-list workshops);

• greater involvement in management at a higher level, for those with the necessary skills;

• the opportunity for greater positive influence on projects; and

• reduced emphasis on justifying and churning through the bureaucracy of risk lists.

With this in mind, one might expect a difference in views between project risk managers (who stand to gain) and other people working on projects (who might lose out perhaps). However, in this survey the views of these two groups were essentially the same, conforming to the clear pattern found across the whole sample.

Comments by respondents

A number of respondents made comments that explained or added to answers given. The key parts of comments were as follows:

"It's good to follow 7 [calculations] by 8 [probability] which forces a rethink :-) Usually probability is just a dressed up suspicion anyway."

(I suspect this comment may have used the wrong question numbers and was actually trying to say that judgement and probability calculations should be combined.)

"I have little idea of the downside of incorporating 'unnecessary' probabilistic assessments into project risk management. On Q10 [linked decisions] my logic is that it's generally better for decisions (actions, quantifications, whatever) to be taken together even if the decisions/risks are not perceived to be causally connected (a) because decision consistency is a good idea even if we *know* things are independent (b) because it's darned hard to determine causality etc."

"There is an over emphasis in trying to quantify risk and it detracts from the fact that it is BIG risks. Low, medium, high, and very-high (probability and impact) are often much better than to haggle over 60% or 70%, $10,000 or $15,000 (add zeros as needed)."

"Qs 7 [calculations],8 [probability calculations], & 9 [pure judgement]: The effort expended in probability calculations should not be disproportionate to the level of risk. The decisions should not be taken on the basis of probability calculations alone. Q10 [linked decisions]; this was a close call. If the decisions are causally connected then they should be taken in sequence which could be one interpretation of answer 1."

"I think there should definitely be space for qualitative judgements to be used but these should be clearly defined at the project start as not all projects have clear financial or statistical risks...especially when you deal with marketing or in the not for profit sector."

"Most 'others' above are 'it depends'. I wanted more options to be able to say this."

"I'm interpreting 'project risk management' to be 'risk management around projects'. Therefore it's something that should be picked up at the very earliest of stages, including helping answer the question 'should we even be thinking about this kind of thing?' as well as 'what kind of supplier would be helpful?' I found the latter questions more difficult to answer. I wanted to say 'it depends' a lot. I can imagine a simple back of the envelope calculation or something that's even simpler than a calculation may be sufficient to give good guidance at the very earliest stages of a project. E.g. 'Should we acquire this company?' might be effectively answered by 'The following survey shows that this kind of acquisition fails in 90% of cases'. That's a simple response, it addresses the risk question, and it's not even a calculation of any kind. Come to think of it, if a 'project risk manager' thinks they should spend most of their time doing calculations then I think they'll be a a far less rounded individual than is useful."

"Qs 4 [lifecycle scope], 5 [decisions types], 6 [integration], & 10 [linked decisions]: if you separate out the risk management then you make a judgement before you have all the data. Q7 [calculations]: calculations help to concentrate the mind: even if some data be lacking. Q10 [linked decisions]: a causal connection might not be perceived until after two areas have been considered."

"The emphasis here seems to be on negative consequences. Similar reasoning and rigour should be applied to evaluating upside uncertainty as well."

"In general I would recommend quantification based on factual evidence of all aspects of a business so permission to use judgement unsupported by calculation, even where the project is small, should apply only where necessary and because there is no historical data to refer to."

"I think that Critical Chain (from the Goldratt school of management) is the most effective methodology for delivering projects on time, budget and according to specification. As it addresses practical management issues of: multi-tasking (which is inefficient), student syndrome (procrastination), and gaming (where the benefit of early finishes are not past on)."

"On 8 [probability calculations] & 9 [pure judgement], I am interpreting the answer as 'significant' or 'cost-effective' which are the only criteria that Doug Hubbard will condone as 'unmeasured.' Otherwise, it is lazy/unwise to drop rigor from decision making calculations. Q10 [linked decisions] - often we discover interaction between decisions - certainly 'causally connected' should be done at the same time (a la Impact Estimation), but temporally connected belies hidden causal connections..."

"It seems quite sensible for there to be a specific project risk management activity/stream. However I believe it should also be cultivated as a project management mind set."

Respondents

Invitations to participate in the survey were sent to the PMA Forum (an internet discussion list about performance management), the Auditnet discussion list (for auditors), and a varied selection of my contacts. 84% of respondents stated their first language as 'English' or 'British English', with the rest having a variety of other first languages. Total response rate cannot be calculated but will have been low.

Clearly this is a limitation of the survey and the selection of respondents will have affected the pattern of answers given to some extent. However, two points suggest that this was not a significant problem:

1. The overall pattern is a very strong one, with massive majorities in most cases.

2. When different sub-groups were compared there seemed to be no major differences in their views. The sub-groups compared were:

   • the roles (project management, project risk management, risk management, and audit);
   • risk role vs not a risk role;
   • project management only vs project risk management only;
   • not specialized role vs some specialised role;
   • English as first language vs not; and
   • known to agree with me on many risk related points vs known to have no particular knowledge of risk management.