Working In Uncertainty
Results of a survey on 'integrated risk management'
Many thanks to all who participated in this survey. As always, without your donations of time and thought this would not have been possible.
Respondents were asked to choose the more 'integrated' approach to 'risk management' out of a pair in 10 different scenarios, and asked if they would recommend either or both of the approaches.
The results of the survey provide rich information about what 'integrated' risk management means to people in practice. It shows that most people recognize it when they see it, and would be much more likely to recommend risk management that was integrated in that way.
None of the preferred, more integrated, more recommended approaches included listing risks, while seven of the less preferred approaches did. Clearly, listing risks is not essential for integrated risk management in the eyes of most people.
The scenarios and alternative approaches are commonly occurring in practice so even where generalizations across scenarios cannot be made the results of individual scenarios are still useful.
The survey method
The survey was completed by 111 people. Each respondent considered three simple scenarios (drawn randomly from a bank of 10) about management methods and was asked to choose for each the more 'integrated' example of risk management from two alternatives. They were also asked, for each alternative approach, whether or not they would recommend it.
Before the study began one approach from each pair was designated as the integrated approach. This was based on a variety of features but a commonality is that none of the designated integrated approaches involved making lists of risks. However, this was not revealed to respondents at any stage. Also, no explanation of the term 'integrated risk management' was provided and respondents were told that exploring the meaning of this was a goal of the survey.
In all 10 scenarios it was the approach initially designated as more integrated that was chosen by most respondents. It was also the approach more likely to be recommended in each pair.
None of the preferred, more integrated approaches involved making lists of risks.
All 10 integrated approaches would be recommended by most respondents, but only 3 out of 10 of the other approaches would be recommended by most, and then the majorities were very small.
Across all scenarios, the more integrated approach was chosen by 75.1% of respondents, the more integrated approach would be recommended by 78.7%, and the less integrated approach would be recommended by only 39.0%.
It seems that people can recognize the more integrated approach, most of the time, and they much prefer it.
Here are the summary numbers for each scenario, starting with the scenarios in which the selection majority was largest:
The tendency to select the more integrated approach as being more integrated correlates well with the difference in recommendation percentages between the integrated and other approach. The linear correlation coefficient is 0.80. This confirms that people prefer the more integrated approach as well as recognizing it.
In the survey, respondents were asked to indicate if they considered themselves to be professionals in performance management, risk analysis, risk management, or audit. Regardless of these specialities, all groups of respondents showed the same pattern of preferences across scenarios and approaches. However, the risk oriented professionals (risk analysts, risk managers, and auditors) were slightly more likely to select the integrated approach.
Respondents whose first language was English (the language of the survey) were also slightly more likely to select the integrated approach.
This was an exploratory study, designed to cover a variety of situations and characteristics of integration. Each scenario is informative.
Explaining measurement uncertainty
Here is the scenario and the two approaches offered, with results from the 32 respondents offered this scenario:
There were no comments by respondents on this scenario.
Devising policies on risk taking
Here is the scenario and the two approaches offered, with results from the 36 respondents offered this scenario:
The task is what has, confusingly, been called 'articulating a risk appetite', by some. The British Standard on risk management, BS 31100:2011, is unusual in describing the process as one of formulating and applying policies designed to control risk taking decisions. This contrasts with the approach more often suggested prior to this, which was to put limits on each of a list of risks.
Respondents emphatically preferred the decision-focused approach as more integrated and were far more likely to recommend it.
One respondent made a comment specific to this scenario. With the advantage of hindsight we know that the first part of this comment is not consistent with the strong collective preference for the decision-focused approach:
"Not sure one is any better or more integrated than the other; I would recommend both risks and likely decisions are identified with a prioritised risk tolerance developed based on both. This overcomes lots of unnecessary limits for risks that are unlikely to be encountered - or can't be controlled anyway!"
Applying policies on risk taking during planning
The results are from the 27 respondents offered this scenario:
This scenario is related to the previous one about devising policies on risk taking.
One respondent, auditor, made a comment specific to this scenario:
"While it appears A) is more integrated, I would feel more comfortable if, after A) was acted upon, B) was utilized to review/determine if decisions made in A) were appropriate. My reasoning is that the individuals involved in B) would most likely have more experience in risk analysis."
Forecasting under uncertainty
Here is the scenario and the two approaches offered, with results from the 37 respondents offered this scenario:
Two respondents commented on this scenario:
"B) is most likely, A) is a nice idea I'd go for but I would expect a board to send it back saying 'what do you really think'?"
"Human capacity to understand is also a temporal attribute; risk needs to be tightly integrated with potential financial implications."
The second part of the first comment suggests a slightly world-weary view of boards, but the results of various surveys I have carried out usually show that people generally do understand that uncertainty exists and is important. Most people, even board members, probably would not think it was a good idea to say "Yes, but what do you really think?" thus pushing the forecaster into making a guess and presenting it as an opinion.
Appraisal of a proposed project
The results are from the 35 respondents offered this scenario:
One respondent commented:
"A) is most sensible. B) would be good IF the board had the experience to assess the risks or to know how likely they were or which had already happened to other people."
Developing a plan
The results are from the 30 respondents offered this scenario:
One respondent commented:
"Case does not indicate 'time to market' constraints; nor is there an option to address health issues of sugar in drinks, which is not a typical risk concern."
Approaching a difficult decision
The results are from the 25 respondents offered this scenario:
Two respondents commented:
"I would have thought a combination of A) and B)."
"What a wonderful world if our minister had the talent (and inclination) to execute option A)."
Policy on project acceptance
The results are from the 39 respondents offered this scenario:
One respondent commented:
"My response ... assumes the risk score is wholly negative since the postulation is risk score versus benefits. IF, however, the 'risk' score considers both +ve and -ve (as in 'risk' being both threat and opportunity) then my answer would be B)."
This comment brings to the light the fact that neither of the two approaches offered in the survey would be logical for all ways of creating a risk score from both positive and negative outcomes. The logical approach would be to vary the level of risk for higher benefits, not necessarily raise it.
The results are from the 44 respondents offered this scenario:
One respondent commented:
"Although I believe A) is the most integrated - and have recommended that approach - I can also see elements of B) that I would also recommend. My thoughts are that after identifying scenarios any strategy must ultimately take the direction of the 'best estimate forecast' albeit whilst always recognising the limitations of that forecast and having alternative strategies for emerging situations. In addition I also recognise different hierarchies of risk: and whilst scenario planning is an integrated way of mitigating strategic risk, in my mind it will also be necessary to then consider tactical and operational risks for the implementation of that strategy."
Searching for opportunities
The results are from the 28 respondents offered this scenario:
One respondent commented:
"B) random sample is fine when you've already decided to audit, in real life tax inspectors would look for bad smells that guide A)"
Invitations to participate in the survey were sent to the PMA Forum (an internet discussion list about performance management), the Auditnet discussion list (for auditors), the RISKANAL discussion list (for risk analysts), and a varied selection of my contacts. 75% of respondents stated their first language as 'English' or 'British English', with the rest having a variety of other first languages. Total response rate cannot be calculated but will have been low.
Clearly this is a limitation of the survey and the selection of respondents will have affected the pattern of answers given to some extent. However, two points suggest that this was not a significant problem:
The study did not identify all the underlying principles that determine whether an approach is 'integrated' or not, or that determine whether people like it.
However, it is clear from these results that approaches do not have to involve making lists of risks to be regarded as 'integrated risk management.' None of the 10 preferred approaches involved making lists of risks, while 7 of the less preferred approaches did.
The results are also useful because the scenarios are common ones and the competing approaches are also common. Some respondents, in their comments, indicated that they liked the integrated approach but thought others would choose the less integrated approach. Perhaps this is because they see the less integrated approaches as more common in practice or more commonly suggested. The survey results show that in fact most people prefer the integrated approaches, even though some may be less common at present.
Getting more integrated approaches adopted may be easier if they are suggested and then the views of all concerned are polled, rather than relying on the first reactions of a few to decide what is to be done.
Company: The Ridgeway Expertise Company Ltd, registered in England, no. 04931400.
Registered office: 29 Ridgeway, KT19 8LD, United Kingdom.
Words © 2011 Matthew Leitch