Working In Uncertainty
Integration in future risk management guidance and standards: results from a Risk Improvement Group survey
The cross-governmental Risk Improvement Group meeting (RIG) on 12th March 2015 offered a great opportunity to get the views of risk professionals on key integration issues for future guidance and standards on risk management. Many thanks to everyone at the meeting who kindly participated in this survey. Without your generous donations of time and thought this would have been impossible. Thanks also to Trevor Marchant, who invited me to present a session.
The RIG is open to all government departments and agencies and currently has over a hundred members, with meetings regularly attracting more than 60 people. It was set up to share best practice across government, for networking, and to influence developments of common interest.
The next generation of generic risk management standards and guides is likely to tackle the challenge of making risk management an integral part of core management activities. So far this has been an aspiration but practical methods have not been described. Instead, well known guides to risk management (e.g. The Orange Book) typically describe just one method: making lists of risk events, derived from objectives or from organizational activities or assets, and then deciding how to respond to those risks. This method does not integrate into core management activities easily and previous survey evidence shows there are well-known alternative techniques that are seen as more integrated and better overall.
The survey respondents were largely UK public sector risk managers operating under the guidance of The Orange Book, which has been in place in its current edition since 2004 and has been the main technical guide and constraint for them.
Despite this, the survey results showed a positive view of techniques that would integrate risk management into core management activities (including techniques with no risk register), tempered by understandable caution over raising potentially unreasonable expectations of risk managers or reducing the perceived importance of continuing with what has already been put in place.
Most thought it would be good to say in standards and guidance that using ways to manage risk effectively in core management activities 'is desirable.' However, a substantial minority thought it would be bad to say that 'not doing so means you are not managing risk adequately.' Analysis of comments revealed an underlying concern that people in organizations might not respond to encouragement to manage risk in core management activities. This failure would reflect badly on risk managers if successful integration was seen as essential for effective risk management.
Also, there were concerns over stances that might reduce the perceived importance of risk registers. A substantial minority thought it would be bad to say that 'risk registers, populated after decisions have been taken, are helpful only in limited situations and should not be relied on as a main technique.' Also, most respondents thought it would be bad to say that 'risk registers, populated after decisions (e.g. business planning) have been taken, are not worthwhile if you manage risk within core management activities, and can be dropped.'
In comments, several respondents defended the use of risk registers, but most did not distinguish between different uses of them. Distinctions were emphasized later in the survey, revealing that the most common use of risk registers has been to track progress against high level 'risks' that are more like objectives than risk events.
There was a tendency for people with influence at a higher organizational level to be more positive about the effects of stances in favour of integrated techniques, and also a tendency for people with wide experience of techniques to be more willing to promote more techniques in future. These tendencies are consistent with the cautions expressed in comments.
A set of guidelines for authors of future guidance and standards on risk management was generated on the basis of these survey findings.
The key point for future risk management standards and guidance is that this group, and probably other groups like it, would welcome a broader approach that gave practical advice on a range of techniques that integrate risk management into core activities. There is no need to fear an adverse reaction to going beyond the single technique of risk event listing, provided the advice is sensitive to the concerns identified by this research.
Integration of risk management into core management activities (e.g. planning, design, decision-making, evaluation) has long been an aspiration for risk management, but the best known guidance and standards (e.g. ISO 31000:2009, BS 31100:2011, COSO's ERM Framework, The Orange Book) do not contain practical guidance on how to do it. Instead, the focus of practical guidance has been on listing risk events in a risk register and then deciding what to do about them. This procedure focuses on decisions about actions seen as just responses to risks and is best suited to reviews after the main decisions have been taken. This keeps risk management separate from core management activities.
However, my surveys over the last few years have shown that most people with an interest in management see risk management as being more than listing risk events in a register (2014), see other techniques as being more integrated (2011b), and are interested in using those techniques (2011b, 2012). The techniques involved are auditable and well established. If risk management specialists were to promote other methods they might well find that resistance to using them is lower than it is with risk event listing, and that people more often follow up their words of agreement with positive action.
What could future guidance and standards say about integration? Since risk management specialists are the main users of standards and guides, their reactions to possible new advice are crucial. This new survey was designed to answer some of the key questions about how open to new techniques risk managers are and how they would react to stances that guidance might take on integration.
Survey procedure and context
The survey was conducted during a presentation to the Risk Improvement Group, held on 12 March 2015 in London. The morning sessions included presentations on using risk registers in various ways, with some examples of high level 'risks' as well as uncertainties, along with other examples more similar to The Orange Book's risk event listing method.
After a break for lunch Matthew Leitch presented key results from three earlier surveys on integration (Leitch 2013, Leitch 2011a, and Leitch 2011b) which gave specific examples of techniques that most people think more integrated than using a risk event list, and many more would recommend.
Immediately after that Matthew led the group through the survey, question by question, providing additional explanation of the questions and providing more detail when asked. Everyone in the audience was asked to participate and it seems that all or very nearly all did so. Two respondents indicated during the session that they were confused and repeated this message on their questionnaires, but no other respondents showed obvious signs of significant confusion.
Questionnaires were collected before the open discussion session to ensure that survey responses were not influenced by respondents knowing the opinions of others.
The respondents were attendees at the Risk Improvement Group meeting held in London on 12th March 2015, and all or nearly all the attendees present for the afternoon session completed the questionnaire. Two respondents said they were confused by one or more questions and remained confused despite verbal explanations. Their responses were not used. This left a total of 36 responses.
The first question on the survey asked for information about the current role of the respondent. The results were as follows:
86% of respondents were risk management specialists in some kind of risk management support role. (A few were risk managers for more than one organization.)
Questions about alternative stances on integration in future guidance
In this section the text in italics is the text used in the survey itself. Other text discusses the results and their implications. After the initial question on roles the survey continued as follows:
Since most people seem to have a preference – in principle at least – for managing risk in a way that is just part of management and not a separate exercise, future standards should probably try to move further this way. But how? There are various ways. It is important to understand how these could affect people using the guidance, both positively and negatively.
For example, some risk management specialists might be helped by guidance that stressed the value of integration because it would help them argue for good changes to management methods and make their job more valuable. But, for others, the same guidance might cause problems because they cannot influence how core management activities are done and their performance would be regarded as failing, relative to the guidance. These are just hypothetical illustrations.
Q2: Think about your personal position only (ignoring how others might be affected), and say how you personally would be affected by authoritative risk management guidance taking the following positions:
Here are the options given and the number and percentage of respondents choosing each answer:
The pattern of results shows a positive view of techniques that would integrate risk management into core management activities (even if they do not involve a risk register), but also some caution over making successful integration a condition for successful risk management, and concern at statements that might make risk registers optional.
A large majority thought that more practical descriptions of ways to manage risk effectively in core management activities would be good for them personally. Most also thought it would be good for them if the guidance said that doing so was desirable. This positive response was despite the fact that such techniques would often not feature a risk register.
However, about a third of respondents (a substantial minority) thought it would be bad for them if the guidance said risk management would be inadequate if techniques to manage risk effectively in core management activities were not used. A number of reasons were given for this in response to question 3 and these are discussed later.
Also, most respondents thought it would be bad for them if the guidance said risk registers could be dropped if risk was being managed effectively within core management activities. Again, a number of reasons were given for this.
There was also no clear favourite view on saying that risk registers should not be relied on as a main technique, though more said this would be good for them than that it would be bad.
Q4: What about the effect for your organization, if this advice was taken?
Again, the pattern of results shows a positive view of techniques that would integrate risk management into core management activities (even if risk registers are not involved in all of them), but caution over making successful integration essential, and worry about making risk registers optional.
The pattern of responses was very similar to that for question 2 on the personal effect of the different stances. The differences were that more respondents had no idea what the effect would be for their organization and there was a slight tendency to think the effect for the organization would be better than the effect for the individual respondent. One way to see this shift towards 'good' effect is to look at the number of 'good' responses less the number of 'bad' responses for each item:
Questions about reasons
Questions 3 and 5 asked for reasons behind answers to questions 2 and 4, which were about alternative stances that future guidance might take towards integration. The pattern of results for questions 2 and 4 showed a positive view of techniques that would integrate risk management into core management activities, but caution over making successful integration a condition for successful risk management, and concern at making risk registers appear optional.
The comments in response to questions 3 and 5 helps to explain this pattern.
Respondents commented that having a wider range of techniques to choose from would be helpful, especially if those techniques left people free to think imaginatively. Comments also noted that techniques that did not create new responsibilities for management would be welcome and that integration was the aspiration, and the best way to avoid poor outcomes. It was also pointed out that integrated techniques were already in place in some cases.
However, several respondents were concerned that their management teams would not be willing or perhaps would not be capable of making changes to the way they managed, and that they would take advantage of a relaxation over risk registers to just stop doing any effective risk management at all. This was the main stated explanation for the caution over making successful integration a condition of successful risk management, and for the concern over suggesting that risk registers are optional.
Other respondents pointed out that better decision-making was not the only goal for risk management, and that requirements for evidence, audit, escalation, and monitoring were also important, so any approach would need to support those too. Some respondents asserted that risk registers would remain essential, but none of these distinguished between risk registers used to list risk events and those with higher level 'risks' that are really more like objectives.
(Remember that, at this point in the survey process, respondents had only a rough idea of what alternative techniques for managing risk might be. More clarity was provided later in the questionnaire by questions 6 and 7, at which point it would have been clearer that alternative techniques also provide documentation and can be used for monitoring.)
Questions about techniques
Questions 6 and 7 looked at experience with other techniques and willingness to promote them in various ways.
Q6: Which techniques have you personally been involved with, and to what extent?
(It was explained verbally that 'Valued' meant 'privately given credit for', while 'Praised when done' meant giving credit that others were aware of. Also, this question referred to all previous roles, not just the respondent's current role.)
It should be expected with this set of respondents that the only technique described in The Orange Book, which is risk event registers not linked to a decision, would be the technique that by far the majority of respondents would have promoted. However, it was not.
Using 'risk' related objectives or high level 'risks' (effectively objectives) was something more respondents had acted positively towards in the past, and so was scenario planning. There was also plenty of evidence of positive action on the other techniques on the list.
Q7: To what extent would you personally be willing to pursue these techniques, if they were recommended by authoritative guidance?
(It was verbally explained, in response to a question from a respondent, that if one would be willing to promote a technique but there was no opportunity to do so in one's current role, then it was better to write 'NA' next to the answer and not tick it. Several respondents did this, especially for war gaming. It was also established that this question referred to the respondent's current role, not to any future role.)
Willingness to act positively towards the various techniques was again not focused on the technique described in The Orange Book. There was more willingness to act positively towards scenario planning, what-if questions, incremental delivery, and 'risk' related objectives/high level risks.
Experience and willingness to promote other techniques was widely distributed among the respondents. The number of techniques other than risk event registers that each respondent had promoted, or was willing to promote, in at least one way was distributed like this:
Summarized another way, 94% of respondents had experience of promoting techniques other than risk event registers, and 92% had experience of promoting techniques that did not involve any kind of risk register. Furthermore, 92% were willing to promote techniques other than risk event registers, and the same 92% were willing to promote techniques that did not involve risk registers at all.
There may be a relationship between experience of techniques and views on the value of guidance containing practical descriptions of ways to manage risk effectively in core management activities. The four respondents who thought such guidance would be bad for them or their organization had unusually low experience of other techniques, and low willingness to promote them in future.
Factors influencing answers
The size of the sample for this survey was adequate for the emphatic percentages found but not large enough to relate the various answers to each other because the correlations are low. However, some suggestive patterns are present.
The independent variables studied were:
Perhaps surprisingly the links between years of experience, technical experience, and level of influence were weak, making it a little easier to distinguish the effects of each independent variable.
The dependent variables studied were:
The strongest links between the independent variables and the dependent variables seem to be (a) the tendency for respondents with influence at a higher level to think stronger stances were good and not bad, and (b) the tendency for people with a lot of varied technical experience to be willing to promote alternative techniques in future. The strength of the links was judged simply by the appearance of a steady trend with high variation.
Sampling and generalization
The sample size is adequate for the very clear cut overall results, but not for reliably exploring finer points such as the weaker connections between answers. So, the main messages of this data can be generalized, but to what population?
Very few if any of the attendees present at the RIG meeting session failed to return completed survey questionnaires and only two were unusable due to confusion over some questions. So, these results are probably a good representation of the RIG membership, and of public sector risk management specialists interested enough to attend a group like the RIG. These are also the sort of people most likely to comment on new guidance and standards, and so their views are a good indication of the kind of feedback a new standard or guide would provoke within the UK public sector.
However, the RIG members are not necessarily representative of all risk management specialists in the public sector or beyond. They may be people who are particularly interested in complying with requirements (The Orange Book) and in following their peers, or they may be particularly interested in being progressive and trying new ideas.
It is likely that a different selection of respondents would have produced different answers to some extent. The previous section of this report showed that there appeared to be a tendency for the least progressive answers to be from people with low years of experience, influence at a low level, and low previous experience of techniques. However, the four respondents with all these disadvantages nevertheless answered questions 2 and 4 in a similar way to other respondents and, on the crucial question about whether more techniques should be explained, most thought that would be good for them and half thought that would be good for their organizations. In summary, even the most disadvantaged respondents were positive, but not as strongly as other respondents.
Implications for future guidance and standards
The following guidelines for authors of future guidance and standards on risk management take into consideration the answers to the questions on stances and techniques, and the comments made.
Guidelines based on the overall pattern of answers
1. Respond to the perception of benefits from more practical guidance on a range of relevant integrated techniques with the following steps:
2. Allay concerns about raising potentially unreasonable expectations of specialist risk managers with the following steps:
3. Allay concerns about reducing the perceived importance of risk registers with the following steps:
Guidelines based on sensible suggestions by at least one respondent
4. Give evidence-based advice with the following steps:
(One respondent pointed out the importance of evidence of effectiveness. This would be a new approach for writers of standards and guidance on risk management, and is long overdue.)
5. Avoid being overly prescriptive about technical detail with the following steps:
(A respondent emphasized the value of allowing variation and adaptation to particular circumstances. Another sought to free the imaginations of managers.)
6. Explain the significance of risk/uncertainty within escalation rules and procedures.
(One respondent stressed the importance of escalation, and this remains important even if it is not 'risks' that are being escalated.)
BS 31100:2011. Risk management. Code of practice and guidance for the implementation of BS ISO 31000.
Committee of Sponsoring Organizations of the Treadway Commission (2004). Enterprise Risk Management - Integrated Framework. Executive Summary available online at: www.coso.org/documents/COSO_ERM_ExecutiveSummary.pdf
Department for Transport (2014). DfT analytical assurance framework: strength in numbers. Available online at: www.gov.uk/government/publications/dft-analytical-assurance-framework-strength-in-numbers.
HM Treasury (2011). The Magenta Book: Guidance for evaluation. Available online at: www.gov.uk/government/uploads/system/uploads/attachment_data/file/220542/
HM Treasury (2013). Review of quality assurance of government models. (The Macpherson report). Available online at: www.gov.uk/government/publications/review-of-quality-assurance-of-government-models.
HM Treasury (2014). The Green Book: appraisal and evaluation in central government. Available online at: www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in-central-governent.
HM Treasury (2015). The Aqua Book: guidance on producing quality analysis for government. Available online at: www.gov.uk/government/publications/the-aqua-book-guidance-on-producing-quality-analysis-for-government.
ISO 31000:2009. Risk management: Principles and guidelines.
Leitch, M. (2011a). Results of a survey on 'project risk management'. Available online at: http://www.workinginuncertainty.co.uk/study_pram_report.shtml
Leitch, M. (2011b). Results of a survey on 'integrated risk management'. Available online at: http://www.workinginuncertainty.co.uk/study_integ_report.shtml
Leitch, M. (2012). The Reality of Risk: culture, behaviour, and the role of accountants. Available online at: http://www.accaglobal.com/content/dam/acca/global/PDF-discover/2012/tech-afb-rrm.pdf
Leitch, M. (2013). Results of a survey on ISO 31000:2009 and future editions. Available online at: http://www.workinginuncertainty.co.uk/study_iso_report.shtml
Leitch, M. (2014). Results of a survey on 'risk management'. Available online at: http://www.workinginuncertainty.co.uk/study_rmgmt_report.shtml.
Appendix: The techniques in more detail
The techniques in questions 6 and 7 were described verbally to some extent while the survey was presented, so respondents had more information than was given on the face of the survey. This section presents more information still, so that readers can better understand the thinking behind the survey.
1. Scenario planning
A wide range of scenario planning methods has been developed. The common features include:
Published scenario planning cases usually look at big, long term events but the method is, arguably, more useful for events that unfold more quickly.
The method is widely seen as a good example of integrated risk management and, when done properly, generates auditable documentation of the thinking done. It can be used to promote monitoring in three ways:
2. War gaming
Where there is an enemy or competitor to consider, planning methods need to respond to this challenge. War gaming involves role playing to understand better how the enemy might be thinking. There are various methods for doing this.
Done properly, war gaming generates auditable documentation. Also, monitoring can be accomplished by repeating the exercise and by tracking strategies developed through war gaming in the usual way.
3. Decision-support model with probabilities
This is the main approach used today for major decisions and has recently gained focus in UK government thanks to the Macpherson report and subsequent developments (e.g. the AQuA Book, DfT's Analytical Assurance Framework). Considering uncertainty is a key part of this. Decision-support models with probabilities have been the main focus of decision analysis for several decades and are widely seen as a good example of integrated risk management.
The approach is impossible to do without creating auditable documentary evidence of the thinking involved. Typically, such models are used repeatedly over time to revisit decisions. Forecast and actual results can be recorded and compared.
4. Decision-support model used to answer 'what if' questions
Even when a decision-support model (e.g. a forecasting spreadsheet) does not represent uncertainty explicitly using probabilities, it can usually be run repeatedly with different input values to understand the effect. This is very common. Done systematically this can be a form of sensitivity analysis. Done on a massive scale automatically it becomes Monte Carlo simulation.
Using an explicit model to support decisions provides auditable documentation but particular 'what if' runs are not necessarily recorded. This is a refinement that might be introduced to provide even more auditable documentation. Models are often used to revisit decisions repeatedly over time, which is monitoring.
5. Incremental delivery/agile
Delivering a challenging project in stages has such a powerful effect on risk and results that it should be done whenever possible. The opening of Heathrow's Terminal 5 is a good example of what can happen if you don't. The Magenta Book contains good advice on phased introduction of policies.
The documentary evidence of having chosen to deliver incrementally is in the project itself of course. However, if a decision is taken not to deliver incrementally then some good reasons should have been considered properly. Auditable documentation of these considerations is not always created, though it would not be hard to devise and implement a policy requiring it.
Once incremental delivery begins, monitoring is relatively easy and reliable. Instead of tracking 'earned value', project managers can track value delivered.
6. Establishing high level 'risks' or objectives that cover 'risk' concerns
This is a collection of three very similar approaches that are probably widely used, but without being noticed. They are particularly important for risk management.
These techniques generate the same auditable documentation as any other system of objectives, and support monitoring in the same way too.
7. Risk register with 'risks' for each of the alternatives in a decision
This technique was included in the survey because it was known to have been used and yet is not the approach described in The Orange Book. There is debate about whether ISO 31000:2009 is consistent with this approach but it certainly does not describe it clearly if it does so at all.
In the survey, respondents were instructed to interpret this strictly so that a risk register and risk register terminology needed to be used. However, similar techniques without the risk register format or terminology are likely to be in use too.
Documentary evidence is generated and the evaluation can be repeated at intervals as part of monitoring.
8. Risk register with risk events not linked to a decision
This technique is the one described in The Orange Book and (in a more abstract way) in ISO 31000:2009. The risk event list is created by considering objectives or activities, not by considering a decision in which some factors are uncertain.
This technique is suitable only for decisions on actions seen as wholly responses to the risks and so provides documentary evidence of considering risk only for such decisions. In theory the risk event lists can be used for monitoring, but the lists are usually long and the rating methods are unreliable, with aggregation of risk levels being particularly difficult.
Company: The Ridgeway Expertise Company Ltd, registered in England, no. 04931400.
Registered office: 29 Ridgeway, KT19 8LD, United Kingdom.
Words © 2015 Matthew Leitch