Working In Uncertainty

A comparative overview of risk management and internal control guidance

by Matthew Leitch, 26 May 2010.

The famous guides to risk control – like COSO's ERM framework, ISO's 31000 standard, and the UK government's Orange Book – have had a huge influence on theorising about risk control and also driven practical choices by organizations around the world.

Unfortunately, fame is no guarantee of quality or suitability. The table below is a simple guide to the main contenders in alphabetical order of the issuing organization. As you can see, the highest overall ratings do not go to the most famous documents.

In future I hope to add more detailed reviews, plus advice on implementing each of the guides.

Ratings

The meaning of each of the criteria is explained below the table.

Please notify me if you know there is a more recent version of any of these available, and if you know of other guidance that might be worth including. If you think any of this information is wrong in some way please let me know.

Guide to ratings

The ratings are based on my personal view after reading the document.

Clarity: This is out of 100, where 100 means perfectly clear and 0 means incomprehensible. Problems that can give a low score here include ambiguity, vagueness, low readability, and meaningless diagrams.

Ideas: This is out of 100, where 100 means the document has lots of good, fresh ideas in it and 0 means it has no good, fresh ideas in it. You might read a document with good ideas in it to gather those ideas. Having bad ideas too does not affect this score, nor does repeating very well known ideas.

Guidance: This is out of 100, where 100 means the guide can be understood and followed and nothing more is needed for an effective, efficient approach. A score 0 means the guide is useless. Problems that can give a low score here include failing to cover important topics, lack of clarity, and advocating practices that don't work very well or are logically flawed.

Openness: This is out of 100, where 100 means the document is consistent with all possible practices and 0 means it is not consistent with any practices. The more open a guide is the more likely it is that an organization can be consistent with it without doing anything differently.

Overall star rating: This is out of 6 stars, where 6 stars would be the ultimate document in this area and no stars means the document is abysmal. The fact that most guides get a low star rating reflects my belief that we can do much, much better.

Common flaws

In reading these documents certain flaws came up repeatedly:

• The ‘Risks are Real’ group: These reflect a tendency to think that risks are real things, like buses, or at least that they are naturally occuring things that define themselves and whose existence and probabilities exist without a human mind. In fact risks are mental constructs with which we think about the real world and we can define their boundaries as we wish. The probabilities involved are also a function of the information we have and choose to use. Risk analysis involves structuring the total uncertainty we face in a useful way.

• Lack of risk definition: The mistaken belief that risks are real things is probably why many guides fail to explain that risks need to be defined carefully. In practice, poorly defined risks are common, so this is an important omission.

• Lack of structure: Many guides fail to point out the benefit of structuring a risk analysis. Their idea is that risks are identified, meaning that they already exist and just have to be spotted. In fact they need to be defined and it helps to do so in a structured way.

• Ignoring information level: Guides often forget that our assessments of likelihood reflect the information we choose to use. Consequently they talk about 'the' probability of something happening and have little or not guidance on getting more information.

• Ignoring spread: Not only are risks seen as real things, but they are also seen as single real things. Consequently, it is common to see risk rating systems refer to the impact of some risk if it happened as if that is just one possible outcome with just one possible level of impact. In reality almost all risks that people think of have a range of possible impacts.

• Ignoring connections: Not only are risks seen as real and single, but they are also often seen as separate from each other, meaning that the occurrence of one has no implications for others. In reality this is almost never the case, so a list of unconnected items (usually known as a risk register) is a poor representation.

• The ‘Analysis Focus’ group: The mistaken idea here is that analysis of risk is incredibly important and difficult but thinking of things to do about it is easy and requires almost no time, effort, or skill. Design of controls/risk responses doesn't appear at all in some of the guides despite chapters on various stages of analysis.

• Premature acceptance: Several of the guides advise making decisions about what risks are acted on before considering what you could actually do. Others advise making decisions when the possible actions are described only at the most abstract and generic level possible, long before a usefully accurate view of effectiveness or cost can be formed. In practice, people do have a vague idea of feasibility and costs in the backs of their minds and this stops the whole process from failing completely as it would if the advice was followed rigorously.

• Lack of design effort: Few of the guides give any attention to the importance of devoting time, effort, and skill to the design of actions/controls. In reality this is a huge part of the total effort of risk management. The value of risk management is limited to the value of the best controls that get thought of.

• The ‘Sausage Machine’ group: These are problems arising from seeing risk management as somehow separate from other activities. Despite frequently advising readers that their risk management should be 'embedded' and 'integrated' into management activities most guides continue to describe linear or circular risk management processes with nothing else involved, specify risk reporting without suggesting that reports generally should include uncertainty, and generally carry on as if nothing else exists. Actual practice inspired by these guides continues to be separate from other management, with separate meetings and documents, despite the fact that uncertainty is a continual and ubiquitous challenge in management.

• Lack of integration: The guides rarely have much to say about exactly how their processes are to be integrated into management activities.

• Lack of progression: For a number of important practical reasons it is a good idea for risk control thinking to be improved over successive meetings or drafts, not just updated. This improvement will include small refinements, major reorganizations, and a general move towards more structure, more connection, and more clarity. This way new insights and ideas for action arise and the activities do not become stale and boring. Guides often fail to point out the importance of progression and instead write as if you go through the sausage machine process once and after than further iterations are just updates.

Words © 2010 Matthew Leitch. First published 26 May 2010.