Working In Uncertainty
Why the COSO frameworks need improvement: Issues with COSO's internal control framework and ERM framework
by Matthew Leitch; first appeared on www.irmi.com in March 2005.
I can't think of a document that has had more influence on thinking about internal control than COSO's ‘Internal Control – Integrated Framework’. It is endlessly quoted and paraphrased in control and governance documents for different sectors and has recently become the de facto standard for controls over financial reporting thanks to the SEC's interpretation of the Sarbanes-Oxley Act 2002. Thousands of people have written hundreds of thousands of pages about their internal controls using formats taken from this framework.
More recently COSO has published ‘Enterprise Risk Management – Integrated Framework’ which some are already calling ‘COSO II.’ This looks set to be as influential as the internal control framework.
So, even quite small technical weaknesses in these documents could have huge practical implications.
In this article I will show that weaknesses do exist and they are far from small. The practical implications are huge and we need to press for improvements with the minimum of delay.
What's wrong with COSO's internal control framework?
COSO's internal control framework was an exciting breakthrough in internal control thinking. Suddenly internal controls became a system instead of just a list of objectives or controls. There were definitions that expanded and defined the concept in an exciting new way.
At the time it seemed a great step forward but with the benefit of time and experience we can see the practical implications of some of its conceptual weaknesses.
The definition of internal control is so wide that almost every aspect of management is arguably part of management control. The definition reads:
‘Internal control is broadly defined as a process, effected by an entity's board of directors, management and other personnel, designed to provide reasonable assurance regarding the achievement of objectives in the following categories:
When the internal control framework was launched its most ardent supporters (in Coopers & Lybrand for example) saw it as a complete guide to management. The idea was that for business success you just define some objectives and the rest is internal control.
The framework divided risks into three categories: operational, financial reporting, and compliance.
At first this seems clear enough, but what about financial reporting that must be reliable to be compliant? Where do you draw the line between data processing for doing business and data processing for financial reporting?
Most confusing of all for most people are the five components of internal control. The ‘Control activities’ component is straightforward enough but who can honestly say they aren't just a tiny bit hazy on ‘information and communication’, ‘monitoring’, ‘risk assessment’, and of course ‘control environment’?
All these problems are minor compared to a part of the framework that isn't even mentioned in the executive summary. One of the books in the COSO set is called the Evaluation Tools. It includes a large number of illustrative control matrices showing what controls might be in place for every major process in a typical business. In effect these matrices are lists of control objectives, with controls next to them.
This format has been taken up by auditors and companies desperate to comply with s404 of the Sarbanes-Oxley Act 2002 so there are already hundreds of thousands of such matrices around the world.
And that's a pity because, for reasons I will now explain, the format is unreliable and impractical. When COSO's internal control framework was written and consultation was in progress who at that time had any inkling of the use to which these matrices might be put? How many reviewers had the interest or patience to even comment on them?
If people had known at the time what use would be made of these matrices I don't think they would have been published, at least in their current form.
Unless you use a computer system that can display controls in other ways too the COSO matrix will produce the following problems:
In short, COSO matrices are very hard to review properly, are rarely of good quality, and don't give a usable list of controls.
The sooner this situation is corrected the better. The most practical thing to do in the short term is simply to remove the Evaluation Tools from the framework.
What's wrong with COSO's ERM framework?
COSO is to be congratulated on a document that was produced with public consultation and tries hard to recognise a wide variety of alternative ways to manage risk. It shows great knowledge of risk management techniques and contains many interesting examples.
Unfortunately, with two volumes totalling 246 pages, it is so large that it is hard to see how every part of it can have received adequate comment during the consultation phase. Although the published documents reveal that there were 78 responses to the consultation the responses themselves have not been made public so we cannot know how much of the documents was seriously considered.
My impression of the two volumes is that there are a lot of ideas there that are new or different from usual practice, and some distinctions that will not be understood by most readers.
For example, many people will not notice on initial reading that ‘risk tolerances’ do not relate to risks (because there is no element of uncertainty). The distinction between ‘risk responses’ and ‘control activities’ will also be confusing.
In short, the ERM framework is far too big for a first version. Not surprisingly, it contains some obvious technical flaws.
For example, although keen to talk about ‘opportunities’ it doesn't have the logic worked out properly and the crucial paragraphs on what to do with them are unclear. For example, the document explains that if an event happens that is favourable then this is an opportunity that is sent to strategic planning so that plans can be made to take the opportunity. There is no such comment on what happens if an event happens that is unfavourable. Does that mean plans are left unchanged?
The crucial paragraphs on what happens to upside risks are unclear. My best guess is that some get taken out of risk management to be looked at elsewhere, while others stay in risk management. This appears to exclude the possibility of integrated uncertainty management that deals with both unexpected good and bad events in one approach.
Another problem is the many examples of rating risk items for their probability and impact.
When risk register items are rated using (1) a number for probability of occurrence, and (2) a number for impact on occurrence, their risk is systematically understated. It is hard to see the problem when ratings are as rough as High/Medium/Low, but when numbers are given the fault is obvious.
Imagine that at an early stage in a project the risk of ‘Overspend due to client originated changes’ was rated as Probability = 0.5 and Impact = £10m. By this method the possibility of an impact other than £10m has been excluded and we should be particularly concerned that the risk of impact greater than £10m has disappeared from view.
For an item like this there may well be a 20% chance of an overspend of more than £15m, for example, and this is obviously something people need to know!
The framework is also overly narrow, something else that would have been less likely to happen with a shorter document. For example:
Should we tolerate a document with the influence of COSO's framework containing logical flaws and being excessively prescriptive? The current version says that the Internal Control framework remains the document for internal controls assessment (i.e. for Sarbanes-Oxley purposes) but the ERM framework is clearly designed to supersede it one day. Sooner or later we will face the prospect of the ERM framework having virtually the same status as law.
What now may seem trivial theoretical gripes will in time emerge as major barriers to spreading the word about the benefits of great risk management.
COSO's internal control framework urgently needs updating, and the Evaluation Tools in particular should be removed until something better is available.
The ERM framework is new but before it becomes the basis for some future regulatory paper-chase we should press for it to become shorter, more open, and less flawed.
Words © 2005 Matthew Leitch. First published March 2005.