Working In Uncertainty
Report of a survey on recommendations by auditors and risk managers
by Matthew Leitch; first published 2004
This study was first reported in August 2004 on www.internalcontrolsdesign.co.uk and has been reanalysed and reformatted for Working In Uncertainty.
It explores the extent to which people who make recommendations on ‘risk management’ and ‘internal control’ are able to recognize the value of other improvements to working in uncertainty.
The results of this research strongly suggest that auditors, risk managers, and others who make recommendations for improving ‘internal control’ and ‘risk management’ recognize the value of improvements to working in uncertainty that are not from their familiar repertoire of sign offs, documentation, segregation of duties, reconciliations, and separate risk listing processes.
However, there was a gap between what they recognized as a good recommendation and what they thought was a recommendation they were expected to make.
These findings suggest that auditors and others who make recommendations on ‘internal control’ and ‘risk management’ could be encouraged to make more recommendations for improving performance under uncertainty that go beyond their traditional repertoire.
Survey questions and results
The survey form presented respondents with eight imaginary reviews (i.e. the scenarios) in the order shown below:
Each scenario was followed by five potential recommendations for improvement, displayed in a random order that differed for each respondent.
Respondents were asked three questions about each recommendation using the following instructions:
‘Imagine that internal auditors or other risk management or internal control specialists have been doing some reviews of activities in an imaginary organisation. Each review found problems and some recommendations are under consideration.’
‘For each recommendation consider the following:’
‘All the recommendations should be taken individually. They are not intended to be linked. Also don't worry about the fact that the imaginary organisation seems to do a lot of unrelated things! The examples are drawn from life but not in the same organisation.’
The specific recommendations and the percentage of respondents who thought each recommendation was probably a good one are shown in the following table. Respondents classified themselves as either ‘Risk manager’, ‘Auditor’, ‘Performance Manager’, or ‘Other’ but because the number of respondents was quite low, the differences between groups should not be taken too seriously. Only the two larger groups – risk managers and auditors – are shown separately in the table.
(The Group was not shown to respondents but is used in the analysis that follows these raw results.)
The key point is that most respondents recognized the value of all the potential recommendations, whether they were familiar internal control (IC) or risk listing process (RMP) ideas, or other ideas that would still contribute to better working in uncertainty (WIU) but are not emphasized in audit training, guidance, and audit programmes. The lowest level of overall support was 67%, and that was for a conventional recommendation about using targets. In contrast, 100% of respondents supported the more progressive item within ‘conference choices’ where the conferences team was recommended to review past experiences and consider predictability of results, then think of ways to manage more flexibly.
Here is a scatter graph showing how the concept groups are distributed:
The most interesting types of recommendation are towards the bottom right of the graph.
The recommendations in the Targets group were brought down by the context used. The two recommendations on using targets were the least popular of all, though still thought to be good by most respondents. The recommendation is to set targets for individual products and ensure that each product meets its target. These were the only items in the survey I considered bad recommendations and it is comforting to know that one marketing specialist who responded to the survey (and identified himself to me) agreed. In other contexts the idea of setting fixed targets and managing towards them would have got higher approval, but in the product portfolio case more people are aware of the problems. The choice of this context for the Targets recommendations probably brought down the Targets group more than most would have done.
In the table above the concept groups have been classified as ‘New’ or ‘Old’ according to my perception of which ones are traditional audit mainstays, based on over 10 years of audit experience. The final summary is of the figures for new and old recommendations:
Public sector versus private sector
Public and private sector responses did not differ noticeably in any way except one. The proportion of recommendations public sector respondents felt they were expected to make was dramatically higher than for the private sector respondents, and this difference was greatest for recommendations in the ‘new’ concept groups.
Only 16 of the 46 respondents were from the public sector, so to give an idea of how statistically significant the difference is I have calculated that if there was in fact no difference between the public and private sectors the probability of getting the result I got or something more extreme is only 12.4%, using a 1 tailed T test. In other words, it is very likely that this is a real difference, despite the sample size.
The results seemed pretty much the same regardless of whether respondents were auditors, risk managers, performance managers or held some other role. This was a bit of a surprise as I had thought risk managers would be expected to make more varied recommendations than auditors.
Differences between scenarios
Different scenarios produced different responses, but I could see no meaning in the pattern. The low figure for ‘In place’ for the overall review is perhaps because respondents may have had difficulty seeing a specific control within the recommendation.
Implications of results
Auditors in particular are relentlessly pushed to focus on a range of familiar internal control techniques and to make recommendations about risk management that are based on risk listing, despite its many shortcomings. Many ‘operational’ risk managers have a similar background and focus.
And yet, despite this, respondents were quite capable of seeing the value of other types of recommendation, including quite progressive ideas that do not rely on management-by-targets and approved documentation.
Although the four groups (Risk managers, Auditors, Performance managers, and Others) are too small for firm conclusions to be drawn, the differences do seem to meet reasonable expectations, in most cases. For example, the risk managers more often thought they were expected to make recommendations about risk management processes and other working in uncertainty ideas than the auditors did.
More importantly, this analysis shows a gap between what respondents recognized as valuable and what they thought they were expected to recommend. In particular, though WIU recommendations had 90% support from auditors, only 48% thought they were expected to make those sorts of recommendations.
Furthermore, WIU recommendations in particular seemed to be worth attention because they were about equally supported by respondents but much less likely to be in place already. Consequently, if you go hunting for opportunities to make these recommendations then you are more likely to find them.
It seems that if auditors and risk managers were encouraged to make more recommendations of the WIU type then they would be willing and able (provided they could think of the recommendations themselves rather than just recognizing suggestions as valuable), and would find more opportunities to do so.
Comments by respondents
The survey asked if the respondent had any other comments they would like to make about the survey or about recommendations. Excluding comments purely about the survey, the respondent comments were:
‘Wow, these questions made my head hurt. Any time you wandered into recommendations regarding modelling and statistics, I backed off. I haven't thought about those matters since college Econometric courses some 20 years ago. As internal auditors are primarily accountants first, and operational observers secondly, we would not be expected by my current organization to make comments on statistical models or product marketing recommendations. Because of the detailed nature of these comments, I'm much more comfortable suggesting a topic for strategic direction than a specific management model. The point is to get management to clarify their aims and directions. Internal audit attempts to help them formalize this and then audit to their stated objectives. We are not subject matter experts. But we can comment on the effectiveness of processes based on outcomes.’
‘There is a lot of work to be done, especially in the government sector in incorporating risk management (information/operations risk). I would like to see more concrete work on how to integrate or rather align specific business goals/missions with the IT part of the business. To date, it still seems to me there is a big gap. The push from C-level executives to figure out ROI/ROSI (Return on Investment/Return on Security Investment) are still rather “fuzzy” numbers. How do we really get from here to a fully integrated risk management organization where risk is just business as usual and as much as possible, fully automated and dynamic?’
‘The “probably expected” box I interpreted to mean “probably expected within my current role”. Working on a Sarbox project, I am planning to raise business issues but am not generally expected to do so. Some of the survey's suggested recommendations are also somewhat outside management's own expectations of a process or touching on areas that management are not expecting me to review – e.g. strategic marketing decisions and are not included on those grounds.’
‘Risk management itself is a risk process – the degree of risk evaluation/management depends on many factors, and the answers to the questions above, of necessity, cannot assess all the factors that would apply, especially between e.g. SMEs and multinationals.’
‘Quantification doesn't help if there is no suitable information to quantify – which is often the case.’
‘The oil case encourages the use of real options methodologies. However, virtually all the cases would benefit from this perspective.’
‘Much of what's “recommended” above is (or should be) standard public sector practice, given the push to formal project management (It's all in PRINCE). Risk management (Per HM Treasury) and OGC Gateway reviews.’
Respondents were invited to participate using professional discussion lists on the Internet and by some personal e-mails, but only where I was confident the person would not be predisposed to answer in a particular way. For example, if someone wrote to me about how they were interested in evolutionary project management I would not invite them to participate.
Most respondents were from the USA and UK.
Most were internal auditors and risk managers.
Limitations of the survey
Besides the obvious limitations of a study where respondents represent a tiny section of a large population and are self-selected, and where the number of respondents is fairly small, there are some other imperfections in the design that should be borne in mind.
The recommendations were nearly all intended to be good ones, so that some respondents may have been led into a pattern of answering ‘Good’ to all of them. If there had been more bad recommendations to consider then the proportion of Good responses to the good recommendations might have been lower.
Some respondents had technical difficulties with their browser's rendition of the survey and one reported that he had been unable to amend answers to some questions. This was not part of the design. The survey worked perfectly when tested on Internet Explorer.
Although the evidence suggests that the new types of recommendation are likely to be very useful to auditors and others who make recommendations on controls there is a missing step in the chain of reasoning. The survey does not show how often in practice the newer types of control are appropriate. It only shows that there are good recommendations for controls that are unlikely to be in place already.
Finally, the simple binary responses asked for do not tell us how confident respondents were that particular controls would already be in place or how good they thought the recommendations were. We only know what proportion of respondents thought the controls would probably be in place and what proportion of respondents thought the recommendations were probably good. There is a difference, though one would expect strength of feeling and volume of support to be correlated to some extent.
Company: The Ridgeway Expertise Company Ltd, registered in England, no. 04931400.
Registered office: 29 Ridgeway, KT19 8LD, United Kingdom.
Words © 2004 Matthew Leitch