Working In Uncertainty

Report of a survey on recommendations by auditors and risk managers

by Matthew Leitch; first published 2004

Contents

This study was first reported in August 2004 on www.internalcontrolsdesign.co.uk and has been reanalysed and reformatted for Working In Uncertainty.

It explores the extent to which people who make recommendations on ‘risk management’ and ‘internal control’ are able to recognize the value of other improvements to working in uncertainty.

Summary

The results of this research strongly suggest that auditors, risk managers, and others who make recommendations for improving ‘internal control’ and ‘risk management’ recognize the value of improvements to working in uncertainty that are not from their familiar repertoire of sign offs, documentation, segregation of duties, reconciliations, and separate risk listing processes.

However, there was a gap between what they recognized as a good recommendation and what they thought was a recommendation they were expected to make.

These findings suggest that auditors and others who make recommendations on ‘internal control’ and ‘risk management’ could be encouraged to make more recommendations for improving performance under uncertainty that go beyond their traditional repertoire.

Survey questions and results

The survey form presented respondents with eight imaginary reviews (i.e. the scenarios) in the order shown below:

NameScenario description
product developmentThe review looked at the way product ideas are developed and approved in a particular business unit.
conference choicesThe review looked at the way potential conferences were chosen and, in particular, how estimates of likely attendances were made. These are vital to the decision of whether to go ahead or not.
backbilling projectThe review looked at a project that is trying to identify past billing errors and, where possible, raise back charges with customers who have been under-charged.
project planningThe review looked at a project plan being developed for a large project that is vital to the future of the organisation and expected to last over 2 years.
service improvement planningThe review looked at plans to improve services to customers by introducing several innovations.
product managementThe review looked at how a set of products have been managed.
oil exploration business caseThe review looked at a business case for exploring a region for oil. The case includes extensive financial projections.
overall managementThe review looked at the management of risk and uncertainty throughout the organisation.

Each scenario was followed by five potential recommendations for improvement, displayed in a random order that differed for each respondent.

Respondents were asked three questions about each recommendation using the following instructions:

‘Imagine that internal auditors or other risk management or internal control specialists have been doing some reviews of activities in an imaginary organisation. Each review found problems and some recommendations are under consideration.’

‘For each recommendation consider the following:’

  • ‘Probably already in place’: In your experience of organisations would you expect the recommended action to have been taken already? Click the first checkbox on each line if you think that most organisations would not need the recommendation because they would already have done what is being recommended.

  • ‘Probably a good recommendation’: Click the second checkbox if you think the idea would probably be a good one, assuming the action has not already been taken. Don't get picky about the details of the recommendation or its wording. The basic idea being suggested is what matters. Assume there are appropriate details that go along with this simple summary.

  • ‘Probably expected’: For the third checkbox it depends if you make recommendations on risk management / internal control in your job.

    • If you do make recommendations about risk management/internal control in your job click the checkbox if you think you are expected to suggest ideas like this.

    • If you do not make recommendations on these topics in your job click the box if you would expect auditors and other risk management specialists to suggest the idea.

‘All the recommendations should be taken individually. They are not intended to be linked. Also don't worry about the fact that the imaginary organisation seems to do a lot of unrelated things! The examples are drawn from life but not in the same organisation.’

The specific recommendations and the percentage of respondents who thought each recommendation was probably a good one are shown in the following table. Respondents classified themselves as either ‘Risk manager’, ‘Auditor’, ‘Performance Manager’, or ‘Other’ but because the number of respondents was quite low, the differences between groups should not be taken too seriously. Only the two larger groups – risk managers and auditors – are shown separately in the table.

(The Group was not shown to respondents but is used in the analysis that follows these raw results.)

The key point is that most respondents recognized the value of all the potential recommendations, whether they were familiar internal control (IC) or risk listing process (RMP) ideas, or other ideas that would still contribute to better working in uncertainty (WIU) but are not emphasized in audit training, guidance, and audit programmes. The lowest level of overall support was 67%, and that was for a conventional recommendation about using targets. In contrast, 100% of respondents supported the more progressive item within ‘conference choices’ where the conferences team was recommended to review past experiences and consider predictability of results, then think of ways to manage more flexibly.

 Probably a good recommendation 
Review topicRecommendationRisk Mgrs (n=13)Auditors (n=25)All respon
-dents (n=46)
GroupConcept group
product developmentEnsure that product development approvals are given in writing and signed off by suitable officials in the organisation.92%96%96%ICSign.
product developmentWrite a policy on risk management for product development in the business unit.85%88%87%RMPDoc.
product developmentFormal proposals for new products to be approved should include a section listing risks and how, if at all, they can be managed.100%96%98%RMPRisk mgmt
product developmentProvide education/training for product developers on how risk and uncertainty affect their work and how identifying uncertainties that matter can guide their research efficiently.100%88%93%WIUEducate
product developmentInclude in each iteration of the product development process a step where the current areas of uncertainty/risk relevant to the idea under development are quickly listed and their impact is considered before actions to reduce the uncertainties or manage the risks are decided.92%100%98%WIUUncertainty
conference choicesApproval to proceed with a conference should be given by a committee of suitable managers and their approval should be evidenced in writing, for example in the minutes of the committee's meetings.85%96%91%ICSign.
conference choicesEvidence relating to likely attendance, revenues, and costs for proposed conferences should be documented in a written business case.92%100%96%WIUDoc.
conference choicesThe spreadsheet model estimating the financial results of a proposed conference should treat attendance as an uncertain variable with a probability distribution, and show the projected financial result as a distribution. From that it would be possible to see the estimated risk of, for example, making a loss. The number crunching can be done easily using widely available Excel add-ins.92%88%91%WIUQuantify
conference choicesThe conferences team should take time out to consider the range of outcomes from past conferences, how predictable they really are, and to think of ways they can manage conferences more flexibly and gain more information about visitors that will help in selecting conference topics, venues, and dates.100%100%100%WIULearning
conference choicesMore should be found out about interest in potential conferences, for example by using surveys and looking at the readership of related magazines and journals.85%92%91%WIULearning
backbilling projectWhere it is discovered that past bills to a customer have been incomplete but the back charges are to be waived this should be authorised appropriately and documented.100%96%96%ICSign.
backbilling projectPredictions about how much money the back-billing project will eventually raise should be reviewed independently before being used in revenue forecasts.77%92%85%ICSign.
backbilling projectThe project should have a formally agreed scope and definition document, and a project plan.85%92%89%ICDoc.
backbilling projectProjections about how much money the back-billing project will eventual raise should be expressed as ranges with probabilities rather than as a spuriously accurate 'best guess' number. For example, say the range of recoveries that is now 80% probable.85%92%87%WIUQuantify
backbilling projectA small set of back charges should be taken through to bills and attempted recovery of money from customers as soon as possible to learn more about what it will take to do this on a larger scale.85%84%83%WIULearning
project planningA risk management process should be put in place to identify significant risks to the project, plan responses, and track progress.100%96%98%RMPRisk mgmt
project planningThe project steering committee and project management team should set a good example by being open about uncertainties, communicating them, and showing that they expect others to be open with them.100%96%98%WIUUncertainty
project planningThe project management team should consider some form of training to develop their ability to talk openly about risk and uncertainty on the project and encourage others to report progress and risks honestly and completely.100%92%96%WIUEducate
project planningThe project plan should be reviewed to see if the dependencies can be reduced to improve the risk profile of the project.100%96%98%WIUStructure
project planningAs far as possible without creating inefficiency the project should be divided into short term deliveries to stakeholders, not just internal deliveries within the project. This would accelerate business benefits, reduce committed resource before benefit delivery, and increase learning from experience.85%100%93%WIUStructure
service improvement planningThe service improvement plan should be authorised in writing at a high level.92%92%91%ICSign.
service improvement planningRisks to improving service should be identified, documented, and assigned owners.92%92%93%RMPRisk mgmt
service improvement planningMore short term indicators of progress should be sought, as the existing indicators are too long term to be used alone.92%88%89%WIULearning
service improvement planningPriorities should be revised regularly – probably more often while ideas are still relatively untried.77%84%83%WIUAdaptability
service improvement planningSince the results from the new ideas are not certain the ideas should be trialled rapidly and revised as necessary as they are rolled out more widely. It is very important to learn as much as possible from experience.100%84%89%WIULearning
product managementChanges to resource allocations between products should be authorised appropriately and in writing.92%96%93%ICSign.
product managementClear revenue, growth, and profit targets should be agreed for each product annually.92%88%89%ICTargets
product managementProducts should be managed tightly to ensure that each product meets its annual targets.77%64%67%ICTargets
product managementThe products should be managed as a portfolio during the year, with new products that go well being given extra resources to develop, while disappointing products get less.100%80%85%WIUAdaptability
product managementAlternative promotional strategies should be tried to find out which work best in each category.85%76%83%WIULearning
oil exploration business caseA formal risk assessment exercise should be carried out.92%100%98%RMPRisk mgmt
oil exploration business caseThe computer model underlying the financial projections should be independently reviewed to ensure that it is correctly programmed.92%100%96%ICSign.
oil exploration business caseThe source of all evidence used in making estimates should be stated, even if it is just to point out the name of the person whose gut feel it is.77%88%87%WIUUncertainty
oil exploration business caseThe financial model needs to reflect the fact that decisions about whether to proceed further and how will be taken at various points in the proposed exploration. These options should be valued.100%96%96%WIUQuantify
oil exploration business caseThere are various uncertain variables in the projection and these should be modelled using probability distributions to explicitly represent the uncertainty and avoid the flaw of averages.100%80%89%WIUQuantify
overall managementDocumentary evidence of internal controls / risk management should be enhanced so that any failure to carry out agreed controls is highlighted promptly.92%100%96%ICDoc.
overall managementRisk management procedures should be revised to encourage people to revisit risks and responses much more often, to stay up to date, and to focus on things that are more specific and topical.100%96%98%RMPUncertainty
overall managementMore effort should be made to incorporate risk/uncertainty awareness into strategic decision making and not just routine clerical procedures.100%100%100%WIUUncertainty
overall managementThe risk management approach should also address the personal risk/uncertainty awareness, skills, and attitudes of staff, particularly managers at all levels.92%84%89%WIUEducate
overall managementThe way risk and uncertainty are quantified should be improved so that more numerical modelling and empirical support are used where appropriate.77%80%83%WIUQuantify

Concept groups

Here is a scatter graph showing how the concept groups are distributed:

The most interesting types of recommendation are towards the bottom right of the graph.

The recommendations in the Targets group were brought down by the context used. The two recommendations on using targets were the least popular of all, though still thought to be good by most respondents. The recommendation is to set targets for individual products and ensure that each product meets its target. These were the only items in the survey I considered bad recommendations and it is comforting to know that one marketing specialist who responded to the survey (and identified himself to me) agreed. In other contexts the idea of setting fixed targets and managing towards them would have got higher approval, but in the product portfolio case more people are aware of the problems. The choice of this context for the Targets recommendations probably brought down the Targets group more than most would have done.

In the table above the concept groups have been classified as ‘New’ or ‘Old’ according to my perception of which ones are traditional audit mainstays, based on over 10 years of audit experience. The final summary is of the figures for new and old recommendations:

 In PlaceGoodExpected
Old51.2%90.9%58.1%
New30.2%90.6%50.1%

Public sector versus private sector

Public and private sector responses did not differ noticeably in any way except one. The proportion of recommendations public sector respondents felt they were expected to make was dramatically higher than for the private sector respondents, and this difference was greatest for recommendations in the ‘new’ concept groups.

Only 16 of the 46 respondents were from the public sector, so to give an idea of how statistically significant the difference is I have calculated that if there was in fact no difference between the public and private sectors the probability of getting the result I got or something more extreme is only 12.4%, using a 1 tailed T test. In other words, it is very likely that this is a real difference, despite the sample size.

Respondent roles

The results seemed pretty much the same regardless of whether respondents were auditors, risk managers, performance managers or held some other role. This was a bit of a surprise as I had thought risk managers would be expected to make more varied recommendations than auditors.

Differences between scenarios

Different scenarios produced different responses, but I could see no meaning in the pattern. The low figure for ‘In place’ for the overall review is perhaps because respondents may have had difficulty seeing a specific control within the recommendation.

ScenarioIn PlaceGoodExpected
prod dev33.5%94.3%57.4%
confer47.4%93.9%51.3%
backbill37.8%87.8%52.6%
project39.1%96.5%55.2%
service38.3%89.1%51.3%
prod mgmt59.6%83.5%47.8%
oil38.7%93%53.5%
overall18.7%93%58.7%

Implications of results

Auditors in particular are relentlessly pushed to focus on a range of familiar internal control techniques and to make recommendations about risk management that are based on risk listing, despite its many shortcomings. Many ‘operational’ risk managers have a similar background and focus.

And yet, despite this, respondents were quite capable of seeing the value of other types of recommendation, including quite progressive ideas that do not rely on management-by-targets and approved documentation.

Although the four groups (Risk managers, Auditors, Performance managers, and Others) are too small for firm conclusions to be drawn, the differences do seem to meet reasonable expectations, in most cases. For example, the risk managers more often thought they were expected to make recommendations about risk management processes and other working in uncertainty ideas than the auditors did.

More importantly, this analysis shows a gap between what respondents recognized as valuable and what they thought they were expected to recommend. In particular, though WIU recommendations had 90% support from auditors, only 48% thought they were expected to make those sorts of recommendations.

Furthermore, WIU recommendations in particular seemed to be worth attention because they were about equally supported by respondents but much less likely to be in place already. Consequently, if you go hunting for opportunities to make these recommendations then you are more likely to find them.

It seems that if auditors and risk managers were encouraged to make more recommendations of the WIU type then they would be willing and able (provided they could think of the recommendations themselves rather than just recognizing suggestions as valuable), and would find more opportunities to do so.

GroupRisk managers (n=13)Auditors (n=25)Performance managers (n=4)Other (n=4)All (n=46)
In place
RMP42%38%71%25%41%
IC59%51%59%45%53%
WIU40%28%37%25%32%
Good idea
RMP95%95%100%96%95%
IC89%92%86%84%90%
WIU92%90%95%92%91%
Expected
RMP72%63%54%29%62%
IC58%60%50%36%57%
WIU54%48%63%37%50%

Comments by respondents

The survey asked if the respondent had any other comments they would like to make about the survey or about recommendations. Excluding comments purely about the survey, the respondent comments were:

‘Wow, these questions made my head hurt. Any time you wandered into recommendations regarding modelling and statistics, I backed off. I haven't thought about those matters since college Econometric courses some 20 years ago. As internal auditors are primarily accountants first, and operational observers secondly, we would not be expected by my current organization to make comments on statistical models or product marketing recommendations. Because of the detailed nature of these comments, I'm much more comfortable suggesting a topic for strategic direction than a specific management model. The point is to get management to clarify their aims and directions. Internal audit attempts to help them formalize this and then audit to their stated objectives. We are not subject matter experts. But we can comment on the effectiveness of processes based on outcomes.’

‘There is a lot of work to be done, especially in the government sector in incorporating risk management (information/operations risk). I would like to see more concrete work on how to integrate or rather align specific business goals/missions with the IT part of the business. To date, it still seems to me there is a big gap. The push from C-level executives to figure out ROI/ROSI (Return on Investment/Return on Security Investment) are still rather “fuzzy” numbers. How do we really get from here to a fully integrated risk management organization where risk is just business as usual and as much as possible, fully automated and dynamic?’

‘The “probably expected” box I interpreted to mean “probably expected within my current role”. Working on a Sarbox project, I am planning to raise business issues but am not generally expected to do so. Some of the survey's suggested recommendations are also somewhat outside management's own expectations of a process or touching on areas that management are not expecting me to review – e.g. strategic marketing decisions and are not included on those grounds.’

‘Risk management itself is a risk process – the degree of risk evaluation/management depends on many factors, and the answers to the questions above, of necessity, cannot assess all the factors that would apply, especially between e.g. SMEs and multinationals.’

‘Quantification doesn't help if there is no suitable information to quantify – which is often the case.’

‘The oil case encourages the use of real options methodologies. However, virtually all the cases would benefit from this perspective.’

‘Much of what's “recommended” above is (or should be) standard public sector practice, given the push to formal project management (It's all in PRINCE). Risk management (Per HM Treasury) and OGC Gateway reviews.’

Respondent profile

Respondents were invited to participate using professional discussion lists on the Internet and by some personal e-mails, but only where I was confident the person would not be predisposed to answer in a particular way. For example, if someone wrote to me about how they were interested in evolutionary project management I would not invite them to participate.

Most respondents were from the USA and UK.

Most were internal auditors and risk managers.

Limitations of the survey

Besides the obvious limitations of a study where respondents represent a tiny section of a large population and are self-selected, and where the number of respondents is fairly small, there are some other imperfections in the design that should be borne in mind.

The recommendations were nearly all intended to be good ones, so that some respondents may have been led into a pattern of answering ‘Good’ to all of them. If there had been more bad recommendations to consider then the proportion of Good responses to the good recommendations might have been lower.

Some respondents had technical difficulties with their browser's rendition of the survey and one reported that he had been unable to amend answers to some questions. This was not part of the design. The survey worked perfectly when tested on Internet Explorer.

Although the evidence suggests that the new types of recommendation are likely to be very useful to auditors and others who make recommendations on controls there is a missing step in the chain of reasoning. The survey does not show how often in practice the newer types of control are appropriate. It only shows that there are good recommendations for controls that are unlikely to be in place already.

Finally, the simple binary responses asked for do not tell us how confident respondents were that particular controls would already be in place or how good they thought the recommendations were. We only know what proportion of respondents thought the controls would probably be in place and what proportion of respondents thought the recommendations were probably good. There is a difference, though one would expect strength of feeling and volume of support to be correlated to some extent.

 

Company: The Ridgeway Expertise Company Ltd, registered in England, no. 04931400.

Registered office: 29 Ridgeway, KT19 8LD, United Kingdom.

Words © 2004 Matthew Leitch