Working In Uncertainty
Report of a survey on recommendations by auditors and risk managers
This study was first reported in August 2004 on www.internalcontrolsdesign.co.uk and has been reanalysed and reformatted for Working In Uncertainty.
It explores the extent to which people who make recommendations on 'risk management' and 'internal control' are able to recognize the value of other improvements to working in uncertainty.
The results of this research strongly suggest that auditors, risk managers, and others who make recommendations for improving 'internal control' and 'risk management' recognize the value of improvements to working in uncertainty that are not from their familiar repertoire of sign offs, documentation, segregation of duties, reconciliations, and separate risk listing processes.
However, there was a gap between what they recognized as a good recommendation and what they thought was a recommendation they were expected to make.
These findings suggest that auditors and others who make recommendations on 'internal control' and 'risk management' could be encouraged to make more recommendations for improving performance under uncertainty that go beyond their traditional repertoire.
Survey questions and results
The survey form presented respondents with eight imaginary reviews (i.e. the scenarios) in the order shown below:
|product development||The review looked at the way product ideas are developed and approved in a particular business unit.|
|conference choices||The review looked at the way potential conferences were chosen and, in particular, how estimates of likely attendances were made. These are vital to the decision of whether to go ahead or not.|
|backbilling project||The review looked at a project that is trying to identify past billing errors and, where possible, raise back charges with customers who have been under-charged.|
|project planning||The review looked at a project plan being developed for a large project that is vital to the future of the organisation and expected to last over 2 years.|
|service improvement planning||The review looked at plans to improve services to customers by introducing several innovations.|
|product management||The review looked at how a set of products have been managed.|
|oil exploration business case||The review looked at a business case for exploring a region for oil. The case includes extensive financial projections.|
|overall management||The review looked at the management of risk and uncertainty throughout the organisation.|
Each scenario was followed by five potential recommendations for improvement, displayed in a random order that differed for each respondent.
Respondents were asked three questions about each recommendation using the following instructions:
"Imagine that internal auditors or other risk management or internal control specialists have been doing some reviews of activities in an imaginary organisation. Each review found problems and some recommendations are under consideration."
"For each recommendation consider the following:"
"Probably already in place": In your experience of organisations would you expect the recommended action to have been taken already? Click the first checkbox on each line if you think that most organisations would not need the recommendation because they would already have done what is being recommended.
"Probably a good recommendation": Click the second checkbox if you think the idea would probably be a good one, assuming the action has not already been taken. Don't get picky about the details of the recommendation or its wording. The basic idea being suggested is what matters. Assume there are appropriate details that go along with this simple summary.
"Probably expected": For the third checkbox it depends if you make recommendations on risk management / internal control in your job.
If you do make recommendations about risk management/internal control in your job click the checkbox if you think you are expected to suggest ideas like this.
If you do not make recommendations on these topics in your job click the box if you would expect auditors and other risk management specialists to suggest the idea.
"All the recommendations should be taken individually. They are not intended to be linked. Also don't worry about the fact that the imaginary organisation seems to do a lot of unrelated things! The examples are drawn from life but not in the same organisation."
The specific recommendations and the percentage of respondents who thought each recommendation was probably a good one are shown in the following table. Respondents classified themselves as either "Risk manager", "Auditor", "Performance Manager", or "Other" but because the number of respondents was quite low, the differences between groups should not be taken too seriously. Only the two larger groups - risk managers and auditors - are shown separately in the table.
(The Group was not shown to respondents but is used in the analysis that follows these raw results.)
The key point is that most respondents recognized the value of all the potential recommendations, whether they were familiar internal control (IC) or risk listing process (RMP) ideas, or other ideas that would still contribute to better working in uncertainty (WIU) but are not emphasized in audit training, guidance, and audit programmes. The lowest level of overall support was 67%, and that was for a conventional recommendation about using targets. In contrast, 100% of respondents supported the more progressive item within 'conference choices' where the conferences team was recommended to review past experiences and consider predictability of results, then think of ways to manage more flexibly.
| ||Probably a good recommendation|| |
|Review topic||Recommendation||Risk Mgrs (n=13)||Auditors (n=25)||All respon|
|product development||Ensure that product development approvals are given in writing and signed off by suitable officials in the organisation.||92%||96%||96%||IC|
|product development||Write a policy on risk management for product development in the business unit.||85%||88%||87%||RMP|
|product development||Formal proposals for new products to be approved should include a section listing risks and how, if at all, they can be managed.||100%||96%||98%||RMP|
|product development||Provide education/training for product developers on how risk and uncertainty affect their work and how identifying uncertainties that matter can guide their research efficiently.||100%||88%||93%||WIU|
|product development||Include in each iteration of the product development process a step where the current areas of uncertainty/risk relevant to the idea under development are quickly listed and their impact is considered before actions to reduce the uncertainties or manage the risks are decided.||92%||100%||98%||WIU|
|conference choices||Approval to proceed with a conference should be given by a committee of suitable managers and their approval should be evidenced in writing, for example in the minutes of the committee's meetings.||85%||96%||91%||IC|
|conference choices||Evidence relating to likely attendance, revenues, and costs for proposed conferences should be documented in a written business case.||92%||100%||96%||WIU|
|conference choices||The spreadsheet model estimating the financial results of a proposed conference should treat attendance as an uncertain variable with a probability distribution, and show the projected financial result as a distribution. From that it would be possible to see the estimated risk of, for example, making a loss. The number crunching can be done easily using widely available Excel add-ins.||92%||88%||91%||WIU|
|conference choices||The conferences team should take time out to consider the range of outcomes from past conferences, how predictable they really are, and to think of ways they can manage conferences more flexibly and gain more information about visitors that will help in selecting conference topics, venues, and dates.||100%||100%||100%||WIU|
|conference choices||More should be found out about interest in potential conferences, for example by using surveys and looking at the readership of related magazines and journals.||85%||92%||91%||WIU|
|backbilling project||Where it is discovered that past bills to a customer have been incomplete but the back charges are to be waived this should be authorised appropriately and documented.||100%||96%||96%||IC|
|backbilling project||Predictions about how much money the back-billing project will eventually raise should be reviewed independently before being used in revenue forecasts.||77%||92%||85%||IC|
|backbilling project||The project should have a formally agreed scope and definition document, and a project plan.||85%||92%||89%||IC|
|backbilling project||Projections about how much money the back-billing project will eventual raise should be expressed as ranges with probabilities rather than as a spuriously accurate 'best guess' number. For example, say the range of recoveries that is now 80% probable.||85%||92%||87%||WIU|
|backbilling project||A small set of back charges should be taken through to bills and attempted recovery of money from customers as soon as possible to learn more about what it will take to do this on a larger scale.||85%||84%||83%||WIU|
|project planning||A risk management process should be put in place to identify significant risks to the project, plan responses, and track progress.||100%||96%||98%||RMP|
|project planning||The project steering committee and project management team should set a good example by being open about uncertainties, communicating them, and showing that they expect others to be open with them.||100%||96%||98%||WIU|
|project planning||The project management team should consider some form of training to develop their ability to talk openly about risk and uncertainty on the project and encourage others to report progress and risks honestly and completely.||100%||92%||96%||WIU|
|project planning||The project plan should be reviewed to see if the dependencies can be reduced to improve the risk profile of the project.||100%||96%||98%||WIU|
|project planning||As far as possible without creating inefficiency the project should be divided into short term deliveries to stakeholders, not just internal deliveries within the project. This would accelerate business benefits, reduce committed resource before benefit delivery, and increase learning from experience.||85%||100%||93%||WIU|
|service improvement planning||The service improvement plan should be authorised in writing at a high level.||92%||92%||91%||IC|
|service improvement planning||Risks to improving service should be identified, documented, and assigned owners.||92%||92%||93%||RMP|
|service improvement planning||More short term indicators of progress should be sought, as the existing indicators are too long term to be used alone.||92%||88%||89%||WIU|
|service improvement planning||Priorities should be revised regularly - probably more often while ideas are still relatively untried.||77%||84%||83%||WIU|
|service improvement planning||Since the results from the new ideas are not certain the ideas should be trialled rapidly and revised as necessary as they are rolled out more widely. It is very important to learn as much as possible from experience.||100%||84%||89%||WIU|
|product management||Changes to resource allocations between products should be authorised appropriately and in writing.||92%||96%||93%||IC|
|product management||Clear revenue, growth, and profit targets should be agreed for each product annually.||92%||88%||89%||IC|
|product management||Products should be managed tightly to ensure that each product meets its annual targets.||77%||64%||67%||IC|
|product management||The products should be managed as a portfolio during the year, with new products that go well being given extra resources to develop, while disappointing products get less.||100%||80%||85%||WIU|
|product management||Alternative promotional strategies should be tried to find out which work best in each category.||85%||76%||83%||WIU|
|oil exploration business case||A formal risk assessment exercise should be carried out.||92%||100%||98%||RMP|
|oil exploration business case||The computer model underlying the financial projections should be independently reviewed to ensure that it is correctly programmed.||92%||100%||96%||IC|
|oil exploration business case||The source of all evidence used in making estimates should be stated, even if it is just to point out the name of the person whose gut feel it is.||77%||88%||87%||WIU|
|oil exploration business case||The financial model needs to reflect the fact that decisions about whether to proceed further and how will be taken at various points in the proposed exploration. These options should be valued.||100%||96%||96%||WIU|
|oil exploration business case||There are various uncertain variables in the projection and these should be modelled using probability distributions to explicitly represent the uncertainty and avoid the flaw of averages.||100%||80%||89%||WIU|
|overall management||Documentary evidence of internal controls / risk management should be enhanced so that any failure to carry out agreed controls is highlighted promptly.||92%||100%||96%||IC|
|overall management||Risk management procedures should be revised to encourage people to revisit risks and responses much more often, to stay up to date, and to focus on things that are more specific and topical.||100%||96%||98%||RMP|
|overall management||More effort should be made to incorporate risk/uncertainty awareness into strategic decision making and not just routine clerical procedures.||100%||100%||100%||WIU|
|overall management||The risk management approach should also address the personal risk/uncertainty awareness, skills, and attitudes of staff, particularly managers at all levels.||92%||84%||89%||WIU|
|overall management||The way risk and uncertainty are quantified should be improved so that more numerical modelling and empirical support are used where appropriate.||77%||80%||83%||WIU|
Implications of results
Auditors in particular are relentlessly pushed to focus on a range of familiar internal control techniques and to make recommendations about risk management that are based on risk listing, despite its many shortcomings. Many 'operational' risk managers have a similar background and focus.
And yet, despite this, respondents were quite capable of seeing the value of other types of recommendation, including quite progressive ideas that do not rely on management-by-targets and approved documentation.
Although the four groups (Risk managers, Auditors, Performance managers, and Others) are too small for firm conclusions to be drawn, the differences do seem to meet reasonable expectations, in most cases. For example, the risk managers more often thought they were expected to make recommendations about risk management processes and other working in uncertainty ideas than the auditors did.
More importantly, this analysis shows a gap between what respondents recognized as valuable and what they thought they were expected to recommend. In particular, though WIU recommendations had 90% support from auditors, only 48% thought they were expected to make those sorts of recommendations.
Furthermore, WIU recommendations in particular seemed to be worth attention because they were about equally supported by respondents but much less likely to be in place already. Consequently, if you go hunting for opportunities to make these recommendations then you are more likely to find them.
It seems that if auditors and risk managers were encouraged to make more recommendations of the WIU type then they would be willing and able (provided they could think of the recommendations themselves rather than just recognizing suggestions as valuable), and would find more opportunities to do so.
|Group||Risk managers (n=13)||Auditors (n=25)||Performance managers (n=4)||Other (n=4)||All (n=46)|
Comments by respondents
The survey asked if the respondent had any other comments they would like to make about the survey or about recommendations. Excluding comments purely about the survey, the respondent comments were:
"Wow, these questions made my head hurt. Any time you wandered into recommendations regarding modelling and statistics, I backed off. I haven't thought about those matters since college Econometric courses some 20 years ago. As internal auditors are primarily accountants first, and operational observers secondly, we would not be expected by my current organization to make comments on statistical models or product marketing recommendations. Because of the detailed nature of these comments, I'm much more comfortable suggesting a topic for strategic direction than a specific management model. The point is to get management to clarify their aims and directions. Internal audit attempts to help them formalize this and then audit to their stated objectives. We are not subject matter experts. But we can comment on the effectiveness of processes based on outcomes."
"There is a lot of work to be done, especially in the government sector in incorporating risk management (information/operations risk). I would like to see more concrete work on how to integrate or rather align specific business goals/missions with the IT part of the business. To date, it still seems to me there is a big gap. The push from C-level executives to figure out ROI/ROSI (Return on Investment/Return on Security Investment) are still rather "fuzzy" numbers. How do we really get from here to a fully integrated risk management organization where risk is just business as usual and as much as possible, fully automated and dynamic?"
"The 'probably expected' box I interpreted to mean 'probably expected within my current role'. Working on a Sarbox project, I am planning to raise business issues but am not generally expected to do so. Some of the survey's suggested recommendations are also somewhat outside management's own expectations of a process or touching on areas that management are not expecting me to review - e.g. strategic marketing decisions and are not included on those grounds."
"Risk management itself is a risk process - the degree of risk evaluation/management depends on many factors, and the answers to the questions above, of necessity, cannot assess all the factors that would apply, especially between e.g. SMEs and multinationals."
"Quantification doesn't help if there is no suitable information to quantify - which is often the case."
"The oil case encourages the use of real options methodologies. However, virtually all the cases would benefit from this perspective."
"Much of what's 'recommended' above is (or should be) standard public sector practice, given the push to formal project management (It's all in PRINCE). Risk management (Per HM Treasury) and OGC Gateway reviews."
Respondents were invited to participate using professional discussion lists on the Internet and by some personal e-mails, but only where I was confident the person would not be predisposed to answer in a particular way. For example, if someone wrote to me about how they were interested in evolutionary project management I would not invite them to participate.
Most respondents were from the USA and UK. Most were internal auditors and risk managers.