Working In Uncertainty

Results of a survey on the locations of uncertainty

Introduction

Many thanks to all who participated in this survey. Without your donations of time and thought this would not have been possible.

The results of this survey indicate an opportunity to expand guidance on risk management so that it helps people deal with uncertainty in all aspects of decision-making and similar management activities, rather than focusing just on uncertainty around predictions of the future. Remarkably, dealing with uncertainty around predictions of the future was rated the least crucial of the five areas, though still important.

The survey results

The survey was completed by 68 people. Questions 1 - 5 (the scenario questions) were based on one of two scenarios. Each respondent saw just one scenario, which was chosen randomly by a script on the web page.

There were three reasons for showing two scenarios:

• Thinking about at least one realistic situation was helpful preparation for questions 6 - 10, which were questions of general principle.

• Using two different scenarios would reveal at least a little of the differences in uncertainty levels between situations.

• Comparing the in-principle answers of people who faced different scenarios would show whether their in-principle generalizations were unhelpfully biased by the scenario they had just considered. If in-principle answers varied systematically with answers on the scenarios then that would have indicated bias.

Here are the main questions that were asked, with percentages of respondents choosing each available answer:

"Do you consider yourself to be a professional in any of these? (click all that apply)"

[40 people answered questions on the following Marketing Scenario.]

"Imagine you are a very important person in a big company and you're just starting to think about launching a new range of products into a new market."

"How uncertain would you expect to be at this early stage as to:

[28 people answered questions on the following Administrative Scenario.]

"Imagine you are an administration manager in a medium sized company and you're just starting to think about replacing the three photocopy machines in your office."

"How uncertain would you expect to be at this early stage as to:

"Typically, for managers of all kinds in all sectors, how important is it to be able to deal with uncertainty in each of the following areas:

To what extent did you think of specific risks or uncertainties to answer the above questions?

In general, the Marketing Scenario was considered to involve a little more uncertainty than the Administrative Scenario, especially in relation to the current situation and the relevant interests (questions 1 and 2). However, there was no discernible difference in answers to questions 6 - 10 between respondents who had faced different scenarios. This indicated that the scenario questions (1 - 5) had not had a distorting effect on answers to the 'in principle' questions (6 - 10).

Implications

The key finding is that 67 out of 68 respondents thought that "typically, for managers of all kinds in all sectors" being "able to deal with uncertainty" was at least Helpful in each of the five areas of decision-making covered in the survey. Indeed, on each of the five areas over 79% of respondents thought it was Important or Crucial.

This contrasts with the extent of guidance given in standards for risk management such as ISO 31000:2009 and BS 31100:2011, which are leading examples. In these guides only uncertainty around predictions of the future is considered explicitly. (This is the focus for risk assessment.) For ascertaining the current situation and interests of stakeholders the advice is simply to communicate and 'establish' some answers, with no explicit advice on what to do about uncertainty remaining once this has been attempted. There is no guidance at all on choosing which courses of action to consider or on deciding when to stop looking for courses of action and just make a choice, let alone detailed guidance on dealing with uncertainty in these areas.

Although the importance of being able to deal with uncertainty in each of the five areas was judged about equally high, there were differences. The highest rated area was deciding when to stop looking for courses of action (on which, typically, there is no guidance), while the lowest rated area was predictions about the future (which gets all the detailed guidance on uncertainty in current standards). Here's a summary, sorted in descending order of rated importance:

Comments by respondents

A number of respondents made comments that explained or added to answers given. The key parts of comments were as follows:

"I think the starting point is being aware of uncertainty. You cannot easily deal with something for which you have little awareness. So is it good or bad to be uncertain? I think it is good to be certain when it is appropriate e.g. in a fairly simple situation where the knowns are known and there are no unknown unknowns. But it is bad to be certain where you have no idea how much you do not know."

"Detailed planning is over-rated. Pre-positioning resources to increase robustness works much better. (Doug Samuelson)"

"If holding a high position as you've described initially, one should be able to have a high degree of confidence that the answers one is receiving from subordinates have a great deal of integrity (accuracy & truthfulness). At that point one can better manage risks with a high tempo of activity or confidence. Greece's misleading behaviour leading up to the Euro crisis is a good example of the opposite; a lack of integrity in reporting going up an 'organisation'."

"There would be no point in almost any endeavour of life if there was no uncertainty. Managing uncertainty should be introduced at school level social studies."

"Risks have become an inherent cause of concern for managers at each level of decision-making. Those risks can be internal (finance constraints, capacity limitation, labour turnover, marketing failure, etc) and external to the organisation (currency fluctuations, political instability, etc). Managers therefore need to blend all these elements of risks to the extent that the risks are not only minimised in their impact but also in their occurrences."

"It's important to start from the place that everything has some uncertainty to it and that the scale and range is then considered and to retain curiosity to investigate and use the uncertainty to achieve opportunities. This influenced my response to the survey."

"A brief idea: An organization should determine what it exists to achieve and, taking account of its context, establish its required outcomes, and what needs to be done and employed to achieve them. Successive levels of the existing organisational structure should be invited to say what they can contribute to the higher level outcomes. Governance should use measures and the relationships between outcomes, activities and resources to predict and enable outcomes and prevent problems. Control information should be comprehensive and immediately visible to all stakeholders through a single integrated system. In this way, the outcome ↔ risk ↔ control causal relationship can be agile and transparent, constantly monitored and delivery predicted."

"The process of enumerating options and evaluating them forward against favorable and unfavorable scenarios is very valuable - the wargaming heuristic. This goes well beyond Peter Schwartz's 'Art of the Long View.'" (Russ Vane)

"Regarding Q10 [about stopping], I think it was Derrida who called practical decisions 'acts of madness'... suggesting that the precise moment of a practically rational decision is incalculable. Hence my key thought, when answering that question, was that the uncertainty present in the moment of a practical decision, which moves from planning to action, has to be 'tolerated' rather than 'dealt with'." (Dr Alasdair Marshall)

Finally, one respondent generously sent his comments by email in order to provide a remarkably complete and insightful account of his thinking on every question. This respondent had been given the Administrative Scenario:

"Q1 [about the situation]. As an admin manager I would expect to be somewhat familiar with performance (complaints etc). Perhaps somewhat less about needs - perhaps I have forgotten the general needs or specialist needs are being met by workarounds. Somewhat Uncertain seems a natural response. (As a user I've never been consulted about purchases of photocopiers/printers, and hardly ever about software on PCs.)"

"Q2 [about interests]. I would have some generic understanding of the objectives of the purchase process - this should be part of the day job, although big hardware purchases are not day-to-day. Hence not Highly Uncertain."

"Q3 [about initial alternatives]. Technology moves on and the relative advantages of different providers also; in a home situation I was shocked 3 years ago by my experience of replacing my efficient Brother Laser printer (reliable, cheap, black and white - found as a result of talking to a friend who had a very similar model) with a more modern Brother update. Bigger, noisier, slower to get going, and needing much more expensive consumables (found as a result of surfing the net)."

"On the next set of questions I become a bit more confused; I interpreted the 'initial work' as not having got detailed responses from suppliers, nor having obtained details on what types of purchase arrangement were available e.g. whether a 1 month trial was possible. Obtaining these details would have helped a lot. While it might have left some uncertainty over the suppliers' claims the impact could have been managed, especially if I could trial just one replacement."

"Q4 [about predictions for the future]. Without being able to ask specific questions of suppliers I could see an emphasis on cost, with reliability and time savings being much more subjective."

"Q5 [about stopping and choosing]. I was still more confused about this. Part of me thought that the start-stop-abandon approach might be largely 'settable' before doing too much research, not least because work sucks you into more work, research, work and ... procrastination. That's why I didn't want to answer."

"On 6-10 the questions were a bit more generic. I felt somewhat more comfortable, but not altogether so. I was struck by how few of the aspects of uncertainty get considered in my day-to-day work (by me and others). Too much would definitely be labelled introspective (to put it politely). I think many would regard 6-10 as 'common sense' or that you just 'take a view' etc."

"Q6 [about the situation]. Events and the current situation seem Crucial. 'Causes' as we've discussed before seem a bit more problematic particularly if you cannot observe directly and the 'incident' had its root outside the company."

"Q7 [about interests]. Seems obviously Important, but there also seemed a sense in which, presented with the right information, a stakeholder could decide or participate in a discussion rather than just accept my recommendation. Any lack of certainty on my part would not be disastrous."

"Q8 [about initial alternatives]. This seems very Helpful indeed. But in the absence of this people will reach a pragmatic decision - so long as there is a deadline. One of the problems I face in completing some of my 'non-office' work is that I don't have such a deadline."

"Q9 [about predictions for the future]. Again my lack of comfort surfaced. The richness of risk means that I generally find it hard to say things are not important (but this might be me saying they are interesting rather than important)."

"Q10 [about stopping and choosing]. Similar feelings as 8. I largely expect that time constraints will be a given. It seems important to have some sort of decision framework. Perhaps some of my lack of comfort is that all-too-often I see little of this!"

Respondents

Invitations to participate in the survey were sent to RISKANAL (an internet discussion list about risk analysis), the PMA Forum (an internet discussion list about performance management), the Auditnet discussion list (for auditors), and a varied selection of my contacts. 79% of respondents stated their first language as some kind of English, with the rest having a variety of other first languages. Total response rate cannot be calculated but will have been low.

Clearly this is a limitation of the survey and the selection of respondents will have affected the pattern of answers given to some extent. However, two points suggest that this was not a significant problem:

1. The overall pattern for questions 6 - 10 is a very strong one, with all areas rated highly for importance.

2. When different sub-groups were compared there seemed to be no major differences in their views. The sub-groups compared were:

   • the roles (project management, project risk management, risk management, and audit);
   • risk role vs not a risk role;
   • not specialized role vs some specialised role; and
   • English as first language vs not.