Working In Uncertainty

Evidence for an efficient approach to evaluating controls effectiveness

by Matthew Leitch, first published 27 April 2004.

Thank you

First, thank you to everyone who responded to this survey. The results are generally clear cut and important for most people involved in audit work, their employers, and regulators. One thing I personally found interesting about the results was how sensitive auditors are to potentially relevant evidence.

Summary

The evidence most often mentioned in regulations on internal controls – and almost the exclusive concern of the PCAOB's requirements for SOX 404 compliance – is of individual tested controls. However, in real audit work other information is relevant and by casting the net wider it is possible to conduct a much more efficient review. This is because it is possible to cream off the most persuasive and easily gathered evidence in each category rather than digging down into expensive but individually unimportant details.

If you are interested in exploring this strategy two questions may have occurred to you: (1) ‘Do other auditors accept evidence other than individually tested controls?’ and (2) ‘Will our external auditors accept other evidence?’ This survey set out to find what kinds of evidence auditors generally accept as relevant to evaluating internal controls effectiveness. The results give strong support to use of the strategy.

The survey was conducted online during April 2004 and respondents were self-selected, though the thinking underlying the survey was not explained and comments from respondents suggested they were puzzled by what the survey was doing. Respondents were presented with 20 statements and asked how relevant each was, as evidence, in evaluating the effectiveness of a system of internal controls over financial reporting. Respondents could choose between ‘Relevant’, ‘Somewhat relevant’, and ‘Not relevant’.

The statements were of 4 main types: (1) ‘distractors’ intended to have no relevance, (2) individual controls tested, (3) inherent risk factors, and (4) process health measures. The tested controls were further divided into application controls, IT controls, and ‘tone at the top’.

The most relevant statements concerned process health measures – a dramatic finding as these are rarely mentioned in auditing theory – closely followed by tested controls and inherent risk factors. Many of the statements designed to be Not Relevant were considered Relevant or Somewhat Relevant by at least some auditors and this is almost certainly because they suggested inherent risk factors.

In short, all the auditors in this survey considered process health measures to have relevance and over 95% considered inherent risk factors to have relevance as evidence. This provides confidence that a strategy that uses them will be persuasive for internal and external auditors. Such a strategy is explained in ‘Sarbanes-Oxley Act section 404 and 302: efficient compliance’.

Relevance

Out of 46 respondents, just two considered all inherent risk factors to be irrelevant as evidence.

All respondents considered process health measures, such as error rates, to be relevant. Indeed, process health measures were, arguably, more relevant for respondents than even clear information about individual controls tested. This is perhaps because they show the end result of control, whereas evidence that individual controls have operated does not demonstrate that the system as a whole is working well.

The full results are shown on this graph and the following table. On the graph the 20 statements are numbered along the bottom and the abbreviations indicate the type of statement: irf = Inherent Risk Factor, phm = Process Health Measure, dis = Distractor, con-env = Control tested – environment (‘tone at the top’), con-appn = Control tested – application control, con-IT = Control tested – IT control:

In addition to the main effects mentioned above, there are some interesting smaller effects.

• The unremarkable inherent risk item: Item 3 was an inherent risk factor but more respondents rated it as only Somewhat Relevant than for the other two. The text of the item was ‘There have been no changes to finance staff or systems during the year.’ Had it said that the staff and systems had changed a lot more respondents would have considered the relevance clear. As it was, the fact that this information removed the possibility that the staff and systems had changed was enough for most respondents to see at least some relevance.

• Relevant distractors: The ‘distractors’ were statements designed to be irrelevant as evidence. The example that some respondents found amusing or puzzling was the statement that the finance director had brown eyes. However, it seems it is very difficult to think of information about an organisation that is completely irrelevant because so much can indicate inherent risk. Items 7, 9, 12, and 14 seemed to be Relevant or Somewhat Relevant to many respondents. The one thing that slightly concerns me is that so many people saw relevance in the statement that ‘The external auditors have merged with a rival firm."

• Apparent control environment not so persuasive: Of the statements of individual controls the least persuasive were clearly those concerning the control environment. When Chiefs say they consider reliable financial reporting to be fundamental we are not hugely impressed.

Internal versus external views

28 of the 46 respondents had at least some external audit experience, but only 10 had spent more time as external auditors than as internal auditors, and only 4 respondents were purely external auditors. This is unsurprising as respondents were obtained mainly by e-mailing subscribers to the AuditNet website, a popular resource for internal auditors.

The four pure external auditors all considered inherent risk factors to be relevant, as did the other six who had more external than internal audit experience.

Aggregating the results of people with more external than internal audit experience, and comparing them with the other respondents shows that there is little difference between them.

Does experience count?

Years of audit experience did not seem to make much difference. Respondents were divided into a tiny group with no audit experience and two large groups, one for people with more than 10 years of any kind of audit experience and one for others. If anything, the respondents with less than 10 years of experience tended by give answers closer to my intention when devising the questionnaire.

Comments by respondents

The survey asked if the respondent had any other comments they would like to make about audit evidence. Most respondents made no comment and several made light hearted comments about the survey, especially the item about eye colour. There was just one serious comment, throwing light on the lower value usually placed on evidence about the attitudes of senior executives.

‘Part of effective internal controls includes good written policies and procedures. However, these are not to be taken in isolation of all other factors. If policies and procedures are circumvented they are of little or no value for being an effective part of I/C. Interviews of persons who do the work to determine just how the work is done will give insight as to employee awareness and the effectiveness of the written policies and procedures.’

Respondent profile

Slightly over half the respondents were from the United States of America. The countries of respondents are shown on this graph:

Most were internal auditors, with predominantly internal audit experience.

Instructions used in the survey

Critical instructions used in the survey were contained in the following paragraphs. Firstly, there was only a vague suggestion as to the purpose and rationale of the survey. This was to reduce the risk of biased responses from people eager to support a more efficient style of SOX 404 compliance. The introductory words were:

‘We need to find out more about what people consider relevant to deciding how effective a system of internal controls is. The range of evidence considered has a big impact on the cost of auditing controls effectiveness so there are immediate practical implications.’

‘For this survey to have any value it is vital that you answer honestly. Your answers will be confidential. Your employer, if you have one, will never know what you answered.’

The twenty statements were listed in a random order (using a random number generator seeded by the time of day in milliseconds) and the instructions were as follows:

‘The following 20 items are things that might be relevant to your evaluation of the effectiveness of a system of internal controls specifically for financial reporting. These statements aren't about the same organisation, so take each one in isolation.’

‘Consider each and decide how relevant it is, if at all, as evidence relating to the current effectiveness of controls over financial reporting, in your opinion. Don't worry about complying with any particular piece of official guidance or regulations. What do you think? What could influence your view?’

Further information

If you would like to analyse the original data yourself I can provide a matrix of the ratings given. The information will not allow you to identify respondents or their organisations.

Copies of the original survey are also available. Please contact me at matthew@workinginuncertainty.co.uk.

Be notified of new publications
and get access to more materials -
register your interest in WIU

Hundreds of people receive notification of new publications every month. They include company directors, heads of finance, of internal audit, of risk management, and of internal control, professors, and other influential authors and researchers.

Made in England