Working In Uncertainty
Sarbanes-Oxley Act section 404 and 302: efficient compliance (updated)
by Matthew Leitch, first version 7 January 2003, updated 20 June 2003 and 18 May 2004.
Sections 302 and 404 say ‘effectiveness’
Thanks to Enron and other financial scandals the USA has enacted legislation that places some new and challenging requirements on virtually all companies registered with the SEC (i.e. with shares listed in the USA). Among these are some requirements on internal control in sections 302 and 404. These sections amount to a requirement for companies to evaluate the effectiveness of their internal controls over information reported to the financial markets. The SEC has issued rules to implement these statutory requirements and these will apply for financial year ends on or after 15 June 2004 (for most US companies) and on or after 15 April 2005 for others.
Although the word ‘effectiveness’ is used repeatedly in the Act and the SEC's rules many companies (and even the AICPA and PCAOB) have not fully realised the significance of this word and are approaching compliance in a needlessly expensive way.
Towards the end of 2002 I assisted in a project to evaluate the internal controls over financial reporting in a large, multi-national company. The approach prescribed was to document the controls in existence and consider any obvious weaknesses seen in the design or volunteered by interviewees. We could have gone on to test the operation of individual controls but at that stage it was not requested.
My team covered the relatively small UK operation. After two weeks we had done the required work, which did not include saying if the controls were effective or not. This is just as well because with this approach I could not say if the controls were effective or not! (Even if individual controls had been tested it would still have been very difficult.)
So how long should it have taken to assess the effectiveness of their controls? Surprisingly, the answer is not extra work. It should have taken about a day to get an initial view and a few more days to do the remaining documentation and testing.
This paper explains the best way to assess the effectiveness of internal controls. If your company has started or plans to approach s302 and s404 compliance purely by exhaustively documenting your controls and testing them individually stop and think again.
Easier ways to assess the effectiveness of internal controls
Suppose you have to evaluate whether an accounting department is under control or not. Here are two approaches for your evaluation. Which do you prefer?:
As an experienced auditor who has tried both I can tell you that the second option is by far the best, and the ideal approach would be a blend of the two. Why?
The first evaluation is theoretical. If we correctly assess the risks, and if we correctly assess the effectiveness of each individual control, and if we can combine this information accurately, then our conclusions about overall effectiveness of controls will be correct. In practice such accuracy is impossible and theoretical assessment is unreliable.
Theoretical assessment is also time consuming. Gathering and documenting the information takes many interviews. The risk-control mapping stage requires skill that few people have so there are many iterations. The final weighing of apparent weaknesses involves much discussion but in truth it is beyond human judgment to evaluate accurately the complex probabilities involved.
Benchmarking or relying on ‘best practice’ are not solutions to this problem. The differences in error and fraud rates between organisations with different people, systems, procedures, etc are so large that ‘standard’ or ‘best practice’ control schemes cannot be considered reliable. They always need to be adapted to fit the requirements correctly.
In contrast, the second type of evaluation involves looking at direct evidence of effectiveness. It is not necessary to analyse the controls or risks fully. Just go straight to the results of the controls instead of trying to guess them from what went before. It's like a doctor taking a patient's temperature.
This diagram also introduces the second key technique in evaluating effectiveness, which is to monitor the very first stage. These are the drivers of risk and control requirements. By looking at simple statistics and news of these drivers it is easy to identify when controls might need attention, either to strengthen them, or to remove costly controls that are no longer needed. The next section explains how to put these techniques together in an efficient process for evaluating effectiveness of controls.
A process for evaluating effectiveness
In practice it is not appropriate to look for direct indicators of controls effectiveness for all processes and all risks, so some initial decisions have to be made about what assessment techniques to use where. Also, gaps in the indicators have to be identified and compensated for. Once the assessment has been done there may be a need for control improvements, and repeated assessments can be better focused if drivers are used. All these points are reflected in the following process for evaluating controls effectiveness.
STEP 1: Draw up an integrated annual cycle for assessment activities to meet all requirements
For most organisations in most countries the requirements for various evaluations of internal controls are so numerous that only an integrated process makes sense. For example, as far as possible, evidence from s302 assessments should feed into s404 assessments.
STEP 2: Identify processes whose controls need evaluation
The objective of this step is to list the processes, not to describe them, which would require much more work. This is simply a list for planning purposes. All the usual considerations about materiality, locations, risk, etc apply when you decide what to cover and what to leave out.
STEP 3: Identify risk and control requirement drivers
The need for controls, and constraints on what types of control are economic and culturally appropriate, is driven by a number of factors for each process. These shape the control system, and when they change so too should the control system. Consider potential drivers under the following headings:
This may seem like a long list but almost all this information is common knowledge in companies and so is easy to research/gather data on. Besides, for many of the headings there will be nothing interesting.
STEP 4: Collect and monitor driver data/news
It is not necessary to complete this monitoring before carrying out the first control evaluations, but once drivers are being monitored it is possible to target controls evaluation and make it still more efficient. Variations in any of the drivers listed in the previous step have implications for the controls required.
Another reason for doing this monitoring is that it is explicitly required by the Sarbanes-Oxley Act in section 302, which says you must comment on any factors that might have affected the effectiveness of your control system since the last evaluation.
STEP 5: Decide what type of evaluation to use for each process and type of risk
The type of evaluation depends on the nature of the process and the type of risk:
This avoids extensive theoretical analysis of controls over the big accounting cycles, and that's how the time is saved.
It is essential to consider the coverage of the indicators used. If there are significant gaps they will need to be compensated for by mapping and testing controls. The one thing that can never been shown in statistics is the undiscovered error, which of course is the risk we are concerned with. The judgment of controls effectiveness is based on the principle that undiscovered errors are more likely where:
STEP 6: Perform evaluations
A well controlled business process or accounting cycle will have a process monitoring report which is used frequently by the process owners to manage the health of the process. This report will show workload and resources used, plus error and backlog statistics, and system support, preferably using graphs to show clearly what is going on. There will also be a section on projected future changes so that risks can be managed in advance.
If this kind of report already exists for a process then evaluating the effectiveness of controls is going on whenever the report is used and meeting Sarbanes-Oxley requirements for the process is easy. Extra work is only needed for the rare risks of major fraud and disasters.
If this kind of report does not exist and the process is a major one then a process monitoring report should be implemented immediately! Now that the SEC have given companies more time to comply it should be possible to get these reports in place for more processes.
In areas where risks and controls have to be mapped because of a lack of direct indicators it is possible to waste a lot of time by choosing the wrong style of matrix. For details on how to do this correctly I offer my paper on control matrices ‘The easiest and best matrices for documenting internal controls’.
STEP 7: Identify the causes of weaknesses
Where your evaluation is a theoretical one based on the design of controls and their individual operation then the location of the weakness is obvious. Either it's a design fault, or failed operation, or a combination of both.
However, if problems have shown up in process health statistics extra work is needed to find out what the weakness is and whether it is design or operation. This is needed if you are to fix the fault, and also appears to be required by the new SEC rules.
Meeting the regulations and working with external auditors
The relevant regulations and other official documents are briefly explained in the appendix below. If you read the new SEC rules and the PCAOB's requirements of external auditors on how to do the section 404 review it looks as if neither body has thought of using process health indicators as part of the assurance on controls effectiveness.
If you want to save effort and reach more reliable conclusions by including direct evidence in your assurance mix you will need to persuade your external auditors that this is acceptable. Here are some suggestions on how to do it.
First, understand the psychology of the external auditor. Not all auditors, even in the top firms, have a theoretical understanding of this kind of evidence. Their theoretical knowledge usually has not kept up with their own practices.
If you describe your approach to them in theoretical, hypothetical terms they may be reluctant. Also, if you ask them what they are expecting or intend to do they will almost certainty talk about documenting controls, evaluating the design, and testing individual controls to see if they have operated. There will be no mention of health stats and they may even forget to mention evaluating risk.
However, external auditors are intelligent human beings and presented with clear examples of evidence of process health used in conjunction with the sort of documentation and testing they are expecting I think they are more likely to see the value of the combined evidence. Just about all auditors think a lot about risk and many use indicators of inherent risk as part of their audit evidence. Auditors of very large organisations often make use of process health statistics even though their theoretical understanding has not been revised to reflect this.
Research conducted recently shows that, in fact, internal and external auditors recognise the value of process health indicators and find them more useful as evidence of controls effectiveness than most other types of evidence. The research is described in ‘Evidence for an efficient approach to evaluating controls effectiveness’.
If you do get into theoretical discussions and objections start to surface, here are some points that may help:
The last point should prove compelling, which shows the value of understanding the psychology of external auditors.
Evaluating the effectiveness of internal controls is something more and more companies are expected to do. Every year, countless people waste countless hours doing it in inefficient and inaccurate ways. This paper explains a way to do the work more easily, and yet also produce a more useful and accurate result.
Appendix: The regulations and some powerful implications
The Sarbanes-Oxley Action of 2002 became law on 30 June 2002 and contains a wide range of rules designed to reduce the risk of an Enron-style corporate scandal happening again to a company listed in the USA regardless of where the company is actually located. The two sections with internal control requirements are sections 302 and 404. Section 103 is also relevant to the objectives of the external auditor. These sections require the SEC to make rules so that companies have to evaluate the effectiveness of certain classes of controls at certain times and publish their conclusion. Section 404 requires external auditors to attest to the conclusion reached by the directors.
These new requirements are a revolution in internal control reporting as explained in my paper ‘The crisis in management control and corporate governance (questionnaire).’
The regulations say that companies should evaluate their controls against a recognised framework and the outstanding candidate for this in many countries will be the ‘COSO framework’. The executive summary of the original framework in freely available on the web at www.coso.org. However, a new framework is being drafted and should be issued some time in 2003.
The PCAOB has issued an official document setting out requirements for 404 reviews and these are currently available for comment and consultation on the SEC's website here.
The new rules on s404 compliance issued by the SEC and proposed PCAOB requirements contain some points with potentially explosive implications:
Words © 2003 Matthew Leitch. First published 7 January 2003.