Working In Uncertainty

Sarbanes-Oxley Act section 404 and 302: efficient compliance (updated)

by Matthew Leitch, first version 7 January 2003, updated 20 June 2003 and 18 May 2004.

Sections 302 and 404 say ‘effectiveness’

Thanks to Enron and other financial scandals the USA has enacted legislation that places some new and challenging requirements on virtually all companies registered with the SEC (i.e. with shares listed in the USA). Among these are some requirements on internal control in sections 302 and 404. These sections amount to a requirement for companies to evaluate the effectiveness of their internal controls over information reported to the financial markets. The SEC has issued rules to implement these statutory requirements and these will apply for financial year ends on or after 15 June 2004 (for most US companies) and on or after 15 April 2005 for others.

Although the word ‘effectiveness’ is used repeatedly in the Act and the SEC's rules many companies (and even the AICPA and PCAOB) have not fully realised the significance of this word and are approaching compliance in a needlessly expensive way.

Towards the end of 2002 I assisted in a project to evaluate the internal controls over financial reporting in a large, multi-national company. The approach prescribed was to document the controls in existence and consider any obvious weaknesses seen in the design or volunteered by interviewees. We could have gone on to test the operation of individual controls but at that stage it was not requested.

My team covered the relatively small UK operation. After two weeks we had done the required work, which did not include saying if the controls were effective or not. This is just as well because with this approach I could not say if the controls were effective or not! (Even if individual controls had been tested it would still have been very difficult.)

So how long should it have taken to assess the effectiveness of their controls? Surprisingly, the answer is not extra work. It should have taken about a day to get an initial view and a few more days to do the remaining documentation and testing.

This paper explains the best way to assess the effectiveness of internal controls. If your company has started or plans to approach s302 and s404 compliance purely by exhaustively documenting your controls and testing them individually stop and think again.

Easier ways to assess the effectiveness of internal controls

Suppose you have to evaluate whether an accounting department is under control or not. Here are two approaches for your evaluation. Which do you prefer?:

1. Document the procedures and controls then map risks to controls to see if there appear to be weaknesses in the design. Then test the controls individually to establish if each is actually operating and being carried out correctly.

2. Take a look at the actual results of the controls, such as current backlogs of processing, suspense item levels, and errors found and corrected. Interview the Controller to see if he/she actually knows about these.

As an experienced auditor who has tried both I can tell you that the second option is by far the best, and the ideal approach would be a blend of the two. Why?

The first evaluation is theoretical. If we correctly assess the risks, and if we correctly assess the effectiveness of each individual control, and if we can combine this information accurately, then our conclusions about overall effectiveness of controls will be correct. In practice such accuracy is impossible and theoretical assessment is unreliable.

Theoretical assessment is also time consuming. Gathering and documenting the information takes many interviews. The risk-control mapping stage requires skill that few people have so there are many iterations. The final weighing of apparent weaknesses involves much discussion but in truth it is beyond human judgment to evaluate accurately the complex probabilities involved.

Benchmarking or relying on ‘best practice’ are not solutions to this problem. The differences in error and fraud rates between organisations with different people, systems, procedures, etc are so large that ‘standard’ or ‘best practice’ control schemes cannot be considered reliable. They always need to be adapted to fit the requirements correctly.

In contrast, the second type of evaluation involves looking at direct evidence of effectiveness. It is not necessary to analyse the controls or risks fully. Just go straight to the results of the controls instead of trying to guess them from what went before. It's like a doctor taking a patient's temperature.

This diagram also introduces the second key technique in evaluating effectiveness, which is to monitor the very first stage. These are the drivers of risk and control requirements. By looking at simple statistics and news of these drivers it is easy to identify when controls might need attention, either to strengthen them, or to remove costly controls that are no longer needed. The next section explains how to put these techniques together in an efficient process for evaluating effectiveness of controls.

A process for evaluating effectiveness

In practice it is not appropriate to look for direct indicators of controls effectiveness for all processes and all risks, so some initial decisions have to be made about what assessment techniques to use where. Also, gaps in the indicators have to be identified and compensated for. Once the assessment has been done there may be a need for control improvements, and repeated assessments can be better focused if drivers are used. All these points are reflected in the following process for evaluating controls effectiveness.

STEP 1: Draw up an integrated annual cycle for assessment activities to meet all requirements

For most organisations in most countries the requirements for various evaluations of internal controls are so numerous that only an integrated process makes sense. For example, as far as possible, evidence from s302 assessments should feed into s404 assessments.

STEP 2: Identify processes whose controls need evaluation

The objective of this step is to list the processes, not to describe them, which would require much more work. This is simply a list for planning purposes. All the usual considerations about materiality, locations, risk, etc apply when you decide what to cover and what to leave out.

STEP 3: Identify risk and control requirement drivers

The need for controls, and constraints on what types of control are economic and culturally appropriate, is driven by a number of factors for each process. These shape the control system, and when they change so too should the control system. Consider potential drivers under the following headings:

1. Control performance requirements
• Speed of processing required.
• Flexibility of processing required.
• Maximum tolerable level of hassle to the customer.
• Precision of timing required.
• Reliability of service to the customer required.
• Target cost of processing required.
• Level of regulation of activity (e.g. selling financial services).
2. Cultural features
• Culture/behavioural norms encouraging fraud/theft. Patterns of crime already established.
• Company wishes to promote empowerment.
• Process management vs functional silos.
• Standard of the control environment.
3. Data features
• Data is standing data or transaction data.
• Complexity of the data.
• Volumes of data.
• Predictability of data values.
• Whether or not transactions can be divided into sub-populations which are highly predictable or at least have very common characteristics.
• Maximum value of individual items.
• Whether data about private individuals is held.
• The extent to which it is a very abstract business based on rules, definitions, possibilities.
4. Process features
• Complexity of the process.
• Who captures the data e.g. employees, customers, suppliers.
• Level of automation.
• Ease with which assets can be disposed of if stolen.
• Amount of money paid out.
• Number of languages spoken by people in the process.
• International/geographic distribution.
• Number of separate databases and interfaces.
• Quality of existing business process controls.
• Whether the immediate environment of the process is within the organisation.
5. Workload features
• Rate of increase/decrease in workload.
• Variability of workload.
• Continuous work is required vs periodic work only vs slow response only is required.
• Environment is very fast changing or very stable.
• Level of change in processes, systems, or people.
• Proportion of work in the process that is controls.
6. Project features (i.e. implementation of the process/system)
• Project health (e.g. wobbly sponsorship, politics, unclear or shifting requirements, over-ambitious objectives and impossible timetables).

This may seem like a long list but almost all this information is common knowledge in companies and so is easy to research/gather data on. Besides, for many of the headings there will be nothing interesting.

STEP 4: Collect and monitor driver data/news

It is not necessary to complete this monitoring before carrying out the first control evaluations, but once drivers are being monitored it is possible to target controls evaluation and make it still more efficient. Variations in any of the drivers listed in the previous step have implications for the controls required.

Another reason for doing this monitoring is that it is explicitly required by the Sarbanes-Oxley Act in section 302, which says you must comment on any factors that might have affected the effectiveness of your control system since the last evaluation.

STEP 5: Decide what type of evaluation to use for each process and type of risk

The type of evaluation depends on the nature of the process and the type of risk:

This avoids extensive theoretical analysis of controls over the big accounting cycles, and that's how the time is saved.

It is essential to consider the coverage of the indicators used. If there are significant gaps they will need to be compensated for by mapping and testing controls. The one thing that can never been shown in statistics is the undiscovered error, which of course is the risk we are concerned with. The judgment of controls effectiveness is based on the principle that undiscovered errors are more likely where:

• not enough checking is done; and/or

• checking is done and reveals a high rate of original error and/or extensive backlogs.

STEP 6: Perform evaluations

A well controlled business process or accounting cycle will have a process monitoring report which is used frequently by the process owners to manage the health of the process. This report will show workload and resources used, plus error and backlog statistics, and system support, preferably using graphs to show clearly what is going on. There will also be a section on projected future changes so that risks can be managed in advance.

If this kind of report already exists for a process then evaluating the effectiveness of controls is going on whenever the report is used and meeting Sarbanes-Oxley requirements for the process is easy. Extra work is only needed for the rare risks of major fraud and disasters.

If this kind of report does not exist and the process is a major one then a process monitoring report should be implemented immediately! Now that the SEC have given companies more time to comply it should be possible to get these reports in place for more processes.

In areas where risks and controls have to be mapped because of a lack of direct indicators it is possible to waste a lot of time by choosing the wrong style of matrix. For details on how to do this correctly I offer my paper on control matrices ‘The easiest and best matrices for documenting internal controls’.

STEP 7: Identify the causes of weaknesses

Where your evaluation is a theoretical one based on the design of controls and their individual operation then the location of the weakness is obvious. Either it's a design fault, or failed operation, or a combination of both.

However, if problems have shown up in process health statistics extra work is needed to find out what the weakness is and whether it is design or operation. This is needed if you are to fix the fault, and also appears to be required by the new SEC rules.

Meeting the regulations and working with external auditors

The relevant regulations and other official documents are briefly explained in the appendix below. If you read the new SEC rules and the PCAOB's requirements of external auditors on how to do the section 404 review it looks as if neither body has thought of using process health indicators as part of the assurance on controls effectiveness.

If you want to save effort and reach more reliable conclusions by including direct evidence in your assurance mix you will need to persuade your external auditors that this is acceptable. Here are some suggestions on how to do it.

First, understand the psychology of the external auditor. Not all auditors, even in the top firms, have a theoretical understanding of this kind of evidence. Their theoretical knowledge usually has not kept up with their own practices.

If you describe your approach to them in theoretical, hypothetical terms they may be reluctant. Also, if you ask them what they are expecting or intend to do they will almost certainty talk about documenting controls, evaluating the design, and testing individual controls to see if they have operated. There will be no mention of health stats and they may even forget to mention evaluating risk.

However, external auditors are intelligent human beings and presented with clear examples of evidence of process health used in conjunction with the sort of documentation and testing they are expecting I think they are more likely to see the value of the combined evidence. Just about all auditors think a lot about risk and many use indicators of inherent risk as part of their audit evidence. Auditors of very large organisations often make use of process health statistics even though their theoretical understanding has not been revised to reflect this.

Research conducted recently shows that, in fact, internal and external auditors recognise the value of process health indicators and find them more useful as evidence of controls effectiveness than most other types of evidence. The research is described in ‘Evidence for an efficient approach to evaluating controls effectiveness’.

If you do get into theoretical discussions and objections start to surface, here are some points that may help:

• The SEC rules and PCAOB requirements neither encourage nor rule out use of direct controls effectiveness statistics. It simply is not mentioned at all. They describe the approach required by saying that documentation should ‘include’ certain things. That doesn't stop you including other things as well.

• Most companies will probably choose to say their evaluation is based on the ‘COSO framework’ and within this the ‘information and communication’, ‘monitoring’ and ‘risk assessment’ elements of a well designed control system should contain process health stats.

• The AICPA guidance mentions that if financial statements are found to be materially mis-stated that is evidence that a material deficiency in controls exists. It also points out that if the financial statements are not mis-stated that does not mean there are no material deficiencies. This acknowledgment that the results of controls are important is helpful.

• Your approach recognises the limitations of process health statistics as a source of assurance on controls effectiveness. The main limitations are: (1) It is not applicable to all processes. (2) It is not very helpful for risks that crystallise only rarely. (3) Stats do not show undetected errors, which are the ones that matter. (4) Clean statistics only carry weight if you trust the controls and processes that generate them. (5) The coverage of the statistics determines their value. (6) Statistics can be fabricated.

• Your approach also recognises the limitations of just evaluating controls in theory and testing their operation individually. These limitations are: (1) It is not possible to estimate risks accurately. (2) It is very difficult to assess the design of controls exactly, even if you know the risks (which you do not). (3) It is very difficult to accurately estimate the reliability of individual controls when the error rate that could be material is very low, because this requires larger sample sizes than auditors typically use. (4) Evidence of controls operating can be fabricated.

• Processes and controls have to be documented, but that does not mean all processes or all controls. Not only are some processes or locations immaterial, but it would be absurd to document every single control that was relevant. That would imply noting, for example, every edit check on every input screen of every computer system used. The aim of the regulations seems to be to get documentation that gives a good overall view of processes and controls and states the controls considered when reaching a conclusion about effectiveness. Using a mix of the most powerful evidence means that fewer controls need to be considered when reaching an overall conclusion and hence the documentation of controls is less.

• Your approach is more efficient as well as more reliable. You have a limited budget for Sarbanes-Oxley compliance and the more documentation your auditors expect you to produce the less is available for their audit fee.

The last point should prove compelling, which shows the value of understanding the psychology of external auditors.

Conclusion

Evaluating the effectiveness of internal controls is something more and more companies are expected to do. Every year, countless people waste countless hours doing it in inefficient and inaccurate ways. This paper explains a way to do the work more easily, and yet also produce a more useful and accurate result.

Appendix: The regulations and some powerful implications

The Sarbanes-Oxley Action of 2002 became law on 30 June 2002 and contains a wide range of rules designed to reduce the risk of an Enron-style corporate scandal happening again to a company listed in the USA regardless of where the company is actually located. The two sections with internal control requirements are sections 302 and 404. Section 103 is also relevant to the objectives of the external auditor. These sections require the SEC to make rules so that companies have to evaluate the effectiveness of certain classes of controls at certain times and publish their conclusion. Section 404 requires external auditors to attest to the conclusion reached by the directors.

These new requirements are a revolution in internal control reporting as explained in my paper ‘The crisis in management control and corporate governance (questionnaire).’

The Act is available in full on the internet: Sarbanes-Oxley Act of 2002. The SEC has now published rules for s302 and rules for s404 compliance.

The regulations say that companies should evaluate their controls against a recognised framework and the outstanding candidate for this in many countries will be the ‘COSO framework’. The executive summary of the original framework in freely available on the web at www.coso.org. However, a new framework is being drafted and should be issued some time in 2003.

The PCAOB has issued an official document setting out requirements for 404 reviews and these are currently available for comment and consultation on the SEC's website here.

The new rules on s404 compliance issued by the SEC and proposed PCAOB requirements contain some points with potentially explosive implications:

• The SEC rules say that you cannot conclude your control system is ‘effective’ if it has one or more ‘material weaknesses’. ‘Material weakness’ is defined in AICPA guidance. However, defining materiality is notoriously difficult (probably impossible) and neither the AICPA nor SEC succeed. Some people believe that external auditors somehow know, and have a shared understanding of, what material means. They do have some idea for mis-statements of financial information but it is vague and much vaguer for control weaknesses.

  The implication of this is that companies will look at their lists of control weaknesses and try to decide which, if any, are material. This will be in the absence of any objective guide and the temptation to dismiss weaknesses as immaterial will be strong.

• However, external auditors will be tempted to argue the other way. The SEC rules say that companies with no material weaknesses in their controls will be able to say that their control systems are effective. If a company says this and the external auditors cannot find any material deficiencies the external auditor will have to agree with this statement. They will be reluctant to do so.

  In the UK it was proposed some years ago that companies be required to report publicly on the effectiveness of their internal controls and that their auditors should attest. This idea was rejected because auditors felt that investors would imagine that ‘effective controls’ meant nothing could go wrong, even though it is widely agreed by auditors that internal control systems are not infallible. Auditors felt that if anything went wrong they would be sued by any party with a grievance.

• Who will win the arguments over materiality and what are the weaknesses most likely to come out, if any? The SEC rules define control of financial reporting to include protection of assets. Strictly speaking this is not a financial reporting risk. If a building is burned down or a valuable asset stolen the accounts can still be correct.

  The bad news for companies is that this definition brings into scope some of the toughest control issues of all, such as business continuity planning and computer network access restriction. These are the sort of issues that live almost permanently on the hit list of most large companies.

• Where there is a material weakness the company will have to write about it in public. For security and fraud weaknesses the words will have to be carefully chosen to avoid giving away information to potential fraudsters or computer crackers, without sounding secretive.

• The rules and guidance strongly support use of the COSO framework as a framework for evaluating controls effectiveness. The original COSO framework was an exciting conceptual model for controls and at that level it is helpful though not very specific. However, one of the volumes in the set was largely ignored. It was intended to provide evaluation tools and includes detailed tables of risks and controls, but the quality of this material is low and most of it is not applicable to most companies. Before you decide to adopt the COSO framework, check the details and decide what parts of the COSO report you are going to apply.

Words © 2003 Matthew Leitch. First published 7 January 2003.