Working In Uncertainty

Efficient reviews of documentation of internal control systems and audit testing: The startling economics of controls documentation reviews

by Matthew Leitch; first appeared on www.irmi.com in September 2007.

All around the world, right now as you read this, there are people reviewing controls documentation such as flow diagrams, risk control tables, and records of audit tests performed and conclusions drawn. Many of these people are experienced, intelligent, and highly paid. Their reviews will result in remedial work by others. The whole undertaking is costly but the audit profession has done little or nothing to study, scientifically, the economics of reviews. Happily, someone else has and the results are startling.

When I was a young trainee accountant and junior auditor the routine of reviews was simple and universal. The audit team would go to the client's offices and fill paper files with working papers and photocopies. Near the end of the visit a manager would arrive and read through the completed working papers, initialling each one and making a list of ‘review notes.’ We, the team on site, would anxiously wait to find out how many review notes we had to deal with and how much over budget they would take us. Every review note had to be dealt with rigorously, which meant going back to people with queries, correcting mistakes, and generally doing work we did not expect to have to do. After that came the Partner's review and possibly more remediation but usually not much.

Later, when I moved up in the pecking order and was a manager myself things had changed, but only a little. Now there was an electronic work file so I could do my reviews from the comfort of my own desk, but still the routine was the same; I reviewed everything near the end of the work and every point I raised had to be dealt with.

If you've ever done this kind of reviewing then you know it is boring almost beyond belief and it is very difficult to stay alert as you plough through hundreds of pages of confusing text and questionable diagrams.

Projects launched to comply with the notorious section 404 of the Sarbanes-Oxley Act of 2002 have generated an unprecedented volume of audit/controls documentation and boosted consumption of strong coffee.

At some point in my career I started to review work by people earlier and earlier, which is easy with electronic work files, and with new people in particular this usually led to some intensive coaching to try to get them writing good documentation so I wouldn't have to list so many corrections later on.

It is quite possible that many people have changed their approach as I have over the last decade or so, but still I have not seen any published research on this or any other aspect of the economics of controls documentation review.

Research from engineering

However, in the world of systems engineering there is even more documentation, it is not dissimilar to controls documentation, and engineers have gathered detailed data that lead to some helpful conclusions.

This research has been pulled together usefully by Tom Gilb, who is an authority on document inspections among other things, and it was Tom who introduced me to this research.

The first finding of interest is that the slower you do a review the more defects you will find. A graph of the relationship between pages per hour and number of defects found per page shows that as you go slower the number of defects found per page rises gradually, but this accelerates so that as you get towards one page per hour the findings shoot upwards.

In other words, reviewing at the usual auditor's speed of perhaps 50 pages in an afternoon (and that's being careful) will identify only a tiny percentage of the total defects.

The second important finding is that if additional reviewers review the same document they find some additional defects, even when all are working at the very slow optimum rate of one or two pages an hour.

The third important finding is related to the first two. It is simply that the number of defects in ordinary systems documentation is staggering – typically more than 100 defects per page.

I have asked people in my seminars to review controls and audit documentation very slowly and found that they find very large numbers of defects, so I think the engineering research findings apply to audit and controls documentation too.

Don't panic

So, does this mean we should review controls documentation at a rate of one or two pages an hour and expect people to fix the thousands of defects we will find?

Fortunately, the answer to that is ‘no.’ The last crucial discovery from the engineering research is that the major benefit of inspection programmes in engineering has been educational. Inspections teach people to avoid writing defects in the first place.

In fact, if you inspect just one page from a longer document and feed back the results properly to the writer, and if you say you will not accept the document until defects per page has reduced to a specific, very low number, then the rate of defects on the next review is usually about halved. Each subsequent review leads to another halving of defects and this applies to all pages the person writes, not just pages that have been reviewed before.

Guidelines for economic reviews

This research, Tom Gilb's advice, and my own practical experience of reviewing controls documentation and coaching people to write it better suggest the following guidelines:

1. Think of reviews as being largely for coaching purposes

There is still a need to check that there are no gaping holes or errors in documentation, but the large bulk of review effort should come before that and be part of coaching. Think of changing reviews from being entirely about remediation to being mostly coaching and partly remediation.

2. Start early

If someone who works for you hasn't had their documentation rigorously reviewed before you can usually start the process after they have drafted half a page of work. That's all you need and why let them waste any more time than that?

3. Use rules

Write down the rules that documents must comply with so that failure to comply is a defect. These rules clarify the process for everyone and make it easier for writers to see what is expected. For example, here are some rules for diagrams. In practice most defects arise from R1.

Rules for diagrams:

R1: Clear: Diagrams must be unambiguously clear to the intended readers (this year and next year).

R1.1: Diagrams must be tied to physical reality, including software reality, and not introduce conceptual views that distort that reality in the interests of explanation.

R1.2: Diagrams must use symbols consistently and those meanings must be explained, e.g. with a key or because they are standards for the project as a whole.

R1.3: Diagrams that show processes must put short explanations of what the process does on each process box.

R1.4: Diagrams that show data flows must put short explanations of what is flowing on each flow arrow.

R2: Complete: If a data flow diagram shows a process then all material data flows to and from that process should be shown.

R3: Separation of process and controls: Controls and the underlying process being controlled must be separated visibly, even if they are combined on one diagram.

4. Review a small sample very slowly

Review a page or two very slowly, identifying as many individual defects as possible.

5. Be nice

A large part of the engineering literature on reviews is concerned with making sure people are told the perceived defects in a way that is not upsetting or unpleasant. It helps if everyone knows that high defect levels are normal for people new to the rigorous review approach.

6. Have a rule about acceptable defect rates

Tom Gilb points out that this encourages people to learn their lessons and make the changes they need to make. If their work cannot progress to the next stage without a specified, measured level of quality being attained, then people focus more on quality.

7. Clarity then suitability

Often, documents are so unclear that it isn't possible to tell if what they are saying is appropriate or not. In these situations the first review(s) will focus on clarity. Then, when clarity has improved, it is possible to move on to suitability.

8. Leave time for the last remediation

If most reviewing has been turned into a form of coaching then the final stage of review can be what it was always intended to be: a rapid search for important defects that must be corrected for the job to be properly completed.

If the coaching reviews have done their job, there shouldn't be many of these to find and reading the papers should be much easier.

Summary

Reviewing controls and audit documentation is an expensive remedy for insomnia but research from engineering suggests that we may be able to get more value from it, particularly in teaching people to write well in the first place.

Perhaps in future there will be research specifically on controls documentation to see what the typical percentages are and how they compare to those from engineering.