Working In Uncertainty

Examples of test documentation improvement

Documenting tests is a vital skill for auditors. Here are just two small illustrations of the difference doing it well can make.

1. The first example is adapted from a real, signed off audit file. I have done nothing to make it less clear than it was.

"Test: Review sample of checks carried out Department Financial Managers and take a view as to whether financial management checks are consistently carried out and in line with the Deparmental Process Checklist.

Results: There is lack of sufficient evidence to demonstrate the monthly checks are consistently carried out by Department Financial Managers. One Manager had evidence to demonstrate instances where unrecognised issued on their SAP Financial Assurance Reports have been raised and resolved with the Regional Controller. The Regional Controller provided evidence to demonstrate queries raised by that Manager as part of the checks."

How many sample items? Are they of different types of check or just several of the same type? What does 'consistently' mean? Is this effectively saying that the department uses a number of different types of control and the test is actually a test of all of them? It is impossible to understand what was done or how strong the evidence is from this documentation. If an audit report based on it were to be challenged by management the conversation could be very awkward indeed.

Here's a better version:

"Test 1: For a sample of three Deparment Financial Managers selected randomly, and for the month of December 2011, check that evidence exists that each control check has been performed. See list below for the list of 12 control checks taken from the Deparmental Process Checklist and the type of evidence expected for each.

Results: As tabulated below, two Departmental Financial Managers had evidence of having performed all the control checks, but the third Manager (the manager of the XYZ department) had no evidence for 8 of the 12 control checks.

Test 2: To distinguish between controls not done and just not evidenced, For all control checks performed by the Manager of the XYZ department, ask the Manager to explain what she actually did for each control check in December 2011 and what was found.

Results: As tabulated below, the Manager of the XYZ department could describe clearly the work done and results found for 3 of the 8 controls for which no documentary evidence was available. She seemed unclear how to perform the others."

With this kind of testing and documentation it is much easier to understand the actual situation and decide what to report.

2. Here's a second example, also based on real documentation.

"Test: Verify whether checks are carried out at the right level (Departmental Financial Manager).

Results: The sample of Departmental Financial Managers interviewed in the audit sample confirmed that the financial management checks have not been delegated to another member of staff in their department."

Again, what is the sample? Which checks are meant? What does 'right level' mean? How was the test actually performed? If Managers were asked to confirm that they had done the control checks themselves then this is a weaker test than asking them who performed the control checks. The second version gives them no clue as to who is supposed to carry out the control checks. The first approach gives them a big hint about what they should say.

A better documentation of the test would be:

"Test: Ask three randomly selected Departmental Financial Managers who performs the control checks as part of the Monthly Assurance Process. (The Standard Procedures require them to be done by Departmental Financial Managers.)

Results: All three stated that the control checks were performed by them personally (not delegated)."

The documentation of the test is improved, though you may not be entirely happy with the test itself. Was there perhaps a better way to establish who actually did the control checks? Probably. Now that the 'test' clearly owns up to being nothing more than an enquiry it is more obvious that its value is limited.

Made in England