ACTITUDES

An important objective of research in individual judgment and decision-making is the contribution to the improvement of the quality of the auditors’ judgments. In Chapters 4 and 5 of this thesis the quality of the auditors’ judgments has been measured in terms of accuracy (i.e., the level of conformity of the auditors’ judgments compared to the expert panel’s judgments) and consensus (i.e. the level of agreement among decision makers).

While lack of consensus among a group of auditors implies that at least some of the auditors are not accurate, high consensus levels need not necessarily imply accurate judgments. In the context of ‘assessment decisions’ (as described in Chapter 5) accuracy, as measure of judgment performance is not feasible. Hence, it would be useful to know the empirical relationship between accuracy and consensus. Given the relatively high consensus levels (mean of .72 reported in Chapter 5), combined with the moderate accuracy levels (reported in Chapter 4), we would expect that there is no relationship between accuracy and consensus. Previous literature (e.g. Ashton, 1985), however, suggested that accuracy and consensus are highly positively associated to each other (a positive correlation of .84 was reported).

This section first describes the way how both measures of judgment performance have been made comparable to each other. Subsequently, the correlations between accuracy and consensus will be reported and conclusions will be formulated directed to dealing with judgment performance measures in future research.

Comparison of judgment performance measures Accuracy and Consensus

The judgment performance measures ‘accuracy’ and ‘consensus’ each measures an aspect of judgment performance. To the accuracy measure, as computed in this thesis, there is an external criterion available. Accuracy has been defined as the conformity of the individual auditors’ judgments with the expert panel’s judgments. To ‘consensus’, on the other hand, no external criterion is available. Consensus measures the agreement between auditors. Judgment performance as measured by consensus, hence, is not related to individual auditors but to groups of auditors.

In order to make both measures more comparable to each other, the accuracy measure has been changed from ‘individual performance measure’ to ‘paired performance measure’, based upon previous literature (Ashton, 1985). For each pair of auditors the

mean level of accuracy has been computed resulting in 3.570 mean accuracy scores.

These scores have been adjusted for non-normality, using the Fisher r to Z transformation (comparable to the computation of consensus). Finally, the pair-wise accuracy (adjusted for Fisher r to Z transformation) scores have been correlated with the Fisher r to Z transformed consensus scores. This procedure has been performed for all four accuracy-scores (1. factor accuracy with respect to the identification of client’s business risks; 2. weighting accuracy with respect to the identification of client’s business risks; 3. factor accuracy with respect to the identification of client’s entity-level internal controls and 4. weighting accuracy with respect to the identification of the client’s entity-level internal controls).

Correlations between accuracy and consensus

Table 5.12 presents the results of these correlations.

Table 5.12 Pearson correlations between accuracy and consensus (2-tailed)

Accuracy1 Accuracy2 Accuracy3 Accuracy4 Consensus auditors -.118 (.000)* -.023

(.161)

-.031 (.062)* -.023 (.173)

Consensus students .114 (.117) -.074 (.309)

.201 (.005)* .105 (.150)

With respect to auditors, all correlations between consensus and accuracy are (slightly) negative. This result contrasts with findings in previous literature (Ashton, 1985). As far as I am aware, Ashton’s research related to the relationship between accuracy and consensus has never been replicated although related studies (e.g., Bonner, 1990) have been reported. Bonner (1990) examined both a cue selection task (judgment performance measured in terms of accuracy) and a cue assessment task (judgment performance measured in terms of consensus). This study, however, does not statistically compare the findings related to the accuracy and consensus measures.

Other researchers (e.g. Shanteau, 1984) had the view that experts disagree with each other by nature, concluding that consensus as such can by no means be a measure of judgment performance. The findings presented in Table 5.12 deviate substantially from Ashton’s (1985) findings. Table 5.12 suggests that high consensus levels in general are not associated with higher accuracy levels. Only for accuracy3 and accuracy4 in the student population, (slightly) positive correlations are reported. In my view, the relatively

low correlation levels imply that identification tasks and assessment tasks fundamentally differ from each other and are accompanied with different auditors’ judgment biases⁴¹. In the identification task, the auditors’ activities involve recall from prior experience and search for relevant information. Assessment tasks assume that relevant cues have already been identified and the information deemed relevant is processed into an assessment decision. The difference of the nature of the identification task and the assessment task is also clear from a decision aid perspective. Should the tasks be similar to each other, one could assume that judgment performance will increase with help of the same decision aids. In audit practice, however, identification tasks are, e.g., performed with use of (general or industry-specific) risk templates and team brainstorm-sessions, whereas judgment in a assessment task is primarily based upon experience (e.g. general experience in the auditing field, engagement-specific experience, task-specific experience, industry-task-specific experience, etc.)⁴². We, hence, argue that auditors having identified the wrong or less relevant risk factors, probably an ineffective audit approach will result. Also, the opposite might be true. E.g., relatively high accuracy levels are not accompanied with high consensus levels.

41 E.g. Conlisk (1996) mentions amongst other limits on human unbounded rationality: the ignorance of relevant information (e.g. in the identification of client’s business risk) and the use of irrelevant information (e.g. in assessing the impact of client’s business risks on audit risk).

42 Conlisk (1996) provides a meta-analysis of studies with mixed evidence related to the concept of bounded rationality. According to Conlisk there are a lot of researchers who do not question whether people are unboundedly rational. The question is whether they act approximately as if unboundedly rational. “Though people’s rationality is bounded they learn optima through practice, in the end acting as if unboundedly rational”. “People can learn from experience, suggesting how people come to act “as if”

smarter than they are. However, the learning logic cuts both ways. Learning is promoted by favorable conditions such as rewards, repeated opportunities for practice, small deliberation cost at each repetition, good feedback, unchanging circumstances, and a simple context. Conversely, learning is hindered or blocked by the opposite conditions.” In the context of this thesis, the relationship between general experience and feedback, on the one hand, and judgment performance, on the other hand, has been examined. The evidence presented suggests that general experience is not positively associated with judgment performance (in terms of accuracy and consensus) and that the level of feedback is positively associated with consensus, however, not with accuracy. From this study follows, hence, that the learning argument is partially favorably supported by feedback mechanisms (partially, because positive feedback

Future research directed to accuracy / consensus

As stated in the previous section, researchers disagree about the appropriateness of the consensus measure in measuring the auditors’ judgment performance. The results presented in this thesis also give rise to question the applicability of a consensus measure in judgment/decision-making research. From a practical point of view, however, there is a call for more transparency in audit files. Recently (2004), as a result of public scrutiny related to the quality of financial statements audits, the PCAOB issued new guidance with respect to the content of audit working papers. Independent reviewers, having reviewed the audit documentation, must reach the same conclusion as the preparer (or the audit team) has formulated. Audit firms more and more recognize the need for team sessions resulting in more uniformity and unity of audit conclusions.

Additionally, the technical offices of the Big4 audit firms have changed the status of the technical office. Auditors, requesting consult from their technical office, are no longer free in their choice whether or not to agree with the conclusions formulated by the technical office. Technical offices have become directive instead of only consulting and sparring. Hence, the audit practice calls for more consensus in audit judgments/decisions. It might be expected that, from this perspective, the consensus measure might evolve from a ‘confusing’ measure to a ‘standard’ measure during time.

From a methodological perspective, both the accuracy measure and the consensus measure have their distinct advantages and disadvantages. E.g., the consensus measure lacks an external (validation) criterion; i.e., group judgments represent the best available solution. From the start of their career, however, auditors start becoming experienced and framed by the engagement-specific issues they experience although becoming experienced also contributes to improved auditor performance. In the real world, group audit judgments are not readily available as reference material to the individual auditor. In the audit practice, the circumstance that groups of auditors who actually decide on the same case-setting is rare. Making use of consensus measure in auditing research, hence, is not in conformity with the real auditors’ world. However, the same “real world” is highly complex. Case-settings in auditing research are often simplified representations of this complex world. Indeed, the choice of (simplified) case-setting can be motivated by the phenomenon of ‘bounded rationality’ (which implies that even the auditors’ professional judgment is accompanied with framing effects and sub-optimality), but a perfect external criterion which should be favorable to making use of accuracy measures is also not readily available or not available at all. There is at least one criterion that can be used to increase internal and external validation. This refers to actual (material) errors in the financial statements. Recently, some evidence has

become available of the relationship between the predictive value of audit risk assessments to financial statement errors. In my view, auditing research would improve making use of a combination of accuracy and consensus measures.

Recently, the first research studies regarding the auditors’ judgments using the business-risk audit approach have been published and more studies on this topic can be expected. Audit researchers have become acquainted to the use of accuracy measures and consensus measures (although they are not often combined in a single study) in examining the traditional audit-risk based audit approaches. Longitudinal research, including a comparison of the traditional and modern audit approaches, should include both judgment performance measures.

5.7 Summary of empirical results related to the auditors’ ‘assessing the impact