FACULTY WORKING PAPER NO. 982 ocTHB

DEC. 5

3>S

Analysis of Audit Judgment Through an Expert System

Christopher Dungan John S. Chandler

College of Commerce and Business Administration Bureau of Economic and Business Research University of Illinois, Urbana-Charnpaign

BEBR

FACULTY WORKING PAPER NO. 982 College of Commerce and Business Administration University of Illinois at Urbana-Champaign November 1983

Analysis of Audit Judgment Through an Expert System

Christopher Dungan University of South Florida (Sarasota)

John S. Chandler, Assistant Professor Department of Accountancy

This research was supported by a grant from the Peat, Marwick, Mitchell & Co., Research Opportunities in Auditing Program.

This is a working paper and should not be quoted or reproduced in whole or in part without the written consent of the authors Comments and suggestions should be forwarded to the authors.

Digitized by the Internet Archive

in 2011 with funding from

University of Illinois Urbana-Champaign

http://www.archive.org/details/analysisofauditj982dung

Abstract

The goal of the exploratory research described in this paper was to create an interactive computer program which would function as a dynamic, operating model of auditors' judgment and provide a tool for investigating the domain of audit judgment. The system would be operated in a noisy, natural setting to make credible audit decisions. The auditor's evaluation of the adequacy of the Allowance for Bad Debts con- stituted the functional area of interest in this study. The result of this work was the creation of AUDITOR — a model of an audit judgment in the form of an interactive computer program called an expert system. This paper describes how AUDITOR was designed and constructed. The results of two successful validation exercises are presented. A discus- sion of the insights into audit judgment provided by AUDITOR concludes this paper.

INTRODUCTION

Interest in the methods by which accountants make professional judgments has stimulated considerable research in recent years. Com- prehensive reviews of this work may be found in Ashton (1982) and Libby (1981). Whether effected via regression modeling or focused on the probabilistic aspects of judgment, it is generally agreed that some of the research to date incorporates in its design, features which may operate as limitations on the external validity of the work. That is, typically, a relatively passive subject has been requested to perform a highly-structured and well-defined task via a simplified response scale in a laboratory setting while utilizing a limited set of data with which he has been provided.

Several approaches have been used to mitigate these limitations, for example, by introducing less-restricted search behavior. Biggs and Mock (1980) avoided the introduction of any restriction on subjects' cue choices by studying their spontaneous verbal protocols as they per- formed in an experimental setting. Abdel-Khalik and El-Sheshai (1980) allowed subjects to expand the number of factors which might be examined by choosing their own cues from among those provided in a shopping list. Shields (1980) allowed subjects to make choices of cues from information boards in a study of the effect of information load upon information search patterns.

Like the work on verbal protocols, this present study utilizes self-reports from the subjects themselves instead of inferring a deci- sion model from the relationships between outputs and inputs. Although the validity of self-reports of mental processes is the subject of on- going debate, c.f. Ericsson and Simon (1980), Nisbett and Wilson (1977), Payne, Braunstein, and Carroll (1978), Einhorn, Kleinmuntz, and Kleinmuntz (1979), certain aspects of the present study distinguish it from previous work.

The goal of this present, exploratory work was to create an inter- active computer program which would function as a dynamic, operating model of auditor's judgment. The system would be operated in a noisy, natural setting to make creditable audit decisions. While operating within the confines of the available programming technology, the expert auditors who participated in building the system, themselves, freely determined the cues, their weights, and the form and size of the system. The result of this work was the creation of AUDITOR — a model of an audit judgment in the guise of an interactive computer program of a type pro- perly identified as an expert system.

Auditors' evaluation of the adequacy of the Allowance for Bad Debts (ABD) constituted the functional area of interest in this study. This particular audit judgment was chosen for study and modeling for several reasons. It was one of the judgments most frequently named by the experts themselves when asked, "What critical audit judgment requires the attention of an experienced professional and is never entrusted to 'green' auditors?"

-2-

Also, when auditors make judgments of the adequacy of the ABD they are presented at a subsequent visit to the client's office with an oppor- tunity for a referent outcome — the "outcome feedback" which is sometimes considered to be a necessary condition to sharpen and calibrate the expert's skills (Harrell, 1977; Ashton, 1981). That is, at a later date auditors can observe from the client's records which accounts have indeed proved to be uncollectible. Such a judgment seemed to the researchers more likely to provoke the consensus which Einhorn (1974) felt to be necessary requisite for expertise. Thus, in the absence of any other procedure for assuring that the decision process chosen for study would be one which would clearly be recognized as demonstrating expert judgment, the researchers believed expert judgment in auditing would most likely be fostered where the judgments were made under con- ditions which provided the judges with the possibility of objective outcome feedback. Finally, this area of -judgment gave promise of trac- tability in initial interviews with the experts, in that they seemed able to recite the cues to which they attend when making the judgment, and the cues seemed similar among several experts.

EXPERT SYSTEMS

Expert systems are computer programs which offer consultative advice in a bounded knowledge domain on a level of competence often rivaling that of a human practitioner who is recognized as an expert in his field. Such systems are constructed by a system-builder utilizing the active collaboration of one or more of the experts themselves, who are challenged to justify the manner in which they make their iudgments These systems are now providing valuable consultative advice to clients in real-world settings (Michie, 1980).

Expert systems trace their lineage to attempts in the 1950s to create computerized problem-solving routines of broad generality, for example the Logic Theorist program of Newell, Simon, and Shaw (1958). This program and later generalized problem-solvers such as ARGUS (Reitman, 1965) and GPS (Ernst and Newell, 1969) made contributions to subsequent work but did not themselves display great breadth of achievement. More recent work proceeds from a belief that the high- level skill demonstrated by a human expert derives from his accumulated experience of a concentrated nature which enables him to perform opti- mally in situations to which that experience is pertinent but is of little help in genuinely novel situations (Simon, 1978).

Although frequently applied to programs of medical diagnosis or treatment, expert systems' technology appears to be appropriate wher- ever evidence of less than certain reliability must be evaluated by a skilled practitioner according to experientially based rules in order to make a judgment. Thus, the field of auditing bears similarities to domains already approached via the technology of expert systems. Comments of workers in the field of "knowledge engineering," as it is sometimes called, support this analogy.

-3-

The domain is one in which diverse factors must be identified and synthesized to form judgments, evaluate alternatives, and make decisions. Years of experience are brought to the problem at hand; experience and subjective judgment play a major role. The domain is not easily amenable to precise scientific formulation. (Duda et al . , 1979)

The domain lacks a strong mathematical structure, is incorrigibly non-numerical, and is too complex for adequate analytical specification. (Michaelson, 1982)

The knowledge which the expert brings to the task is largely heuristic knowledge, experimental, uncertain — mostly good guesses and good practice in lieu of facts and rigor — much of this private to the expert. How else explain internships of guild-like apprentice- ship to a presumed master of the craft? What the master really knows is not written in the textbooks. (Feigenbaum, 1979).

Examples of successful, currently operating expert systems are frequently reported in the popular press as instances of "artificial intelligence." For example, MYCIN infers disease identity in blood or meningitis infections and suggests antibiotic treatment (Shortliffe and Buchanan, 1975). PROSPECTOR evaluates core samples to infer the pre- sence of significant mineral deposits (Duda et al. 1979). AL/X diagnoses causes of shutdowns occurring under the control of automatic safety devices on oil production rigs (Reiter, 1980).

AL/X (as did MYCIN) included in its design features the goal of separation of domain-specific knowledge from the computer control pro- gram which utilizes the knowledge. Thus, while AL/X was originally expert in only one realm of application its inference structure provides a foundation upon which can be built expert systems in other domains. AUDITOR utilizes the inference structure and control capabilities of AL/X, described in more detail in Appendix [A] . For a fuller treatment of its capabilities see Dungan (1983)."

AUDITOR FROM A USER'S PERSPECTIVE

The AUDITOR model was constructed in three distinct phases: initial modeling, refinement, and validation, all of which will be described in the course of this paper. However, certain aspects of this process of system-building can best be understood after the system has been de- scribed in use.

When a user operates the system interactively at a computer terminal for purposes of consulting AUDITOR for its advice concerning the large, delinquent receivable (which is the object of this inquiry), the

-4-

system asks him a series of questions about the extent of his personal knowledge of the presence or absence of the evidence cues required in the rules. AUDITOR begins each of its queries with the phrase, "How certain are you..."? which is prefaced to each of the rules in the rule base. For example, one query will be worded, "How certain are you that recent collections toward the delinquent portions of this account are proceeding satisfactorily?" The user's response provides a means for the system to acquire data about the underlying cue, suggested by the experts, which in this case concerns the debtor's recent payment performance.

The user responds to the system's query by typing a numeral called a certainty value (CV). CVs represent subjective certainty (or uncer- tainty) on a scale from -5 to 5. The response, "5," is given by the user of the system when he is certain that the matter referred to in the query is true. When he is certain the matter is false his response is "-5." In effect, the CV scale from .1 to 5.0 in absolute terms (unsigned) communicates increasing amounts of certainty, to which a positive sign is attached for truth and a minus sign for falsity. When the user has no information about the matter, he is unable to judge its truth or falsity, or he feels the question is irrelevant, he responds with "0." The researchers did not investigate individual differences in the use of this scale, although the process appears somewhat akin to the extraction of a utility curve, c.f. Newton (1977).

AUDITOR utilizes the data in the user's response in two ways. First, it updates the current strength of belief of the hypothesis, which in AUDITOR is stated in degrees of belief (DB), (DB = 10*logl0 (Odds)) The nature of the updating depends upon the nature of the links between the rules and the hypothesis: Bayesian (for IF: THEN links), Logical (for AND, OR, NOT), or contextual (adaptable for unique linkages).

Second, the system inspects its rule base to determine which one of the remaining questions to ask next. It does this by following criteria built into the AL/X control (unless they have been modified by contex- tual links). The criteria are simply stated: Ask that question next which could possibly impact greatest upon the DB of the hypothesis , con- sidering the inventory of questions, their diagnostic links, the current DB of the hypothesis, and the CV responses which might be given by the user. Thus, AUDITOR'S inquiry proceeds in economical fashion down that line of questioning which could have the maximum impact upon the hypo- thesis .

This process of query, response, and update continues until (1) all questions in the system have been exhausted, (2) the user terminates the session, or (3) the updating process has caused the DBs of the hypo- thesis to reach a threshhold level beyond which further questioning could cause no significant improvement, according to criteria adjustable within the system. In any of these cases, the system can then report its conclusion, which constitutes its "expert judgment" in the form of a statement of the current degrees of belief to which the hypothesis has been updated. In a typical consultation with AUDITOR this report might read:

-5-

The delinquent portion of this account should speci- fically be reserved-for in the allowance for bad debts to a substantial extent: RESERVE. Prior degree was 0.0, current degree is -19.5. At this point this goal is probability .05 or less.

This report informs the user of the system that the data provided by him has changed the degree of belief of the hypothesis from its ini- tial state, 0.0 (.5 probability), to -19.5 (approximately .013 proba- bility). At this point the user must decide for himself whether or not -19.5 DB (.013 probability) dictates to him a course of action, since there is insufficient experience with AUDITOR to provide guidance to relate these DB and probabilities definitively to decision points in an expert's judgment process.

CONSTRUCTION OF THE SYSTEM

A model of expert judgment, constructed in the form of an expert system, is assembled from components consisting of:

One or more hypotheses stating the judgment which is the end product of the expert's reasoning process.

Rules which express the relationship between the evidence (cues) and the hypothesis.

Parameters which express (1) the diagnostic value or impact which the expert believes are implied by each cue, and (2) the strength of the expert's beliefs prior to examination of any evidence, that is, the initial state of the system.

These components were assembled and tested during three stages: initial modeling, refinement, and validation. Initial modeling encom- passed development of the hypothesis which is the goal of the system, interviews with the auditing experts to elicit cues, conversion of these cues into rules, and polling of the experts to elicit parameters adequate to construct a preliminary, yet operating, system. In the refinement stage, experts operated the model interactively on a com- puter terminal and presented their suggestions for improvements to be integrated into the system. These improvements consisted primarily of changes in parameters, additions of interactions between the rules, and a few instances of rewording of the cues. Finally, the completed sys- tem was exposed to validation by testing it to learn the extent to which a different group of experts would similarly acknowledge it to be expert in its performance. (Further references within this paper to "experts" will mean the several auditors who participated in the several stages of building, refining, and validating the AUDITOR system.)

-6-

INITIAL MODELING: HYPOTHESIS

Expert systems in other knowledge domains are constructed under a belief that an expert's information search and processing are focused upon and organized around hypotheses, one or more of which ultimately will express the judgment which he will render upon the data. There is speculation that the predecisional behavior of accountants and audi- tors is similarly organized around hypotheses (Libby, 1981). After selection of the judgment area for modeling the next step in the con- struction of AUDITOR consisted of formulating an hypotheses to reflect the expert's judgment. In an expert system this judgment is referred to as the goal hypothesis.

AUDITOR'S single, goal hypothesis is expressed, "The delinquent oortion of this account should specifically be reserved for in the allowance for bad debts to a substantial extent." (For ease of refer- ence and recognition, the name of this hypothesis, RESERVE, as well as the name of each rule in the system will be written in capital letters.) The phrase, "this account" refers to the one individually large account — or in some cases a single invoice — which is under scru- tiny by the auditor. "To a substantial extent" was agreed among the participants of this project to mean all cases in which they judged recovery is likely to be no more than an amount considered insignifi- cant in relation to the delinquent balance.

RESERVE is intended to reflect the approach which the experts testified they visualized when scrutinizing the individually large, delinquent accounts of a commercial, audit client. It can be labelled a "worst-yet-possible-case" approach. This approach envisions that each, individually large account (or invoice) be judged collectible or uncollectible. The magnitude of the client's allowance for bad debts (ABD) is then considered adequate or inadequate in comparison to the aggregate of, the large, delinquent accounts which have been judged un- collectible.

DEVELOPMENT OF THE RULE BASE

The rule base for AUDITOR was developed from the cues provided by four practicing auditors chosen from among the eight who were consulted initially when choosing an area for modeling. Those actively partici- pating in the bulk of the project were three managers and one senior on the audit staff of a CPA firm of international scope. (By compari- son, for the development of expert systems in other fields the whole- hearted cooperation of one practitioner who is clearly expert is often considered adequate.)

In individual interviews these auditing experts recalled and named the cues to which they said they attend when evaluating the collecti- bility of a client's delinquent receivables. For example, one cue

-7-

relates to whether the debtor continues to be an active customer. Other cues relate to the opinions of the client's credit manager as to collec- tibility of the delinquent account, response to confirmation requests, and contents of the client's credit file on the delinquent customer. A total of twenty-five such cues were obtained, differing widely among themselves in the diagnostic impact which they have upon the auditor's conclusion. The complete list may be examined in Dungan (1983).

After eliminating from the list obvious redundancies and a few errors in transcribing, the researchers then transformed each of the cues into a simple rule in IF:THEN form. For example, a cue named by one of the experts was: "The customer's stated intent regarding pay- ment." This became the rule called NOTPAY: "The customer has stated his intent to pay little or nothing of the delinquent account," which is processed by AUDITOR utilizing a Bayesian revision as IF, NOTPAY: THEN (to the extent determined by parameters), RESERVE. The connector AND was later added (during the refinement stage of the work) in order to provide linkages between rules said by the experts to be interrelated. The connector NOT was used occasionally to achieve a more natural word- ing of a rule. (Although available for use through AL/X, the connector OR was not used. )

PARAMETERS

Next, parameters were developed for the rules — the diagnostic weights which express what the experts believe to be the evidential importance of each of the cues. This step began with a polling of the four auditing experts. Each of their rankings of "strong," "moderate," "weak," and "no effect" for each of the rules were translated into a quantitative scale on an expedient and preliminary basis. For example, all four experts rated the impact of NOTPAY as "strong." These pre- liminary values were later changed to the values contained in the final version of AUDITOR when the experts themselves operated the system and offered criticisms aiding in its refinement.

AL/X expresses the diagnostic impact of the evidence in the form of Positive Weights (PW) and Negative Weights (NW). As the AUDITOR system is operated interactively, the degree of belief attached to the goal hypothesis (RESERVE) is incremented by a value which is a function of the PW_ and the Certainty Value (CV) of the user's response if the user has keyed-in a positive CV in answer to the system's query, and upon the NW and the CV when the user's response is a CV in the negative range. The role of PW and NW are explained more fully in Appendix [A] .

REFINEMENT

Armed with a basic set of rules and parameters the researchers assembled a working model of AUDITOR with the aid of the AL/X software package. For purposes of refinement, this working model was returned to the experts to secure their suggestions for improvements. As a

-8-

result of this refinement stage, major changes were made in the PW and NW, numerous interactions were added, and a few changes in the composi- tion of the rule base were found to be desirable. An example of these changes, the rule called WORKOUT was originally developed from a cue which expressed the auditor's interest in a delinquent debtor's perfor- mance toward fulfilling a "workout alreement" or similar negotiated understanding between the client and the debtor. Before refinement the rule was stated, "Payments are being received currently under a workout agreement." As a result of reactions received during refinement of the system, this rule was restated to read, "Recent collections toward the delinquent portions of this account are proceeding satisfactorily," in order to eliminate the restrictive nature of the reference to "workout agreements . "

Also at this stage of the work interactions in the form of AND statements were added to the rule base upon the suggestions of the experts. For example, the individual effects of the rules called LEGAL and NOTPAY are enhanced when both of these cues are found present at the same time. That is, when a delinquent debtor has stated his intent not to pay (NOTPAY) and the same debtor also presents a counterclaim which would appear to make legal action fruitless (LEGAL), the AUDITOR system operates by implementing each of these rules individually and also implements a third rule, LEGAL&NPAY, which carries its own impact (in the form of PW and NW) upon the degree of belief in the goal hypo- thesis .

After interactions had been inserted, to accomplish further refine- ment the experts operated the system in "full trace" mode (so called). Although this mode was slow in operation, even tedious, it allowed the experts to observe the effect upon the hypothesis which might be caused by alternative possible responses of a user, thus stimulating them to suggest changes in the values of the parameters, PW and NW. The goal sought by these changes was to calibrate AUDITOR to the extent that a probability of at least 90%, (DB 9.5) should be reported by AUDITOR at the time of the expert's own report of a subjective feeling of "satis- faction" in his judgment, despite the difficulties with this proce- dure which might be predicted by a review of recent research, c.f. Lichtenstein, Fischoff, and Phillips (1977), Crosby (1981), and others, summarized both in Libby (1981) and in Ashton (1982). Although consensus was found among the experts in the direction of the changes which they suggested, the model proved somewhat insensitive to the exact values to which che changes were made, a result reminiscent of the work of Dawes (1975) and of Einhorn, Kleinmuntz, and Kleinmuntz (1979), a matter which will be explored in a subsequent paper. The complete set of rules and parameters utilized in the system may be examined in Appendix [C].

VALIDATION

The results of AUDITOR'S operations were compared against the judgment of practicing auditors utilizing client audit data contained in work papers. Two validation procedures were employed: "Open-book" and "Blind."

-9-

OPEN BOOK: The validation procedure referred to as "Open-book" was performed by an audit manager in a different office of the same CPA firm which participated in the building of the system. This auditor, serving as validator, selected work papers covering completed audits of two com- mercial clients with which he personally was not involved. From the work papers of each of the two clients the validator chose five individually large, delinquent accounts. The criteria which he used for his selec- tion was not communicated to the researchers. Each set of work papers to which he referred contained the decisions made by the auditors during the audit about their assessment of the collectibility of these five delinquent accounts (and others). Since the validator was unfamiliar with the two clients which he had selected, the work papers presumably also contained all the information from which he determined his responses to AUDITOR'S questions. Since to the validator it was obvious which set of judgments appeared in the work papers and which "judgments" were produced by AUDITOR, this procedure is referred to as an open-book validation.

After a demonstration of operation of the system by the researchers, the validator invoked the AUDITOR program and responded to its ques- tioning by entering information which he gathered from the work papers. That is, he responded to its inquiries by entering a Certainty Value on a scale from -5 to 5 in answer to each question presented to him follow- ing the prefix, "How certain are you...?"

In the case of each delinquent account, the validator himself decided when to exit from the system, thus concluding one session with AUDITOR. Presumably, that occurred either when he believed that he had provided to AUDITOR enough data by which he personally could have made a decision concerning collectibility of each account, or when all data available in the work papers had been entered into the system.

At the conclusion of each session the validator relayed to the re- searcher the judgment of the CPAs who had performed the audit. He also examined AUDITOR'S report of degrees of belief and probabilities which had been produced as a consequence of the interactive question and answer session. On the basis of his own criteria he labeled each result a "hit" (appropriate result) or a "miss." (In every case he commented that his own judgment regarding the write-off of the delinquent account agreed with that reported in the work papers.) The results are summarized in Table 1.

RESULTS: In each case but one, when the validator chose to ter- minate questioning and call for the report by the system, AUDITOR reported a degree of belief which was the equivalent of a probability of at least .865 in favor of the same decision as that called for by the original audit team — that is, either in favor of, or against, the need for an allowance for the delinquent account under scrutiny. The validator and the researchers considered these results to be successful.

BLIND VALIDATION: A second validation procedure was carried out to conform to the suggestions made by the mathematician A. M. Turing (1950).

a

•-4

CC

re

03

—

c

L

E

c

iH

0)

L

c

<2_

•H

•rJ

73

«

cu

co

4)

g

•1-1

4->

M

<

o

Li

u

o_

0)

o

c

u

c

c

0)

2

OS

c

H

1) C

<

z c I— I E- <

<

>

C/3

Cu

C/2

—

a co Ed os

si <

co

3 < 3

L

>> 03 <4-i J2 4J O -H

73

• cu 73

o .* 3

z o re

>

c

>,

oi

-C

CO

73

r-"»

o;

E

2

u

0)

i-i

—

jj

c

CO

03

D«

>.

c

01

CO

•r^

:*:

CC

a:

be

GO

^-v

<

01

H

jj o u

•H 3 >

--i < a!

03

u

a.

0)

c

>

c

u

o

Ol

CU

4)

«

>

>

03

m

o>

u

L

OJ

i— )

>

0)

0)

od

<o-

Li

03

03

0)

0>

a

o

li

03

PA

os

c

>

0)

o

L

o3

c

c

•>

0)

z

z

en

03

0)

a!

sC

CJ> C7*

I

a

X

O

c c

c

CM

O

■H

73

-

•« 0!

•H

4-1 4-1

c o

L

««"«»

0> 0!

CJ

— •

3 <— I

03

—1

a- i-i

01

•H

c o

e

s:

•i- o

CO

0) >,

T3 >H

4J

c c

O 0)

f^ CU

o o o

<r <o-

u_

C

OJ

cr c •H

a> c

73 >H

3 C C

o

o ^

• CCJ X J-i r^ o

CO- 4-1

m

03 0

o

bC

o c -a

C 0)

• tc in re

■H E

<j> re

j-> L

c o

3 n-i

o

CJ E

CJ -H

re re

l-H CJ

re

0)

C 3 O CT C c

in cy co- -c

0)

>

u o

03

01

06

CN

in

oo

i->.

av

vD

CTn

CTN

&

o

CT>

01

J3

CJ Li

re

01

CO

cu J-I

•a

cu

73 L

o u

0>

u

0)

73

co

CC

t-i

0)

CJ

u re cu

03

01

-

-c

Ui

c o cu

L

o>

T3

0 Z

c re

c

CJ

E C •H

3 cr

re

0)

01

t:

0)

bC

Ui

re

c

01 CJ

X

0)

Ol 4-i bO

3 Li

o> re

U £. L CJ

CN

* <

4-1

c

CU

0)

>

Li

01 03 Ol

OS

o z

Oi

L

01 03 01 OS

CU

>

u

01 03

0)

OS

o

z

0)

>

Li

01 03

01

CN

in

o>

in cc

en cr

en

cu

CJ

c re

re

J3

C 01

3

cr

01 CO

X3 3 03

XI CU

u

cj oi

0> -u

^^ re c

CJ J-l

0)

i— I cu

03 03 0)

c

•H 03

3 J3

cu

< 03 O

en

cr

c

01

c c c

<o-

c

3 L

C

re

03

Ol Q

m

u X

C I •

0> Li U *C

03 re —• cu

Oi C 3 £

Li 03 CJ

o. co cu re

CU -H L CU

L x. U

re; -h jj

3 O 3

C

c

os re o

H 3

Q -H 33 -a

< -H >

c o

■H L

a

.c c

ju CU

E

u- be

O TD 3

Li '—

O

> u

re j3

C 03

CU 03

3 w-<

cu n

i—i CU

re cu

o c

03

cu

_ _

CU

i-i 03

i— I CU

0) 03

>

0 re

03 E

cu cu

CU 4-1

U 03

&C >^

CU 03 TJ

: as O

OJ E-i

— —

33

E < O

U 01

U-l £

4-1

T3

CU 03

4_l C

1 re cu cu > E

c

C 4-1

CJ 03

c

03 t-

o) re

._i tc

4-1 <

-J C»

■H C3>

X •

re

£} •>

O 03

L 3

a j=

E-

cu

s_

re •

03 4J

i-i c

re i-i

E re

L &-S o

cr o. o^ cu

L

a

3 :

C 03 C ^ -H CJ CO

re 1-4

4-1 O 03 CU

CJ

B 03

0) -

4-1 E

03 U

>, -H

CO Cu

0> < J3 CU 4-1 C_3

C T3

01

L 73

oi re

CO cu

3 J3

01 3

J3 E

4-J 3

i— (

>> c x a

73 01 Ol j3

X E-

03 •H

3 • L >>

3 !-i

u-i re

CO

CO CO

re cu

CJ

» cu

CU 3 CJ

X

■• c

4J 3

c

3 4-1 •

o re co

CJ J3 L

CJ 4-i 0

re 4J

•> i-i

L 03 73

re tj 3

—i co

CJ

cj u cj re oi

o a vi ^ i

~ CJ 4-1 4-1 4J

-11-

This procedure is considered to be a more stringent test of the validity of an expert system and has become somewhat of a standard in the field (c.f. Yu, 1979). Turing, when confronted with the issue of whether com- puters and their programs could be said to think, proposed instead that the testable question is whether an observer who is himself ignorant — that is, blind — as to the identity of the source could distinguish between the output of the machine and that of a human.

Blind validation of AUDITOR utilized two human auditors, one who served as user of the system, the other as validator. Both auditors were managers with an international CPA firm not previously involved in the AUDITOR project. The auditor who performed as user selected an audit client with which he was familiar. His choice was a NYSE listed company which manufactures and markets through various distribution channels consumer products such as cigarettes and candies. His audit team, in their visit to the client's offices at an interim date, had identified by computerized selection eleven invoices so large and delinquent as to justify the auditors' individualized attention. The criteria for this selection was not communicated to the researchers.

The user was familiarized with the AUDITOR system and then in- structed to respond to its questions using the data contained in the work papers. Since he himself had been a member of the audit team it is not possible to determine if his answers were based solely upon the contents of the work papers or perhaps also relied upon his memory for information not in the papers. He answered the system's questions, responding on the CV scale from -5 to 5. As in the first validation, the user himself decided when to terminate each session. Concurrently, he reported relevant data from the work papers to the researcher who recorded it, such as age, dollar balance, confirmation results, nature of customer's business, etc. This data — recorded on Fact Sheets — became part of the raw data presented to the validator from which he made his judgment. The user also reported the audit teams' judgment, just as it was recorded in the work papers. This information was recorded by the researchers on a Comparison Worksheet, on which was also entered the "judgment" of the AUDITOR system.

Subsequently, the audit manager serving as validator was presented with the Comparison Worksheets which contained the judgments from the two sources and the Fact Sheets containing the data from which they made their judgments. (Of course, the sources of each judgment were disguised when presented to the validator; tbat is, he was "blind" as to the source — auditor or AUDITOR — of each judgment.) The task which he undertook was to study the same evidence on the Fact Sheets which had been available to the Fact Sheets which had been available to the two experts, auditor and AUDITOR, to form his own independent and pre- sumably expert conclusion, and then to accept or reject each judgment from each source on the basis of his opinion of the expertise which each had demonstrated. That is, he accepted or rejected each of twenty-two recommendations which were disguised as to source.

-12-

RESULTS: The validator accepted all of the audit team's judgments and all but one of AUDITOR'S. These results are summarized in Tables 2 and 3. In sum, over the two validation procedures, open-book and blind, the AUDITOR system scored nineteen hits out of twenty-one cases.

ANALYSIS

TAXONOMY

In AUDITOR a high score on one cue can offset a low score on a different cue. Thus, AUDITOR can be called a compensatory, rather than non-compensatory model. For example, credit to the delinquent customer may not have been stopped by the client (CREDITSTOP) , and the customer may have demonstrated a good record in the past of paying his account (GOODRECORD) . However, both of these cues, favorable to an expectation of collectibility of the account, may be offset — compensated for — by evidence that the customer is in bankruptcy proceedings (BANKRUPTCY) and the client's lawyer advises that recovery is unlikely (LAWYER).

AUDITOR incorporates probabilistic aspects of decision making, as do other successful expert systems, recognizing that no evidence is perfectly diagnostic of the condition which the expert must evaluate and that real-world judgments are made on the basis of information which must be probabilistically evaluated. The query format, "How cer- tain are you...?" reminds the system's user to perform a subjective assessment of the reliability of the evidence.

In contrast with Lens Model studies, and indeed with most studies of auditors' judgments except those utilizing Verbal Protocol Analysis, AUDITOR processes the independent variables with which it operates, in the form of user's CV applied to the PW or NW of a cue, in a sequential fashion. That is, AUDITOR continuously revises and expands the system as its queries are successively put to and answered by the user. In sum, AUDITOR incorporates subjective assessment of the evidence through what is basically a Bayesian revision process operating on cues which can compensate for each other. Therefore, the AUDITOR system can be called a subjective, Bayesian, compensatory expert system, which func- tions as a sequential model of auditors' judgment.

COMPARISON WITH OTHER STUDIES

Two previous studies have investigated auditors' perceptions of what was called source reliability (Joyce and Biddle, 1980) or source credibility (Bamber, 1980, quoted in Libby, 1981). Joyce and Biddle explored whether auditors weighted differently the reliability of infor- mation depending upon its origin with the client's credit manager or with an outside credit reporting agency. Joyce and Biddle concluded that unless both sources were called to the auditors' attention (as was done in a within-subjects experiment) they did not differently weight the sources. In the AUDITOR project no rule incorporating outside credit reports was suggested by the experts, and apparently none was referenced

-13-

TABLE 2

CONVERSION OF DEGREE AND PROBABILITY INTO VERBAL JUDGMENTS

AUDITOR'S Reported

AUDITOR was Degree of Belief AUDITOR'S Result

Concealed as and Equivalent Was Reported to

Case # Expert No» Probability Validator as

"OK, no res. nor adj."

"No reserve nor adj . needed"

"Yes , reserve or adj . req 'd"

"No"

"No"

"Yes"

DB

%

3-1

II

-15.5

3.2

3-2

I

-19.5

1.3

3-3

II

15

97.0

3-4

I

-30

.01

3-5

II

-17

2.0

3-6

II

26

99.0

3-7

II

2

61.0*

3-8

I

-30

.01

3-9

II

-30

.01

3-10

I

-30

.01

3-11

I

33

99.9

'At most a partial res or adj . req 'd"

"No"

"No"

"No"

"Yes"

*There were no clear guides for interpretation of the implications of AUDITOR'S result at this level of likelihood (61% probability of need for an allowance). Since the first validation procedure could be interpreted as suggesting that partial allowances by auditors might be associated with the AUDITOR'S results at this level of probability, the researcher chose to state the expert's opinion as "At most a par- tial res. (reserve) or adj. (adjustment) (is) req'd (required)".

-14-

TABLE 3

SUMMARY RESULTS BLIND VALIDATION

Case #3- Brief Description

1 $19,292 over 90 da. active gov't account, not confirmed.

2 $17,553 over 90 da. active gov't account, not confirmed .

3 $16,346, 30-60 da., 100% credit memo not posted. Not confirmed.

4 $35,909, 30-60 da., not confirmed. Workout agreement in file.

5 $59,570, 30-60 da., not confirmed. Bankrupt but letter of credit in file.

6 $13,563, 30-60 da., not confirmed. Customer request to return goods, CM. not posted.

$11,628, over 90 da., confirmed newer items no excep- tion. Creditmanager calls excellent account, but dispute over price.

8 $12,498, 30-60 da., not confirmed, later collected.

9 $14,655, over 90 da. not confirmed, later collected.

10 $22,024, over 90 da. not confirmed, later collected.

11 $36,763 delinq. and total. Letter in file: "Financial difficulty, no pay next six months ."

Judgments

AUDITORS Auditor

Validator Called

No reserve

Accept

No res.

A

No res.

A

No res.

A

Re s . or adjust

Reserve

A

A

No res.

"

A

No res .

A

No re s .

A

No res.

A

Res . or adjust

Res . or

adjust

A

A

At most, partial

Reject

reserve or adj .

No res.

A

No res .

No res.

A

A

No res.

No re s .

A

A

No res.

No res.

A

A

Yes, res.

:es, res

-15-

in the work papers — which covered large, commercial clients — consulted by the auditors during validation of the system. Therefore, no direct comparisons with Joyce and Biddle are possible.

However, several other outside data sources were included in rules suggested by the experts involved in building AUDITOR. Among these are PROBLEMS, relating to the results from confirmation requests, LAWYER, concerning the opinion of outside legal counsel, NOTPAY, which reports the debtor's stated intent not to pay (when that is known), and NONCONTACT, which is a cue activated by the client's and the audi- tors' inability to contact the debtor. These cues carry four of the five heaviest, basic weightings (PW or NW) in AUDITOR. It seems reasonable to believe that these relative diagnostic weightings carry within them a factor which is dependent upon the experts' belief in the reliability of the source of the information. Thus, these heavier relative weightings evidently reflect the experts' belief in the higher reliability of these outside data sources.

Additionally, the low diagnostic weight accorded to the rule CREDITMGR, which reports the opinion of the client's credit manager, seems to indicate that such a source, although frequently resorted to, carries low reliability. Bamber's 1980 study attempted to determine if audit managers differentially weighted the results of work done by audit seniors who were described in the experiment as being of dif- fering reliability. No comparison with the AUDITOR project is possible, since, as might be expected, no cues suggested by the experts focused on the competence of the audit staff.

RELATIVE IMPORTANCE OF THE RULES

Cues frequently used which also have heavy diagnostic weight are, by implication, the major diagnostic tools of the experts. Table 4 lists the cues in decreasing order of their importance in use as diag- nostic indicators during the validation trials. This table has been prepared, first, by totaling for each cue the largest of its Positive Weight or Negative Weight together with the weights of the interactions involving that cue (all in unsigned terms). Then, this combined weight has been multiplied by the number of uses occurring during the valida- tion trials. (Cues with zero usage during validation — implying that no evidence concerning the cue was present in the working papers — are excluded from this table.) The result provides a measure of the extent to which the system utilizes the different cues.

Some comments are appropriate concerning the frequency of usage. The figure for number of uses is a count of the number of times during validation in which a non-zero Certainty Value constituted the response by the system's user. Since twenty-one delinquent accounts were exa- mined during the validation runs, that number is the maximum which could appear as number of uses. The cue called COLLECTED serves less as a diagnostic tool for the experts than as a means within the AUDITOR system to identify which accounts continue to be doubtful, following

-16-

TABLE 4

CUES, THEIR WEIGHTS AND NUMBER OF USES, AS DIAGNOSTIC INDICATORS

Combined

Largest Weight No. of Combined

Unsigned W/Inter- Non-Zero Weight X

Cue PW or NW Actions Uses Uses

COLLECTED (Account is no longer delinquent by audit comple- tion date) 30

ECONOMIC (Economic factors hamper

ability to pay) 3

BANKRUPT (Debtor has entered bank- ruptcy proceedings) 3

CORRESPOND (Recent data in credit

file supports collectibility) 4

ACTIVE (Debtor continues as active

customer) 3

WORKOUT (Recent collections are

proceeding satisfactorily) 3

NOTPAY (Debtor expresses his intent

not to pay) 5

PROBLEMS (Confirmations reveals

serious problems) 8

CREDITMGR (Client representative expresses strong belief in collectibility) 2

LAWYER (Lawyer expects poor prospects

of recovery) 5

LEGAL (Debtor has strong counterclaim) 3

WRITEOFF (Writeoff of this account

represents a material adjustment) 2

COLAGENCY (Collection has been

assigned to agency or lawyer) 2

G00DREC0RD (Debtor has good record

of paying in past) 2

NOPAYEVER (Debtor has made no pay- ments on any invoice) 2

NONCONTACT (Confirmation request

was returned undeliverable) 4.5

30

19

19

11

15

20

14

11

21 9

11

10

9.5

13

13

600

171

152

143

105 78 70 55

45

42 36

22

21

20

20

9.5

-17-

their initial identification and listing as delinquent, perhaps during an early stage of the audit.

Rules with the highest usage reflect aspects of the auditors' apparent decision processes and usage of evidence. WORKOUT reflects the significance to the auditor of his knowledge of recent cash collec- tions from the delinquent debtor. The phrase, "proceeding satisfac- torily," in effect asks the auditor to evaluate the recency, regularity, and adequacy of the collections in comparison to what he judges is some acceptable standard. Because of the crucial nature of the sales and collection cycle, the inspection and testing of subsidiary accounts receivable ledgers and cash receipts records normally would be part of every audit. Thus, the information required in WORKOUT ordinarily would be available in every audit.

The frequent usage of CORRESPOND similarly reflects typical audit practice — that of examining whatever correspondence is available con- cerning a delinquent debtor. To some extent, also, this cue consti- tutes a catch-all. The great variety of different data which might be included in correspondence files by different clients and for different debtors discouraged the researchers from attempting to incorporate in AUDITOR'S rule base each separate cue which might be discovered. For example, one cue might indicate that a delinquent debtor has placed with the client a bank's letter of credit, which effectively removed any doubt about ultimate recovery of the account. A similar but dif- ferent cue to be found in another debtor's files might refer to a parent company's guarantee of indebtedness. To avoid a proliferation of such rules in this first attempt at building an expert system of auditor's judgments, CORRESPOND was formulated to allow the user to sum up in one response the data which he gathered from the client's corre- spondence files. Such a response constitutes in effect a "mini-judgment" by the user, which more experience with the system may suggest should be divided into several rules. ECONOMIC is also a rule of rather fre- quent usage, perhaps as a result of concern about the economic condi- tions which prevailed when this work was done in (1981 and 1982). Usage of CREDITMGR reflects the common audit practice of discussing delinquent accounts with a representative of the client.

On the other hand, several rules carry a relatively heavy impact but evidently are infrequently applied. Auditors report they seldom communicate with an attorney regarding a delinquent account, but would give strong weight to a pessimistic prediction which he might make (LAWYER). The creditor's stated intent not to pay could represent an important cue but is rarely available (NOTPAY). The only instances reported to the researchers of the presence of this cue occurred in the case of debtors who were disputing the propriety of the charges in their accounts, claiming errors in pricing. Similarly, if neither the audi- tor nor his client were able to communicate with the debtor that would represent an important cue (NONCONTACT) . No such situation arose between the large commercial and government entities involved serving as test cases in the AUDITOR studv.

-18-

FURTHER ANALYSIS

Three other aspects of the system and its usage are worthy of com- ment. Michie (1980) and others have suggested that an experienced prac- titioner's expertise consists of his ability to recognize clusterings of cues — that is, patterns — which he has encountered previously and found to be particularly diagnostic. Clusterings of cues may function as mini-hypotheses, allowing the expert to focus his evidence-gathering in an economical fashion. To the researchers, the most obvious of the patterns apparently utilized by auditors in the valuation of accounts receivable seems to be organized around a distinction between "slow-pay" versus "no-pay" as diagnostic categories for each delinquent account. Thus, a slow-pay customer, properly diagnosed, does not require a pro- vision for loss. However, this categorization may hide several patterns of less global impact, which may in reality be the organizers for a practitioner's judgment, such as "the economic-problem" pattern or "the legal-problem" pattern. This matter will be explored in a subsequent paper .

The second matter concerns strength of belief and the meaning of AUDITOR'S report. AUDITOR reports Degrees of Belief. This constitutes its expert judgment. The user of the system must decide for himself the significance of the report, for example whether 10 Degrees (.91 probability) mandates that an allowance be provided. The researchers attempted to calibrate the system during its refinement stage so that AUDITOR would produce a report of about 10 Degrees (positive or nega- tive), coincident with the time a human expert auditor reported that he had become "satisfied" with the data and had made his decision. How- ever, many studies reflect unfavorably on humankind's ability to pro- perly deal with data presented in a probabilistic format, c.f. Joyce and Biddle (1981a). Thus, AUDITOR'S report of Degree of Belief can best be viewed as an indication of relative strength of belief. No claim is intended either that AUDITOR'S report reflects objective prob- abilities nor that the researchers believe human experts process prob- abilities similarly in the construction of their judgments. However, while keeping this disclaimer well in mind, the researchers believe that by the time the users terminated their validation sessions AUDITOR can fairly be said to have reached an unambiguous result, at least in terms of the refinement criteria, in the great majority of cases. For example, for the third client, which comprised the blind validation trial, in only one instance of eleven delinquent debtors was the session terminated when AUDITOR was reporting less than four percentage points from a value which would represent certainty (0% or 100%) Table 2. In the ten open-book trials, eight reported within +/- seven points, results which are within the criteria (Table 1). Thus, the researchers believe this first complete version of AUDITOR can be said to diagnose the collectibility of delinquent accounts in an unambiguous fashion, which bodes well for the development of other expert systems for use in audit situations.

The final matter for discussion concerns the sequence of inquiry followed by the system. AUDITOR asks next that particular question

-19-

which potentially has the greatest impact upon the probability of the hypothesis. Human auditors do not organize their own evidence-gathering in such an efficient fashion, nor do they plan the audit in order to investigate every source of highly reliable information. For example, it is not a standard audit procedure to request confirmations from every large delinquent debtor, even though responses might reveal evidence of high diagnosticity through application of such rules as PROBLEMS, NORESPONSE, NONCONTACT, LEGAL, and NOTPAY. Neither is it common prac- tice uniformly to inquire of the client's attorney or collection agency concerning a delinquent debtor, although evidence of high diagnosticity might result (LAWYER, BANKRUPT, LEGAL, and various interactions). Explanation for this apparent gaffe is easily found. Auditors are cost conscious, AUDITOR is not. This expert system is designed to con- centrate on the diagnosticity of the evidence while ignoring its cost. Human expert auditors, at least those who work for profit-oriented firms, are expected to be time and cost conscious.

Additionally, of course, an eclectism in auditors' pursuit of evi- dence is motivated by factors such as the unpredictable availability of clients' records and employees, conflicting schedules necessitated by a desire to service several clients, and a need to accomplish several audit objectives in a limited time. Also, since both client and auditor seek more from the audit than a satisfactory valuation of the allowance for bad debts, other tasks may intervene.

In only one case during validation did sequencing of inquiry cause difficulty. In that case, #1-2, the user serving as validator ter- minated the validation session immediately prior to a question which had great bearing on that particular debtor and which would have put AUDITOR more in agreement with the judgment of the human auditor, that is, at a higher Degree and probability level. In all other instances, the sequence of greatest-impact-first, with termination controlled by the user, appeared to be a satisfactory method of handling the evidence contained in working papers without provoking a complaint by the user of the system. Thus, the sequencing in AUDITOR seems to be at least satisfactory to its users in the majority of cases. Whether it is rea- sonable to expect auditors to follow a more efficient search pattern is beyond the scope of this paper.

SUMMARY

Expert systems such as AUDITOR can constitute a model of auditors' judgment, answering the researchable and interesting question said to be at the heart of all investigation of diagnostic judgment in the face of uncertainty: what cues do the judges use, what are their weights, and how are they assembled into a judgment model. Thus, subject to the influence of the researchers' intervention, AUDITOR demonstrates the feasibility of a new approach to the descriptive study of auditors' judgments .

-20-

LIMITATIONS

All of the auditors who participated in the initial system-building were members of one office of one large public accounting firm. To a great extent all had been exposed to similar professional training. Additionally, they perform under policies which might perhaps be unique to one firm. However, auditors from other firms were involved in the refinement and validation stages of the project — which lends confidence in the universality of the system as a model of auditors' judgment and reduces the chances of parochialism.

Further refinement of the system may produce better results. Rules might be refined to reduce them more nearly to the elemental cues, reducing the need for the mini-judgments presently contained in such rules as CORRESPOND and LEGAL. Such a revised rule base might be more reliably applied by an unexperienced auditor — one who had not developed competence in the mini-judgments which a few of the rules call for. On the other hand, such changes increase the size of the rule base and lengthen the list of questions to which a user is exposed — factors which tend to decrease the convenience of working with the system. However, work papers used by the auditors who were involved in various stages of this work evidently contained very little relevant data beyond that called for in the rules of the system, with the exception of the age of the delinquent balance which was under scrutiny. Some indication during validation was gained that the absolute age of the account may have to some auditors a significance not reflected in the system. Therefore, a rule relating to age of the delinquent balance might be useful in the system. However, for Che large, successful, client companies whose audit work papers were referred to in various stages of this work it seemed likely that no delinquent account was allowed to gain more than a few months' age before some finality was forced in its disposition. (These can be presumed to be clients with relatively good internal con- trols over sales and collections.) However, there is the possibility that alternative systems of rules and weights may produce equally suc- cessful systems which demonstrate audit judgment.

The influence and beliefs of the observer — the researchers — cannot be ruled out in project of this kind. Also, the subjects' desires to please and accommodate the researchers may work to the detriment of effort to create and particularly to validate an effective expert system.

Criticism can be directed against the system for its use of Bayes ' revision. It is commonly believed that auditors are no more Bayesian in their processing of evidence than are other humans (Libby, 1981). However, the point is worth emphasizing that the use of Bayesian revi- sion in computerized expert systems operating in fields other than auditing produce results similar enough to those produced by human experts as to satisfy these experts themselves.

Further refinement of the system may be called for, particularly, regarding instances in which practicing auditors conclude that partial rather than 100% allowances are proper to provide against a delinquent

-^1 -

account. In the Open-Book Validation procedure, auditors called for a partial allowance against an account for which AUDITOR'S processing produced (upon replay by the researcher) a result of approximately 70%. Further experience with the system may serve to clarify the set of cir- cumstances under which human experts call for partial allowance and the probabilities which AUDITOR might be expected to report in the same circumstances.

On the other hand, the judgments of expert auditors may be suf- ficiently idiosyncratic and inconsistent that attempts to refine the system any further will be fruitless. Changes to the system will be carried out only after an analysis of the objective to be sought. For example, if enhanced convenience of the system for use by practicing auditors were to be an objective of modification, a desirable change might be to engineer a revision in the order in which certain questions are asked. Certain evidence, while of lower diagnostic impact, seems always to be available, and might be asked about early in a consultation to allow the user promptly to contribute his knowledge early in ques- tioning. On the other hand, already existing capabilities within AL/X permit the user to volunteer evidence prior to questioning. For example, if no request for confirmation was sent and no information was received from any outside source, the system could be modified to accept this data even before questioning began. These capabilities have not been exercised yet in AUDITOR.

-22-

FOOTNOTES

Named second in frequency was the judgment process leading to the budget of audit time, the study of which might have allowed useful com- parisons with previous research (c.f. Joyce, 1976). However, the widely differing descriptions of this process which were given by the experts discouraged the researchers from attempting to extract their expertise. The judgment process involved in determining the threshhold for "report- ing materiality" was similarly considered then rejected for study because the aspects of negotiation evidently resorted to between client and auditor have not been amenable to expert system technology.

2 AL/X (Advice Language / X) was developed for Intelligent Terminals

Ltd. by John Reiter, Steve Barth, and Andy Paterson in association with the University of Edinburgh and was supported by BP Petroleum Develop- ment Ltd. It is a Pascal system based upon the Prospector consultant system for mineral exploration developed at SRI International by Richard Duda, Peter Hart, and others, see Duda et al. (1979).

3 The capabilities of AL/X allow formation of queries in a form

calling for a "Yes" or "No" answer. This capability was installed after

the work on AUDITOR had begun. Because of the apparent ease with which

users of the system had already adapted their responses to the CV scale,

this added capability was not used in AUDITOR.

RESERVE does not make provision for situations in which the audi- tor concludes the amount to be provided for potential loss of the account should be more than zero but less than 100% of the delinquent balance. One such case was encountered among the twenty-one delinquent accounts reviewed during validation of the system, and the validator ruled that AUDITOR did not perform satisfactorily.

Four separate sets of cues were accumulated from these experts. The nature of differences between these cue sets will be explored in a later paper. The final AUDITOR system was an aggregation of the cues provided by the four experts.

Rules related by AND linkages are processed according to the "fuzzy logic" rules of L. Zadeh (1979): P(A1 AND A2 AND... AND An) = minimum [P(A1), P(A2) , . . . ,P(An) ] , where P is probability and the An are eviden- tial cues.

It was also necessary to "prime" the system by providing the Bayesian portions of it with initial (prior) degrees of belief. In the absence of any definitive guidance in the literature or from the experts all such initial values were set to reflect 0.0 degrees of belief, which is a probability of 0.5. At the beginning of any session with AUDITOR these values can be changed if desired.

-23-

g In both open-book and blind validation procedures the assumption

was made that the work: papers contained both the auditors' judgments and the support therefore, in conformity with Generally Accepted Auditing Standards, AU 338.

9 Since AUDITOR reports its results in degrees of belief and in prob- abilities, which would be a novel way for a subjective audit judgment to be recorded in work papers, it was necessary for the researchers to interpret and in effect to translate AUDITOR'S report before its presen- tation to the validator in order to maintain anonymity of the source of each judgment appearing on the Comparison Worksheet. This interpretation was easily made in all but one case. In every case but one, AUDITOR'S report produced a DB of at least 15, positive or negative, that is, probability of at least .97 either in favor of or against the hypothesis which called for an allowance. These results appeared to the researchers to be unequivocal and were reported on the Comparison Worksheet as that of an expert auditor who called for "No reserve nor adjustment needed," (in the cases which reported negative DBs) or "Yes, reserve or adjust- ment needed" (in cases which reported positive DBs). (In keeping with common practice the term "reserve" was freely used as a synonym for allowance for bad debts among the researchers and the practitioners.) However, one case, that of number 7, produced a DB of 2 (probability .61) — an equivocal result having neither precedent nor clear interpre- tation within the researchers' experience with the AUDITOR system. The researchers reported this on the Comparison Worksheet as, "At most a partial reserve or adjustment is required." On the same case, the audit team had found no need for an allowance; the validator scored this judg- ment at a hit. The validator initially scored AUDITOR'S result as a miss and that is the way it was recorded by the researchers. Later, however, when reviewing his work, he referred to AUDITOR'S result on number 7 as a "close call." The issue of partial allowance will be studied further in later applications of AUDITOR.

This cue contributes to the researchers ' impression that the expert auditors' basic diagnostic task consists of an attempt to clas- sify each delinquent account as "slow-pay" or "no-pay," that is, these categories constitute competing hypotheses for classification of each delinquent account.

Technical data for this appendix also came from "AL/X USER MANUAL," Andy Paterson, Intelligent Terminals LTD. Oxford, England, 1981.

-24-

APPENDIX A: Description of AL/X and AUDITOR11

The structure of an expert system, in general, and AL/X, in parti- cular, parallels the structure of human decision making. Data is gathered, analyzed, evaluated with respect to some criteria and used, and, if more data is required, what additional data to gather is deter- mined. Figure A-l shows how the specific step in the operation of AL/X map to the generic decision making phases.

The two key operational areas are revision and selection. Revision refers to the process of updating the Degree of Belief associated with the spaces, that is, the rules and hypotheses. Figure A- 2 depicts this revision as a two stage process. At all times the hypothesis, which is the goal of the system's inquiry process, has associated with it a Degree of Belief (DB) or simply degree. Degrees of Belief measure strength of belief on a scale having a range from -100 to +100 but, AUDITOR only uses a range of -30 to +30. The initial ("prior") DB value within AUDITOR of the hypothesis, RESERVE, is set to 0.0 but can be easily set to any initial value. The DB scale is derived from prob- abilities as:

Degree of Hypothesis(H) = 10 log10(Probability(H)/(l-Probability(H))

Similarly, every space carries a DB which reflects its prior strength of belief. The user's response (CV) to each question initiates stage 1; revision immediately of the DB for that space (unless the response is 0) . The value of CV revises the degree in the following manner. A user's response of +5 assigns to that space the maximum DB permitted by the systems, i.e., +100. Similarly, an answer of -5 results in a degree of -100. An answer of zero leaves the degree unchanged at its prior value. An answer other than +5, -5 or 0 results in interpolations between these three points. A linear interpolation is performed on probability values. A positive CV causes interpolation between the prior probability and 1. A negative CV value results in interpolation between zero and the prior probability. The resulting probability value is then converted back to a degree value. The effects of the user's answer are immediately pro- pagated through the inference net to update the DB of all spaces for which the selected question is evidence, including the hypotheses.

The second stage of the two-stage process of revision involves cal- culations of the incremental weight which will be added to the present degree of the hypothesis (and to the present degree of any other spaces which, in a particular system, might be consequences of the immediate question to which the user is responding). First, we will speak about revisions involving spaces connected via IF:THEN inferences. A process of interpolation is again used employing the PW or the NW of the space, together with the degree calculated from the user's CV response. If the evidence is true then a positive weight (PW) of one will increase the degree of belief of the hypothesis by one. Similarly if the evidence

-25-

GATHER DATA

£

SYSTEM

ASKS

QUESTION

USER'S

RESPONSE

(CV)

ANALYZE

REVISION

UPDATE DI

"1 [REPORT TCl

USER

EVALUATE

DETERMINE NEXT QUESTION

MORE UESTION

SYSTEM SELECTS NEXT QUESTION

( STOPj

Figure A-l Overview of Expert Systems

-26-

USER'S

RESPONSE

ON

CV SCALE

r

TRANSFORMATION OF CV TO PROBABILITY EQUIVALENT

CONVERSION OF PROBABILITY EQUIVALENT TO DEGREE OF BELIEF ( DB)

|_R EV I SIO N_: STAGE J,

r

____j

PW OR NW OF SPACE V/HICH IS CAUSING REVISION

1

CALCULATION OF INCREMENTAL WEIGHT ( IVV )

IW=f(DB,PW,NW)

HYPOTHESIS

NEW DB= OLD DB + IW

j REVISION: S TAG E_2

j

Figure A-2 Overview of the Process of Revision

-27-

is false and the negative weight (NW) of one then the degree of the hypo- thesis will decrease by one. The greater the positive weight then the more strongly does the presence of the evidence imply that the hypothesis is true. For negative weights, the corresponding implication is that the absence of the evidence implies that the hypothesis is false. In the situation where the presence of evidence implies that a hypothesis is false rather than true it is perfectly legitimate to have a negative, positive weight and a positive, negative weight (i.e., PW = -4 and NW = +5).

If the degree of the space, as a result of the user's response is greater than the prior degree of that space, then the incremental weight to be added to the degree of the hypothesis is calculated as

Incremental weight = PW/(PW* min(PW, current - prior))

If the degree of the space as calculated is less than the prior degree of that space, then the incremental weight is:

Incremental weight = NW/(NW* min(NW, prior - current)).

AND spaces are composed of the connection by AND of two or more IFrTHEN spaces. Each AND space has its own degree of belief. Also, each AND space has its own PW and NW. The weight which an AND space has on the hypothesis is dependent on the degree of that AND space and its PW and NW, just as if it were an IFrTHEN space. The degree of an AND space is a function of the degree of the components and is calculated as

degree (Al AND A2 AND ... AND An) = min(DB(Al), DB(A2), ..., DB(An))

Selection refers to the manner in which AL/X chooses which, of many, questions is appropriate to ask next. (See Figure A-3). The manner in which this is done is important because human experts are characterized by their ability to focus quickly on the most efficient line of ques- tioning. AL/X chooses and investigates that particular hypothesis which is most likely out of a group of perhaps several alternative hypotheses. The choice criterion is: Select that hypothesis which currently has the highest degree of belief. Once the hypothesis has been chosen, questions are selected which will speed resolution of the diagnosis. In AUDITOR, the process of selection consists of choosing which ques- tion to ask of the user, i.e., which question has the highest potential incremental weight impact on the hypothesis. The incremental weight of each of the remaining questions is calculated prospectively by the system. AL/X carries this out by scanning through all of the, as yet, unanswered questions, calculating the incremental weights which could

-28-

PW AND NW

OF UNANSWERED

QUESTIONS

PW AND NW OF AND SPACES

SYSTEM ASKS QUESTION

USER RESPONDS

SYSTEM CALCULATES

.IW*OF ALL UNANSWERED

QUESTIONS

SYSTEM IDENTIFIES UNANSWERED QUESTION WITH HIGHEST IW

IW= incremental v/eight

Figure A-3 Overview of the Method for Selecting the Next Question

-29-

arise from each question, as a result of a CV response by the user of either -5 or +5. (AND and NOT spaces are included in the calculation.) That question is chosen next which could conceivably cause the greatest impact on the strength of belief of the hypothesis. Since the poten- tial impact of an AND space depends upon the user's answers to the com- ponent spaces, the order of questioning may vary from one consultation to the next depending upon the user's responses to the components of the AND spaces. This knowledge is utilized by the system in the form of rules often in the style I_F: Evidence , THEN (to a specified extent ) :Hypothesis . The connectors AND, OR, and NOT may also be used.

The data are immediately incorporated into the system by the con- trol and processed through the rule base. This action updates the rule base to reflect the user's knowledge and revises the degree of belief in the hypothesis which has been the focus of the expert's inquiry pro- cess. It is the degree of belief in the hypothesis (and its transla- tion into probabilities) which constitutes the goal of the system and its expert judgment.

Goals with degree > 0.0 are:

The delinquent portion of this account should specifically be reserved for in the allowance for bad debts to a substantial extent (RESERVE). Prior degree was 0.0. Current degree is 13.8. At this point the goal is certain: Probability .95 or greater.

The above is AUDITOR'S report that the user's answers to its ques- tions have resulted in increasing the strength of belief of the truth of the hypothesis called RESERVE from 0.0 to 13.8 on the degree (of belief) scale. Prior to questioning of the user, RESERVE is presumed as likely to be true as false, that is, to have prior degree (or belief) of 0.0. A degree of belief (DB) or, simply, degree measures strength of belief in the goal hypothesis. The possible range is from -100 to 100. Since AUDITOR'S report is based upon its Bayesian processing of subjective probabilities, the researchers utilized the report of degrees and probabilities merely as a guide to imputing a judgment from AUDITOR, and initially operated under the assumption that a probability of .90 (degree of 9.5) would indicate the need for an allowance.

A degree of zero indicates that the hypothesis is equally likely and unlikely, i.e., probability of 0.5. The mid-range of the degree scale from about -10 to 10 DB represents a large range of probability (about .10 to .90). Thus, small increments in degree may represent large changes in probability. This effect is considered desirable in this expert system since small changes in degree are considered to be

-30-

relatively more diagnostic when the current truth or falsity of the hypothesis lies in a doubtful or ambiguous area.

Only rarely, however, are the cue weights symmetrical. For example, the rule stated as, "This debtor is in bankruptcy-type proceedings" (BANKRUPT) carries PW = 3.0, NW = -1.0. These values indicate that to the experts the debtor's bankruptcy is considerably more diagnostic — reflected by the PW = 3.0 — of uncollectibility than the absence of bank- ruptcy is predictive of collectibility.

The reader may notice two consequences of these relationships. First, an apparently small portion of the available range on the DB scale, say from about -10 to 10, encompasses the rather large proba- bility range .09 to .91. Additionally, since the effect of an eviden- tial cue is additive upon the hypothesis, the diagnostic or inferential strength of any piece of evidence is at its highest when it impacts an hypothesis of neutral degree and decreases as more evidence is accumu- lated. This is believed to be a desirable feature in an expert system.

-31-

APPENDIX B:

ILLUSTRATION OF AUDITOR'S OPERATION

A hypothetical case situation is used to demonstrate the operation of the system. First, the facts of the situation, as they might be in the working papers, are described. Second, the consultative session between AUDITOR and the expert is shown, including annotations to high- light the features of AUDITOR. And, third, the results of the AUDITOR session are given in terms of degrees of belief (DB) and probability, which represent the revised likelihood of the hypothesis under scrutiny.

SITUATION

The delinquent amount owed by this customer, a regional, publicly held hardware chain, is $82,000. This represents about two-thirds of the balance of one invoice approximately 75 days overdue on which the customer has recently paid $40,000. In total, the customer owes $320,000, the rest of it classified as current (45 days and under). The overdue amount, while large among delinquent accounts, is clearly not material to the client's financial statements. They show accounts receivable of $20 million and current assets of $100 million. The accounts receivable subsidiary ledger shows the most recent three-year history of the account, in which the customer has become overdue only one other time. He ultimately paid the full amount owed after an adjustment was made which represented about 2% of the invoice. Sales this year are above those of a year ago, probably reflecting a trend toward "do-it-yourself" repairs by hardware customers. Correspondence and other files indicate a long-term, on-going business relationship with no suggestion of any problems with the account.

The auditors did not attempt to confirm the account and there is no information available from an attorney or any other source outside of the client. The customer is not bankrupt. The client continues to make credit sales to this customer, who has paid other invoices within their due dates. According to the credit manager who has talked with them, the customer states emphatically he will not pay the balance which is overdue. The credit manager goes on to say, "Sometimes we give them a small adjustment when they put up a fuss like this, but they have been good customers for 15 years and they always pay." The "fuss" referred to in this case evidently resulted from a late delivery

CONSULTATIVE SESSION

The user first signs onto the system. AUDITOR then begins ques- tioning the user about the facts of the case. The order of the ques- tions depends on the user's responses. Material in brackets represents AUDITOR output, in parentheses the user's responses, and in asterisks the researchers annotations. The result of this session is that the

-32-

systera determines that there is only .013 probability (-19.5 DB) that an allowance need be provided for this account. Other, similar accounts would go through a similar set of interactions.

■33-

APPENDIX C: NETWORK DESCRIPTION FILE

MODEL 3ADDE3T VERSION A3DMA

SPACE RESERVE

TEXT DESCRIPTION /* THE DELINQUENT PORTION OF THIS ACCOUNT SHOULD SPECIFICALLY 3E RESEEVED-FOR IN THE ALLOWANCE FOR 3 AD DE3TS TO A SUBSTANTIAL DEGREE */

INFERENCE

PRIOR 0.0

SPACE OUTSTAND

TEXT DESCRIPTION /* WHILE SMALL PAYMENTS ARE 3EING RECEIVED ON THIS ACCOUNT. THE OUTSTANDING DELINQUENT BALANCE IS GROWING LARGER */ INFERENCE

PRIOR 0.0

RULES CONSEQUENTS ( RESERVE PW 1 NW 0 ) .

SPACE AVEAG2

TEXT DESCRIPTION /* THE AVERAGE AGE OF THE UNCOLLECTED PORTIONS OF THIS ACCOUNT IS INCREASING */ INFERENCE

PRIOR 0.0

RULES CONSEQUENTS ( RESERVE PW 0.5 NW -0.5 )

SPACE ALL3UT0NE

TEXT DESCRIPTION /* THIS CUSTOMER'S ACCOUNT 3ALANCS IS ALL CURRENT EXCEPT FCR ONE I^RGS AND DELINQUENT OiJ-JRGZ */ INFERENCE

PRIOR 0.0

RULES CONSEQUENTS ( RESERVE FW 0.5 NW 0 )

SPACE AL3T1&NOT?

TEXT DESCRIPTION /* ALL3UT0NE AND NOTPAY */ INFERENCE

PRIOR 0.0

LOGICAL DEFINITION AND ( ALL3UTCNE NOTPAY )

RULES CONSEQUENTS ( RESERVE PW 2.0 NW 0 )

SPACE CREDITS TOP

TEXT DESCRIPTION /* CREDIT TO THIS CUSTOMER HAS 3EEN STOPPED BY THE CLIENT */ INFERENCE

PRIOR 0.0

RULES CONSEQUENTS ( RESERVE PW 1 NW -0 . 5 )

-34-

4. Kac! been assigned to a collection age,.cy or C.-that this account ha^ DeB" °^ a

lawyer? ^3 (-5)

C .that this debtor has a ^V"^""^!* U>timately payi"9 ^stantially all delinquent balances? 71 (5)

«.,- •,,««■ ever been received from this customer? C.bhat no payments havs ever u

?3 (-5) '

*.~ «r. h^d stated his intent to pay little op C.that the customer haS sw , C.that despite tha

nothing of the e hnquent ^ ance s .^ ^ ^ ^ny

presence of « deh ^ 3uditor's call to the system to paid? ?1 (PG) * PG is Li either than an adequate amount of present its conclusion. £ailable Qvidence. has been entered into evidence* or all of the 3v<s* the system.*

•«««««""**"""to dea^e cf belief > -100.0 are: .-..none at Clnvest19ated goals with deg indicating that the goal

the moment. 3 *This should ^e r other words, thera

hypothesis has not been £hUX ™v£ ^ of the system prefers are still questions "nasked^ but ^ ^^ y^ .fc £g

that they not be asked. Also, t, V uheneVar '.t is over

for the hypothesis whenever

t

set to report the degree -100 DB-*

COther goals with degree > ~100 areS

u • „ ~r this account should specifically bs The delinquent Pj; ^, a^Sfor bad debts to a substantial

reserved- for in tne aixuv degree :RESERVe.

n a rurrent degree is -19.5. At this point this Prior degree was 0 . 0 . cui / | probt3bility is only. 013 goal is P^^i1^-9^;' ,ccount will become uncollectible: SP:;l'; user will Undoubtedly conclude no provision need b, rr.ade for loss from this account.*

^jc******** *************************************** ««*«******-«**************

. fh.t despite the presence of this delinquent CHow certamareyo hat £ esp P ^ # ^ ^^ ^

item, newer items have bj.«n * y P unanswered, and the user^ rp3pat5 the last question wnx.ul . .f •. • • ~ * ?! !u * ^^ r-^^e or Quit its questionn ing.* tells the system to ceasF?

■35-

CThe current goal is whether cr not the ce] inquent portion of this account should specifically be reserved-for in the allowance for bad debts to a substantial degree. RESERVE!. *7he system is announcing its goal hypothesis.*

CHow certain are you that your te"sts of cash receipts indicate that the delinquent portions oP the account in question have been collected substantially in their entirety subsequent to the date of the balance sheet? ?3 (-5) *0n this line the system has given a "prompt" in the form of a question mark ■ and the auditor has replied with "-5" which indicates he is certain the account haa not been substantially collected.*

CHow certain are you that serious problems with the delinquent portions of this account were revealed through confirmation causing you to believe they are invalid?

?3 (0) *No confirmation was requested* so the user responds with a "O" which means* in this case* the question is irrelevant.*

C...that legal counsel gives poor prospects of any significant recovery from this debtor? ?3 (0) #There is no information available from a lawyer.*

C...that the confirmation request was returned by the postal service as undel iverable and the client for several months has been unable to communicate with the debtor? 71 (0) -*The auditor responded to this question as irrelevant.*

•

C...that recent correspondence and other data in the customer's credit file supports your belief in the collectibility of this account? 71 <5) *The auditor believes the long-term business relationship will be continued by the customer* or that for other reasons the account will by collected.*

C...that recent collections toward the delinquent portions of thits account are proceeding satisfactorily? 71 (5) *A substantial amount has recently been received.*

C...that the credit manager expresses a strong belief in the ultimate collection of substantially all this account? ?3 (5)

C...that although a portion of this customer's total balance is still delinquent* he continues to be an active customer? 71 (5)

C...that this debtor is in bankruptcy-type proceedings? 71 (-5)

C...that economic factors are causing particularly bad effects on this customer's 3bility to pay? 71 (-5)

C...that the .r.erits of this debtor's likely counterclaim agiansfc your client's suit indicate that a legal action would be fruitless? °3 (0) *The auditor has no information about a

counterclaim.*

*»

-36-

SPACS NRES&NTACT

TEXT DESCRIPTION /* NORESPONSE AND NOT ACTIVE */ i .«- ERENCE

PRIOR 0.0

LOGICAL DEFINITION AND ( NORESPCNSE NOTACTIVE )

RULES CONSEQUENTS ( RESERVE PW 2.0 NW -1.0 )

SPACE NONCONTACT

text description

/* the conftrmatioh request was retorned 3y the postal service as cndelivsra3le and the client for several Months has been unable to communicate with the debtor */

inference

PRIOR 0.0

RULES CONSEQUENTS ( RESERVE PW 4.5 NW 0.0 )

SPACE NCNT&NOTACT

TEXT DESCRIPTION - -

/* NONCONTACT AND NOTACTIVE */ INFERENCE

PRIOR 0.0

LOGICAL DEFINITION AND ( NONCONTACT NOTACTIVE )

RULES CONSEQUENTS ( RESERVE PW 5.0 NW 0.0 )

SPACE NCTPAY

TEXT DESCRIPTION /* THE CUSTOMER HAS STATED HIS INTENT TO PAY LITTLE OR NOTHING OF THE DELINQUENT BALANCES */

INFERENCE

PRIOR 0.0

RULES CONSEQUENTS ( RESERVE PW 3.0 NW -1.0 )

SPACE BANKRUPT

TEXT DESCRIPTION ■ /* THIS DE3T0R IS IN 3ANXRUPTCY-TYFS PROCEEDINGS */

PRIOR 0.0

RULES CONSEQUENTS ( RESERVE PW 3.0 NW -1.0 )

SPACE LAWYER

TEXT DESCRIPTION /* LEGAL COUNSEL GIVES POOR PROSPECTS OF ANY SIGNIFICANT RECOVERY FROM THIS DE3T0K INFERENCE

PRIOR 0.0

RULES CONSEQUENTS ( RESERVE PW 5.0 NW -1.0 )

SPACE BANXSWRSLAW

TEXT DESCRIPTION /* 3ANXROPT AND WRITEOFF AND LAWYER •/ INFERENCE

PRIOR 0.0

LOGICAL DEFINITION AND ( 3ANKSCPT WRITEOFF LAWYER )

RULES CONSEQUENTS ( RESERVE PW 9.0 NW -2.0 )

-37-

S?ACS ECONOMIC

TEXT DESCRIPTION /* ECONOMIC FACTORS ARE CAUSING PARTICULARLY BAD EFFECTS ON THIS CUSTOMER'S ABILITY TO PAY */

INFERENCE

PRIOR 0.0

RULES CONSEQUENTS ( RESERVE ?W 3.0 NW 0.0 )

SPACE BAN&ECOSLAW*

TEXT DESCRIPTION /* BANKRUPT AND ECONOMIC AND LAWYER */ INFERENCE

PRIOR 0.0

LOGICAL DEFINITION AND ( 3ANKRUPT ECONOMIC LAWYER )

RULES CONSEQUENTS ( RESERVE PW 7.0 NW -1.0 )

SPACE GOCDRECORD

TEXT DESCRIPTION /* THIS DE3T0R HAS A GOOD PAST RECORD OF ULTIMATELY PAYING SUBSTANTIALLY ALL DELINQUENT BALANCES */ INFERENCE

PRIOR 0.0

RULES CONSEQUENTS ( RESERVE PW 2.0 NW 2.0 )

SPACE LEGAL

TEXT DESCRIPTION - /* THE MERITS OP THIS DEBTOR'S LIKELY COUNTERCLAIM AGAINST YOUR CLIENT'S SUIT INDICATE THAT A LEGAL ACTION WOULD 3E FRUITLESS */ INFERENCE

PRIOR 0.0

RULES CONSEQUENTS ( RESERVE PW 3.0 NW -1.0 )

SPACE LEGALSNPAY

TEXT DESCRIPTION /* LEGAL AND NOT? AY */ INFERENCS

PRIOR 0.0

LOGICAL DEFINITION AND ( LEGAL NOT? AY )

RULES CONSEQUENTS ( RESERVE PW 6.0 NW -1.0 )

*

SPACE NPAYSGDREC

TEXT DESCRIPTION /* NOT? AY AND GOCDRECORD */ INFERENCE

PRIOR 0.0

LOGICAL DEFINITION AND { NOTPAY GOODRECORD )

RULES CONSEQUENTS ( RESERVE PW -2.0 NW 0.0 )

STOP

-38--

SPACE NCTACTIVS

TEXT DESCRIPTION /* NOT OF ACTIVE V

INFERENCE

PRIOR 0.0

logical definition not active

s?^cz cr&np3slact

text description /* creditmgr and ( not problems ( :;ot active */

i:jference

PRIOR 0.0

logical definition and ( creditmgr nctprc3lems active ) rules consequents ( reserve ?w -3.0 sw 3.0 )

space correspond

text description /* recent correspondence and other data in the customer's credit file supports 3elief in the c0llecta3ility of this account */

i:;ferencs

PRIOR 0.0

RULES CONSEQUENTS ( RESERVE ?W -2.0 Nil 4.0 )

SPACE CCRRES&ACT

TEXT DESCRIPTION /* CORRESPOND AND ACTIVE */' INFERENCE

PRIOR 0.0

LOGICAL DEFINITION AND ( CORRESPOND ACTIVE )

RULES CONSEQUENTS ( RESERVE P» -3.0 KW 3.0 )

SPACE CCRRESSWORJC

TEXT DESCRIPTION /* CORRESPOND AND WORKOUT */ INFERENCE

PRIOR 0.0

LOGICAL DEFINITION AND ( CORRESPOND WORKOUT )

RULES CONSEQUENTS ( RESERVE PW -3.0 NW 3.0 )

S??JZZ WRITEOFF

TEXT DESCRIPTION

/* TOTAL WRITS OFF OF THIS ACCOUNT, IF REQOIRED, WILL REPRESENT A MATERIAL ADJUST- MENT •/

INFERENCE

PRIOR 0.0

RULES CONSEQUENTS ( RESERVE FW 2.0 NW 0.0 )

SPACE NC RESPONSE

TEXT DESCRIPTION /* THERE WAS NO RESPONSE TO YOUR CONFIRMATION REQUEST NCR TO A FOLLCW-UP REQUEST */ INFERENCE

PRIOR 0.0

RULES CONSEQUENTS ( RE^ZKTt PW 1.0 NW -1.0 )

-39-

SPACE COLAGENCY

TEXT DESCRIPTION /* THIS ACCOUNT HAS 3EEN ASSIGNED TO A COLLECTION AGENCY OR LAWYER */ INFERENCE

PRIOR 0.0

RULES CONSEQUENTS ( RESERVE PW 2 NW -1 )

SPACE PORMEREMP

TEXT DESCRIPTION /♦THIS CELHJ2UENT ACCOUNT IS FROM A FORMER EMPLOYEE */ INFERENCE

PRIOR 0.0

RULES CONSEQUENTS ( RESERVE PW 2 NW 0 )

SPACE NEW? AID

TEXT DESCRIPTION /* DESPITE THE PRESENCE OF THIS DELINQUENT ITEM, NEWER ITEMS HAVE BEEN FULLY PAID INFLUENCE . .

PRIOR 0.0

RULES CONSEQUENTS ( RESERVE PW 1.5 NW 0 )

SPACE NO PAY EVER

TEXT DESCRIPTION /* NO PAYMENTS HAVE EVER 3EEN RECEIVED FROM THIS CUSTOMER */

PRIOR 0.0

ROLES CONSEQUENTS ( RESERVE PW 2 NW -0.5 )

SPACE ISSUENOTS

.TEXT DESCRIPTION /» THIS DE3T0R HAS ISSUED NOTES FOR THE UNPAID PORTIONS OF HIS ACCOUNT */ INFERENCE

PRIOR 0.0

RULES CONSEQUENTS ( RESERVE PW 0.0 NW -0.5 )

SPACE OCTSAVSN?

TEXT DESCRIPTION /* OUTSTAND AND AVEAGE AND NOT? AY */ INFERENCE

PRIOR 0.0

LOGICAL DEFINITION AND ( OUTSTAND AVEAGE NOT? AY )

RULES CONSEQUENTS ( RESERVE PW 1 NW 0 )

SPACE CSTSCOSNPE

TEXT DESCRIPTION /* CREDITSTC? AND COLAGENCY AND NCPAYEVER */ INFERENCE

PRIOR 0.0

LOGICAL DEFINITION AND ( CREDITSTCP COLAGENCY NCPAYEVER )

^ULZS CONSEQUENTS ( RESERVE PW S NW 0 )

SPACE CST&NPS&WO

PEXT DESCRIPTION /* CREDITSTC? AND NCPAYEVER AND WRITEOFF */ INFERENCE

PRIOR 0.0

LOGICAL DEFINITION AND ( CREDITSTCP NOPAYEVER WRITEOFF )

RULES CONSEQUENTS ( RESERVE- PW 3 NW 0 )

-40-

SPACE COLLECTED

TEXT DESCRIPTION

/* YOUR TESTS OR CASH RECEIPTS INDICATE THAT THE DELINQUENT PORTIONS OF THE ACCOUNT IN QUESTION HAS 3EEN COLLECTED SUBSTANTIALLY IN THEIR ENTIRETY SUBSEQUENT TO THE DATE OF THE BALANCE SHEET */ INFERENCE

PRIOR 0.0

RULES CONSEQUENTS ( RESERVE PW -30.0. NW 1.0 )

SPACE PROBLEMS

TEXT DESCRIPTION /* SERIOUS PROBLEMS WITH THE DELINQUENT PORTIONS OF THIS ACCOUNT VERS REVEALED THROUGH CONFIRMATION CAUSING YOU TO 3ELLEVE THEY ARE INVALID */ INFERENCE

PRIOR 0.0

RULES CONSEQUENTS ( RESERVE PW 8.0 NW 0.0 )

SPACE NOTPRCBLEMS - '

TEXT DESCRIPTION /* NOT OF PROBLEMS */ INFERENCE

PRIOR 0.0

LOGICAL DEFINITION NOT PROBLEMS

SPACE RIGOROUS

TEXT DESCRIPTION /* COLLECTION EFFORT 3EING APPLIED BY YOUR CLIENT TO COLLECT THIS ACCOUNT IS LESS RIGOROUS THAN IS DESIRA3LE */ INFERENCE

PRIOR 0.0

RULES CONSEQUENTS ( RESERVE PW 1.0 NW 0.0 )

SPACE WORKOUT

TEXT DESCRIPTION /* RECENT COLLECTIONS TCr'ARD THE DELINQUENT PORTIONS OF THIS ACCOUNT ARE PFOCEEDINC SATISFACTORILY */ INFERENCE

PRIOR 0.0

RULES CONSEQUENTS ( RESERVE PW -3.0 NW 3.0 )

SPACE CREDITMGR

TEXT DESCRIPTION /* THE CREDIT MANAGER, OR OTHER COMPANY OFFICIAL, EXPRESSES A STRONG BELIEF IN THE ULTIMATE COLLECTION OF SUBSTANTIALLY ALL THIS ACCOUNT */ f

INFERENCE *

PRIOR 0.0 RULES CONSEQUENTS ( RESERVE PW -1.0 NW 2.0 )

SPACE ACTIVE

TEXT DESCRIPTION /* ALTHOUGH A PORTION OF THIS CUSTOMER'S TOTAL BALANCE IS STILL DELINQUENT, HE CONTINUES TO BE AN ACTIVE CUSTOMER */ INFERENCE

PRIOR 0.0

RULES CONSEQUENTS ( RESERVE PW -2.0 NW 3.0 )

-41-

REFERENCES

Abdel-Khalik, A. R. , and El-Sheshai , K., "Information Choice and Utili- zation in an Experiment on Default Prediction," Journal of Accounting Research (Autumn 1980), pp. 325-342.

Ashton, R. H., "A Descriptive Study of Information Evaluation," Journal of Accounting Research (Spring 1981) 19.

, Human Information Processing in Accounting (American Accounting

Association, 1982).

Biggs, S. F., and Mock, T. J., "An Investigation of Auditor Decision Processes in the Evaluation of Internal Controls and Audit Scope Decisions," ROADS Paper No. 80-7, (April 28, 1980).

Crosby, M. A., "Bayesian Statistics in Auditing: A Comparison of Prob- ability Elicitation Techniques," Report 80-008, Center for Audit Research, University of Georgia (1980).

Dawes, R. M. , "The Mind, the Model, and the Task," in F. Restle, R. M. Shiffrin, N. J. Castellan, J. R. Lindman and D. B. Pisoni, eds., Cognitive Theory, Vol. 1 (Erlbaum, 1975), pp. 119-29.

Duda, R., Gaschnig, J., and Hart, P., "Model Design in the Prospector Consultant System for Mineral Exploration," Expert Systems in the Micro-Electronic Age, D. Michie, Ed. (Edinburgh University Press, 1979).

Dungan, C. W. , "A Model of an Audit Judgment in the Form of An Expert System," Unpubished Ph.D. Dissertation, Department of Accountancy, University of Illinois (1983).

Einhorn, H. J., "Expert Judgment: Some Necessary Conditions and an Example," Journal of Applied Psychology, 59 (1974), pp. 562-571.

, Kleinmuntz, D. N. , and Kleinmuntz, B., "Linear Regression and

Process-tracing Models of Judgment," Psychological Review, 86 (1979), pp. 465-485.

Ericsson, K. A., and Simon, H. A., "Verbal Reports as Data," Psycholo- gical Review, Vol. 87, No. 3, (May 1980).

Ernst, G. W. , and Newell, A., GPS: A Case Study in Generality and

Problem Solving (Academic Press, 1969).

Feigenbaum, E. A., "The Art of Artificial Intelligence: Themes and Case Studies of Knowledge Engineering," Proceedings of the 5th Inter- national Joint Conference on Artificial Intelligence, (1979).

-42-

Harrell, A. M. , "The Decision-Making Behavior of Air Force Officers and the Management Control Process," The Accounting Review, (October 1977), pp. 833-841.

Joyce, E. J., "Expert Judgment in Audit Program Planning," Studies on Human Information Processing in Accounting, Supplement to the Journal of Accounting Research, (1976).

Joyce, E. J., and Biddle, G. C. , "Are Auditors' Judgments Sufficiently

Regressive," Research Opportunities in Auditing Distribution Service, Peat, Marwick, Mitchell Foundation, N.Y. #80-4, (1980).

Libby, R. , Accounting and Hunan Information Processing (Prentice-Hall, 1981).

Lichtenstein, S., Fischhoff, B., and Phillips, L. D., "Calibration of Probabilities: The State of the Art," in H. Jungermann and G. de Zeeuw (eds.), Decision Making and Change in Human Affairs, (Dordrecht-Holland: Riedel, 1977), pp. 275-324.

Michaelsen, R. H., "A Knowledge-Based System for Individual Income and Transfer Tax Planning," Unpublished Dissertation, Department of Accountancy, University of Illinois, (1982).

Michie, D., "Knowledge-based Systems," Working Paper UIUCDCRS-R-80-1001 , Department of Computer Science, University of Illinois at Urbana- Champaign, (1980).

Newton, L. K. , "The Risk Factor in Materiality Decisions," Accounting Review, (January 1977) _52_, pp. 97-108.

Nisbett, R. E., and Wilson, T. D., "Telling More Than We Can Know:

Verbal Reports on Mental Processes," Psychological Review, Vol. 84, No. 3, (May 1977), pp. 231-259.

Payne, J. W. , Braustein, M. L., and Carroll, J. S. , "Exploring Pre-

Decisional Behavior: An Alternative Approach to Decision Research," Organizational Behavior and Human Performance, (February 1978) 22, pp. 17-44.

Reiter, J., AL/X: An Expert System Using Plausible Inference, (Intel- ligent Terminals Ltd., 1980).

Reitman, W. R. , Cognition and Thought, (Wiley, 1965).

Shields, M. D., "Some Effects of Information Load on Search Patterns Used to Analyze Performance Reports," Unpublished manuscript, University of North Carolina, Chapel Hill, (May 1980).

Shortliffe, E. H., And Buchanan, B. G. , "A Model of Inexact Reasoning

in Medicine," Mathematical Bioscience, Vol. 23, (1975), pp. 351-379.

-43-

Simon, H. A., "On How to Decide What To Do," Bell Journal of Economics, (Autumn 1978), pp. 494-507.

Turing, A. M. , "Computing Machinery and Intelligence," Mind, (October

1950), reprint in Creative Computing, Vol. 6, No. 1, (January 1980), pp. 44-53.

Yu, V. L., Fagan, L. M. , Wraith, S. et . al., "Anti-microbial Selection by a Computer: A Blinded Evaluation by Infectious Diseases Experts, Journal of the American Medical Association, Vol. 242, No. 12, (September 21, 1979), pp. 1279-1282.

Zadeh, L., "A Theory of Approximate Reasoning," in Machine Intelligence, _9_, J. R. Hayes, D. Michie, and L. Mikulich (Eds.), (Ellis Norwood Ltd., and John Wiley and Sons, 1979).

D/174