The discussion up to this point has been concerned with analyzing the contact lens dataset, transferring it to an expert system shell and running a consultation. However, the Interview facilities in the knowledge acquisition tool are designed to elicit the dataset at a stage when the expert may not be able to state the relevant attributes and a complete and consistent set of critical cases as in Figure 4.
Assume that the expert has not already worked out the table of Figure 1 and that the acquisition tool is being used to help him or her to describe the relevant attributes and critical cases. He or she chooses "New Data", specifies a dataset name of "Lens", moves to the Interview status screen and specifies the domain, context and names for cases as shown in Figure 17. The expert leaves the default rating scale range of 1 through 9, allowing intermediate or "fuzzy" attributes to be entered.
Figure 17 Status screen for contact lens knowledge elicitation
He or she then clicks on "Done", moves to the Interview case selection screen, clicks "Done" again and moves to the Interview case entry screen, entering some initial cases as shown in Figure 18.
Figure 18 Initial clients entry
What initial cases does it make sense for the expert to choose? He or she should think in terms of those which represent critical features of the problem. For example, in the contact lens problem, the fact that clients with reduced tear production should not have lenses fitted should be represented, as should the fact that astigmatic clients in general have hard lens and non-astigmatic clients soft lens. The clients entered in Figure 18 correspond to 1, 2, 3, 4, 6, 8 and 10 in Figure 4.
The expert may next choose to add some initial attributes that are clearly relevant to the problem. Figure 19 shows the attributes "not soft--soft", "not hard--hard", "normal--reduced" (tear production ), and "not astigmatic--astigmatic", added as initial attributes.
Figure 19 Initial qualities entry
Note that the expert has not assumed that the decisions to fit soft or hard contact lenses are mutually exclusive. Entering them as separate decisions allows for the possibility that either may be suitable for some types of patient. This example will show that this makes no difference to the behavior or performance of the knowledge acquisition tool or resulting expert system In fact, if the original dataset of Figure 1 is entered with two separate output attributes as is being done now, it will result in exactly the same behavior in the shell.
We know that the attributes entered are not sufficient to solve the problem. How does the acquisition tool help the expert to realize this? The most important capability is that of having interactive analysis at any time during an interview. The expert can see his or her dataset as a knowledge base and come to understand what is missing. Figure 20 shows the tool's hierarchical cluster analysis of the initial dataset with 6 cases and 3 attributes. It can be seen that "astigmatic" is clustered with "hard" and "not astigmatic" with "soft". It is apparent from the case clusters that groups of clients are not being distinguished. Note from the data in Figure 20 that the expert has chosen to use only the two extreme values of the rating scale, for example, in the top row the value 1 means "hard" is selected and the value 9 means "not hard" is selected. He or she could also have expressed doubt about some cases by using intermediate values.
Figure 20 FOCUS analysis of initial data
Figure 21 shows the tool's alternative spatial analysis of the initial dataset. It is apparent that neither type of lens should be fitted if tear production is "reduced", and that "astigmatic" goes with "hard" and "not astigmatic" with "soft". Again the lack of distinction between groups of client is apparent. These two forms of analysis are closely related as will be apparent from checking that cases and attributes that are close together in one are also close together in the other. The hierarchical analysis has the advantage that the relation between the raw data and the clusters is apparent. The spatial analysis has the advantage that the relation between the cases and attributes is explicit.
Figure 21 PrinCom analysis of initial data
To obtain a logical analysis of the entailments between attributes the expert or knowledge engineer has first to specify that the type of the first two attributes is "Output", not "Input", through the Type screen shown in Figure 22.
Figure 22 Setting the type of each quality
Running the Induct logical entailment analyis program then produces the rules shown in Figure 23. These conclude that soft lenses should be fitted when tear production is normal and the astigmatism is not astigmatic, and that hard lenses should be fitted when tear production is normal and the astigmatism is astigmatic. Note that this is already the simple default rule set for the contact lens situation already discussed. The exception rules are missing because the expert has not yet entered clients that should not have lenses fitted for other reasons.
Figure 23 Induct analysis of initial data
How does the expert proceed to add more attributes and cases? Several possibilities are open. He or she may continue the elicitation using triples or breaking case and attribute matches, or may think about the graphic representation of the knowledge base, or the rules produced so far. It does not really matter what prompts the expert to remember that age and prescription are also relevant variables and that there are some special cases that should be entered.
Note that it also does not matter if the expert enters a duplicate case. It will affect the result if the expert enters an incorrect decision. If this conflicts with an existing case it will be noticed quickly. Otherwise, it will be apparent through studying the graphic representations or rules.
Figure 24 shows a display of the dataset when the expert has entered another 7 cases (corresponding to clients 12, 13, 16, 18, 20, 22 and 24 in Figure 4) and the age and prescription attributes.
Figure 24 Display of final dataset
Figure 25 shows a spatial analysis of this dataset.
Figure 25 PrinCom analysis of final dataset
The area behind the clients for whom neither form of lens are recommended has been shaded so that the structure of the knowledge base stands out. Clients rsb, au, lj and btm are standard cases for soft lenses, tw, ts, sv and pml are standard cases for hard. Clients ajh, mmp and erms are standard cases for no lenses because their tear production is reduced. Client mn is the anomalous case where soft lenses might otherwise be fitted. Clients kwa and xc are the anomalous cases where hard lenses might otherwise be fitted.
Figure 26 shows the rules produced by an Induct analysis of the dataset.
Figure 26 Induct analysis of final dataset
This rule set is logically identical to those of Figures 7 and 8. If the priority of the tear production attribute is set to 98 as before using the Type screen then exporting this dataset to the expert system shell produces the same correct behavior as before in consultations. Note that the lack of effect of the differences between the original example of Figure 4, and the elicited data of Figure 24--the use of a 1 to 9 rating scale, of two decision attributes and of a subset of cases. The knowledge elicitation is robust against such differences.
Suppose the expert does not realize that the knowledge base is adequate at this stage and adds further cases? Provided the correct lens prescription is given according to Cendrowska's original model, adding additional cases will have no effect on the rules produced by Induct. It may vary the cluster outputs since the additional cases will weight the analyses differently, but the broad structures apparent in these analyses will remain the same. If the expert adds further attributes in principle the Induct analysis will not be affected. However, it is possible for an arbitrary attribute, such as hair color, to correlate highly with a significant attribute or the decision attributes. On a small dataset of cases this may result in the arbitrary attribute appearing to be significant to Induct. Increasing the number of cases should get rid of spurious correlations. However, it is probably of much greater importance that the expert notices a spurious attribute appearing in the rules and decides to remove it. The acquisition tool is primarily designed to support the transfer of expertise by interviewing experts. It also has powerful techniques for statistically sound empirical induction. However, the number of cases required to be entered increases very rapidly with the proportion of incorrect decisions and irrelevant attributes, and it is better for the expert to clean up the data than to rely on statistics applied to poor quality data.
Figure 27 illustrates this by showing the trade-off between data and knowledge derived from empirical studies using the data of Figure 4 artificially corrupted with noise and irrelevant attributes (Gaines 1989a,b). Using default logic the knowledge in the data can be captured in 5 rules. It can also be entered as 14 critical cases as illustrated above. If a random collection of cases selected from those of Figure 1 is used as a dataset then on average 90 cases are required. If 1 irrelevant random attribute is added then on average 160 cases are required. If 25% of the decisions in the data are incorrect then on average 325 cases are required. If 5 irrelevant random attributes are added then on average 640 cases are required. If 1 irrelevant random attribute is added and 10% of the decisions in the data are incorrect then on average 1,970 cases are required. These results show the significance of using the expert's expertise effectively, compared with attempting to regenerate that expertise through gathering large amounts of data.
Figure 27 The trade-off between data and knowledge
gaines@cpsc.ucalgary.ca 19-Sep-95