Techniques for Evaluating Automated Knowledge Acquisition from Contract Documents: Fid-Bau Portal

Techniques for Evaluating Automated Knowledge Acquisition from Contract Documents

Al Qady, Mohammed / Kandil, Amr

Knowledge Management (KM) has become the focus of a lot of scientific research during the second half of the twentieth century as researchers discovered the importance of the knowledge resource to business organizations. Recent research recommended the use of semantic representation of knowledge that is expressed in natural language to enhance knowledge management systems (KMS). In order to address this need, the CRISP technique (Concept Relation Identification using Shallow Parsing) was developed utilizing a natural language processing tool for extracting concept and concept relations from construction contract documents. The extracted concepts and relations are then used to develop semantic representations of the important knowledge expressed in the documents. The process of knowledge extraction from textual documents is, however, a subjective task that may differ from one person to another. This paper presents the two evaluation methods used to compare the performance of the CRISP technique with human evaluators. In the first method, Kappa was used to measure the level of agreement of human evaluators on the concepts extracted from an evaluation set, and the level of agreement between human evaluators and CRISP on the same evaluation set. In the second method, a Gold Standard was developed from the results of the human evaluators and precision and recall for CRISP and the human evaluators were calculated based on the Gold Standard. F-measure was used to combine precision and recall and to consequently compare between the performance of CRISP and the performance of the human evaluators. Results from both evaluation methods were relatively comparable.