Collaboratively maximizing inter-ontology agreement for controversial domains: A case study of Jewish cultural heritage

paper, specified "short paper"
Authorship
  1. 1. Maayan Zhitomirsky-Geffet

    Information Science - Bar-Ilan University

  2. 2. Eden Shalom Erez

    Computer Science - Bar-Ilan University

Work text
This plain text was ingested for the purpose of full-text search, not to preserve original formatting or readability. For the most complete copy, refer to the original conference program.

Ontology is a semantic scheme which comprises the main classes (concepts) of the given domain of knowledge, their properties, inter-relationships and instances 1. The basic ontological relationship types are IS-A (hyponymy) and instance-of (a concrete type-of) which are the basis of the conceptual taxonomy, and other thesauri-like (e.g. "part-of", "related-to") or content-driven relationships (e.g. "located-at", "produced-by") are allowed as well. One of the standards for formal encoding of ontologies is RDF (Resource Description Framework) 2. RDF-based knowledge representation consists of triples (relations) of the form: (concept1 – relationship type – concept2). For example, in the domain of Jewish cultural heritage the following relations could be included in the ontology: (Passover IS-A Jewish Holiday), (Holiday part-of Cultural Heritage), (Orthodox Jew preserves Jewish Tradition). Every triple corresponds to a certain statement or fact on the domain.
Nowadays, ontologies are widely used as a formal domain vocabulary for content-specific agreements in a variety of knowledge-sharing activities, such as, information organization, retrieval and tagging 3. However, in the current state of the Semantic Web for many domains there are multiple diverse ontologies rather than one standard vocabulary. This is due to the fact that they are typically constructed by different experts who often possess contrary viewpoints especially in cases of controversial domains. These domains include cultural heritage, economy, politics, history, religion, art and even medicine. Apparently, when using several ontologies in a common application, mismatches can create incoherent results for the users. Therefore, reaching maximal agreement between these ontologies is necessary to standardize and unify the domain vocabulary. Hence, building unified consensual ontologies has become a big research challenge.
The objective of this work is to explore ways to maximize the inter-ontology agreement for controversial domains. Particularly, we experimented with the case of the Jewish life style domain which comprises cultural, religious and political aspects. We also aim to explore whether it is possible to identify consensual ontological relations from diverse ontologies and construct a maximal subset of consensual vocabulary for the controversial domain.
Research Methodology

To overcome the semantic heterogeneity problem in ontologies a variety of ontology matching algorithms were presented in the past decade 4 , 5. These systems usually focus on mapping individual concepts (and/or their taxonomic structures) of one ontology to similar concepts in the other one. However, the level of inter-ontology agreement assessed by the automated approaches is limited by the following major factors:
The algorithm's ability to recognize semantically similar concepts, which are frequently conveyed by different terms;
Matching of isolated concepts which does not reveal the maximal potential for semantic similarity of ontologies unless all the direct and indirect relations (triples) binding these concepts can be consistently matched as well.
The low overlap between the explicit terminologies of diverse ontologies (for both, concepts and relations) due to the viewpointdiversity of their composers (especially for controversial domains).

Some partial solutions were lately proposed in the literature. Thus, in order to reduce the impact of the first factor, a recent study by 6 proposed to employ "wisdom of crowds" for detecting similar concepts in two different ontologies. To resolve the second limitation, similarity between ontologies should be computed for relations rather than for individual concepts, as implemented by 7. He counted exactly and partially matching triples from a pair of given ontologies. This methodology is adopted in the current research which further focuses on matching relations rather than individual concepts. However, the third and the most crucial factor, a relatively small amount of common relations, still remains unresolved.
The main question is how to reveal and assess the potential maximal agreement between ontologies despite the low overlap between them. The essence of this problem is the underlying assumption that relations, which are present in one ontology, but are missing from the other ontology, are automatically considered as unmatched and increase the ontology disagreement level. Nevertheless, it can be observed that if the ontology composer did not choose to add a relation to his personal ontology, it is unclear whether he agrees or disagrees with the truth of this relation. To this end, we introduce a new collaborative approach where independent ontology composers can explicitly express their opinions on the others' relations. Thus, after completing the construction of their own ontologies the participants are exposed to the relations of the others and are asked to decide for each of them whether it is true or false. Then, the "real" exhaustive inter-ontology agreement can be calculated based on these votes rather than by counting the common relations in the original ontologies.
We distinguish between two levels of ontology agreement:
The local agreement between a pair of ontologies that can be calculated as follows:

Fig. 1: The local agreement measure

The global agreement definition between all the ontologies for the domain:

Fig. 2: The global agreement measure
These relations constitute the consensual part of the ontologies. The other relations will be considered as controversial. The threshold descriminating the consensual and controversial relations can be computed by applying machine classification on the composer's votes as features for each relation as described in the next section.

Experimental Setting and Results

Based on the above collaborative scheme for relation evaluation we conducted an experiment with 21 ontology composers (students of the Semantic web course in Information Science Department). At the first step, the group has chosen a set of 130 concepts which are the most representative of the domain of Jewish life style. Then, every participant was required to construct up to 100 RDF-style triples (relations) with the above concepts and a set of 15 predefined relationships (such as, IS-A, instance-of, part-of, disjoint-with, entails, located-at, antonym-of) independently from the other members of the group. The relations were inserted into the web-based system implemented for this purpose. Further, each one of 1175 distinct ontological relations, created by all the participants at the first step, was consecutively displayed by the system and independently evaluated as true or false by every participant of the group. The analysis of the results shows that the initial local agreement between the diverse ontologies (the CR component of the measure) was very low (0-22%) reflecting the controversity of the domain. This is despite the fact that all the participants used the similar set of concepts and relationships for relation construction. The exhaustive local inter-ontology agreement assessed after applying the collaborative evaluation procedure appears to be much higher (39-90%), as demonstrated in Fig. 3. Thus, our collaborative scheme substantially enhances the local agreement level between ontologies.

Fig. 3: The local agreement rates computed by the amount of overlapping relations in the original ontologies as a baseline (the blue skew) vs. by the votes after applying the collaborative evaluation procedure (the red skew). Axis X represents pairs of different ontologies in our corpus.
To create a golden standard evaluation required for the global agreement calculation, two experts were asked to annotate all the statements as correct (ground truth) or controversial (depending on one's personal beliefs). First, they worked independently and then reached full consensus through a discussion. Overall, 885 (out of 1175) relations were judged as correct facts (TRgoldstandard), and 290 as controversial viewpoints. For example, the correct relations (like, Passover – is-a – Jewish holiday; Reform Jew – disjoint-with – Orthodox Jew; Western Wall – located-in – Jerusalem) are expected to obtain the large majority of agreement ("true") votes in the collaborative procedure, while the controversial relations (such as, Ultra-Orthodox Jew – resists – Scientific progress; God – created – Universe; Bible – written by – Man) are supposed to gain intermediate scores for both agreement ("true") and disagreement ("false") voting categories.
The |TRoverlap| component needed for the global agreement calculation was rather low 29% (232 out of 885) even with the threshold of 2, as only 256 relations appeared in at least two ontologies. 919 (almost 80%) of the relations appeared in one out of 21 ontologies), while only one relation was present in 9 out of 21 ontologies.
Then, in order to estimate the global agreement after applying the collaborative evaluation procedure, we utilized the WEKA environment 8 to choose the optimal machine classification algorithm. Eventually, the best 10-cross validation results were achieved by the Multilayer Perceptron algorithm which yielded 90% average accuracy. As a result |TRcollaborativevoting| of 876 was obtained. Interestingly, all 232 relations of TRoverlap were included in TRcollaborativevoting. Overall, 99% of the correct relations according to the golden standard were classified as correct by the automatic classifier. The classifier used as features the true and false voting scores. Most of the errors in classification were controversial relations probably reflecting some common viewpoint among the members of the group. So in the future research we intend to conduct crowdsourcing microtask-based experiment (like in 9) with a much larger number of participants.
In summary, our collaborative method significantly increases the baseline agreement that can be achieved manually or automatically from the explicitly overlapping/matching relations. This methodology further leads to construction of a reliable large consensual ontology for controversial domains which seem impossible to achieve from the small overlap of the original ontologies.
References

1. Noy, N.F. & McGuinness, D.L. (2001). Ontology Development 101: A Guide to Creating Your First Ontology. Technical Report KSL-01-05, Stanford Knowledge Systems Laboratory and Stanford Medical Informatics Technical Report SMI-2001-0880, Publisher: Citeseer, (pp. 1-25) doi: 10.1.1.136.5085
2. Hayes, P. and B. McBride. RDF Semantics. W3C Recommendation: www.w3.org/TR/2004/REC-rdf-mt-20040210/ (2004 last accessed July 2013).
3. Gruber, T. R. (1993). Toward principles for the design of ontologies used for knowledge sharing. International Journal of Human-Computer Studies, 43(5-6), 907-928. Retrieved from itee.uq.edu.au/~infs3101/_Readings/OntoEng.pdf
4. Shvaiko P. and J. Euzenat (2013). Ontology matching: state of the art and future challenges. IEEE Transactions on Knowledge and Data Engineering, 25(1): 158-176.
5. Flouris G., Manakanatas D., Kondylakis H., Plexousakis D. and G. Antoniou (2008). Ontology change: classification and survey. The Knowledge Engineering Review, 23(2), 117–152.
6. Sarasua, C., Simperl, E. and N.F. Noy. (2012)Crowdmap: Crowdsourcing ontology alignment with microtasks. Proceedings of the International Semantic Web Conference, Boston, USA, pp. 525- 541.
7. d'Aquin, M. (2010), Formally measuring agreement and disagreement in ontologies. Proceedings of the Fifth International Conference on Knowledge Capture, Redondo Beach, California, USA, pp. 145-152.
8. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P. and I. H. Witten (2009). The WEKA Data Mining Software: An Update. SIGKDD Explorations. 11(1): 10-18.
9. Sarasua, C., Simperl, E. and N.F. Noy. (2012) Crowdmap: Crowdsourcing ontology alignment with microtasks. Proceedings of the International Semantic Web Conference, Boston, USA, pp. 525- 541.

If this content appears in violation of your intellectual property rights, or you see errors or omissions, please reach out to Scott B. Weingart to discuss removing or amending the materials.

Conference Info

Complete

ADHO - 2014
"Digital Cultural Empowerment"

Hosted at École Polytechnique Fédérale de Lausanne (EPFL), Université de Lausanne

Lausanne, Switzerland

July 7, 2014 - July 12, 2014

377 works by 898 authors indexed

XML available from https://github.com/elliewix/DHAnalysis (needs to replace plaintext)

Conference website: https://web.archive.org/web/20161227182033/https://dh2014.org/program/

Attendance: 750 delegates according to Nyhan 2016

Series: ADHO (9)

Organizers: ADHO