Thursday 8 January 2009

OWL benchmarks

There is a great deal of confusion about the tools to realize semantic web applications. This confusion stems from the fact that there are a large number of tools available and the benchmarks that compare these tools are quite diverse.

For the purpose of the clarity, I will summarize these tools and benchmarks here. I intend to comment on them as much as I can during next few weeks.

(As of January 2008)

OWL reasoners: ( for reasoning, inferencing)
  1. Pellet - free/open source with some restriction on commercial use - Supports OWL DL, DL Safe SWRL rules.
    Java based
  2. RacerPro - Educational free evaluation licence, commercial paid licence - Supports OWL DL
    Java based
  3. KAON2 - Free for educational and non-academic use, different licence for commercial use. Supports OWL-DL and SWRL with F-Logic ontologies
    Java based
  4. FaCT++ - seems free,open source
    C++ based
  5. OWL API - free, open source. This is actually OWL API with reasoner interface to FaCT++ and Pellet.
    Java based
  6. OWLIM - free version with limitations, commercial licence is with all features
    Supports RDFS and most of OWL-Lite. More importantly contains a semantic repository over Sesame.
    Java based
RDF Stores (for storing RDF triples)
  1. Virtuso from OpenLink
    Getting started link is here
  2. AllegroGraph from Franz Inc.
    Java/Lisp based
    Java learning centre is here
  3. Mulgara (previously Kowari)
  4. SwiftOWLIM and BigOWLIM from ontotext
    SwiftOWLIM is an open-source library, under LGPL
    BigOWLIM is available under an RDBMS-like commercial licence on a per-server-CPU basis
  5. Sesame RDF store from Aduna
  6. Oracle 11g has RDF store support
  7. Scalable Ontology Repository (SOR) [Previously Minerva] from IBM. It supports DLP (Description Logic Program), a subset of OWL DL.
    Part of the IBM Integrated Ontology Development Toolkit.
  8. D2RServer is a a tool for serving Linked Data views on relational databases.D2R server relies on a declarative mapping between the schemata of the database and the target RDF terms. Based on this mapping, D2R Server serves a Linked Data view on your database and provides a SPARQL endpoint for the database.
Bechmarks to evaluate these reasoners/storage systems:
  1. Lehigh University Benchmark (LUBM)

    Y. Guo, Z. Pan, and J. Heflin. LUBM: A Benchmark for OWL Knowledge Base Systems. Journal of Web Semantics 3(2), 2005, pp158-182.


    One of the early works on the subject, however now seems outdated. Good reference point for any future comparisons.

    Compares the RDF stores.
    timeline: 2004/2005

  2. Berlin bechmark
    from Freie Universität, Berlin
    Benchmark for comparing the performance of storage systems that expose SPARQL endpoints ( does not compare the reasoning capacity)
    timeline: 2008

  3. University Ontology Benchmark (UOBM) from IBM

    Li Ma, Yang Yang, Zhaoming Qiu, GuoTong Xie, Yue Pan, Shengping Liu. Towards a Complete OWL Ontology Benchmark. In Proceedings of ESWC'2006. pp.125~139

    Compares the RDF stores along with its reasoning capability (OWL DL and OWL Lite).
    timeline:2006

  4. Survey paper : What's wrong with OWL Benchmarks?
    Finds (rightly) fault with above (1 and 3) benchmarks and outlines the requirements that if implemented in a benchmark shall ideally cover real world scenarios.


    This list is no way near exhaustive, so I plan to extend it whenever I can. Please drop comments if you have any...
Ta..

First post

Welcome to my Blog!