OWL benchmarks

There is a great deal of confusion about the tools to realize semantic web applications. This confusion stems from the fact that there are a large number of tools available and the benchmarks that compare these tools are quite diverse.

For the purpose of the clarity, I will summarize these tools and benchmarks here. I intend to comment on them as much as I can during next few weeks.

(As of January 2008)

OWL reasoners: ( for reasoning, inferencing)

1. Pellet - free/open source with some restriction on commercial use - Supports OWL DL, DL Safe SWRL rules.
   Java based
   
2. RacerPro - Educational free evaluation licence, commercial paid licence - Supports OWL DL
   Java based
   
3. KAON2 - Free for educational and non-academic use, different licence for commercial use. Supports OWL-DL and SWRL with F-Logic ontologies
   Java based
   
4. FaCT++ - seems free,open source
   C++ based
   
5. OWL API - free, open source. This is actually OWL API with reasoner interface to FaCT++ and Pellet.
   Java based
6. OWLIM - free version with limitations, commercial licence is with all features
   Supports RDFS and most of OWL-Lite. More importantly contains a semantic repository over Sesame.
   Java based

RDF Stores (for storing RDF triples)

1. Virtuso from OpenLink
   Getting started link is here
2. AllegroGraph from Franz Inc.
   Java/Lisp based
   Java learning centre is here
3. Mulgara (previously Kowari)
4. SwiftOWLIM and BigOWLIM from ontotext
   SwiftOWLIM is an open-source library, under LGPL
   BigOWLIM is available under an RDBMS-like commercial licence on a per-server-CPU basis
5. Sesame RDF store from Aduna
6. Oracle 11g has RDF store support
7. Scalable Ontology Repository (SOR) [Previously Minerva] from IBM. It supports DLP (Description Logic Program), a subset of OWL DL.
   Part of the IBM Integrated Ontology Development Toolkit.
8. D2RServer is a a tool for serving Linked Data views on relational databases.D2R server relies on a declarative mapping between the schemata of the database and the target RDF terms. Based on this mapping, D2R Server serves a Linked Data view on your database and provides a SPARQL endpoint for the database.

Bechmarks to evaluate these reasoners/storage systems:

1. Lehigh University Benchmark (LUBM)
   
   Y. Guo, Z. Pan, and J. Heflin. LUBM: A Benchmark for OWL Knowledge Base Systems. Journal of Web Semantics 3(2), 2005, pp158-182.
   
   One of the early works on the subject, however now seems outdated. Good reference point for any future comparisons.
   
   Compares the RDF stores.
   timeline: 2004/2005
   
   
2. Berlin bechmark
   from Freie Universität, Berlin
   Benchmark for comparing the performance of storage systems that expose SPARQL endpoints ( does not compare the reasoning capacity)
   timeline: 2008
   
   
3. University Ontology Benchmark (UOBM) from IBM
   
   Li Ma, Yang Yang, Zhaoming Qiu, GuoTong Xie, Yue Pan, Shengping Liu. Towards a Complete OWL Ontology Benchmark. In Proceedings of ESWC'2006. pp.125~139
   
   Compares the RDF stores along with its reasoning capability (OWL DL and OWL Lite).
   timeline:2006
   
   
4. Survey paper : What's wrong with OWL Benchmarks?
   Finds (rightly) fault with above (1 and 3) benchmarks and outlines the requirements that if implemented in a benchmark shall ideally cover real world scenarios.
   
   
   This list is no way near exhaustive, so I plan to extend it whenever I can. Please drop comments if you have any...

Ta..