Different use cases mainly, machine learning for instance.
Xapian is an Open Source Search Engine Library, released under the GPL v2+. It's written in C++, with bindings to allow use from Perl, Python 2, Python 3, PHP 5, PHP 7, Java, Tcl, C#, Ruby, Lua, Erlang, Node.js and R (so far!)
Xapian is a highly adaptable toolkit which allows developers to easily add advanced indexing and search facilities to their own applications. It has built-in support for several families of weighting models and also supports a rich set of boolean query operators.
Features
Noteworthy features of Xapian include:
Free Software/Open Source - licensed under the GPL.
Supports Unicode 9.0 (including codepoints beyond the BMP), and stores indexed text in UTF-8.
Highly portable - runs on Android, Linux, macOS, FreeBSD, NetBSD, OpenBSD, Solaris, HP-UX, AIX and probably other Unix platforms; as well as Microsoft Windows and OS/2.
Written in C++, with bindings allowing use from many other languages.
Ranked search (so the most relevant documents are more likely to come near the top of the results list) with built-in support for multiple models from the Probabilistic, Divergence from Randomness, and Language Modelling families of weighting models. Custom user-supplied weighting models are also supported.
Relevance feedback - given one or more documents, Xapian can suggest the most relevant index terms to expand a query, suggest related documents, categorise documents, etc.
Phrase and proximity searching - users can search for words occurring in an exact phrase or within a specified number of words, either in a specified order, or in any order.
Full range of structured boolean search operators ("stock NOT market", etc). The results of the boolean search are ranked by the weighting model, and boolean filters can also be applied (which don't themselves contribute to a document's weight).
Supports stemming of search terms (e.g. a search for "football" would match documents which mention "footballs" or "footballer"). This helps to find relevant documents which might otherwise be missed. Snowball stemmers are currently included for Arabic, Armenian, Basque, Catalan, Danish, Dutch, English, Finnish, French, German, Hungarian, Indonesian, Irish, Italian, Lithuanian, Nepali, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish, Tamil and Turkish.
Wildcard search is supported (e.g. "xap*").
Synonyms are supported, both explicitly (e.g. "~cash") and as an automatic form of query expansion.
Dynamically generated snippets from matching documents can be generated, with matching words, phrases and wildcards highlighted.
Xapian can suggest spelling corrections for user supplied queries. This is based on words which occur in the data being indexed, so works even for words which wouldn't be found in a dictionary (e.g. "xapian" would be suggested as a correction for "xapain").
Faceted search is supported.
Supports database files > 2GB - essential for scaling to large document collections.
Platform independent data formats - you can build a database on one machine and search it on another.
Allows simultaneous update and searching. New documents become searchable right away.
As well as the library, we supply a number of small example programs, and a larger application - an indexing and CGI search application called Omega:
The indexer supplied can index HTML, PHP, PDF, PostScript, LibreOffice/OpenOffice/StarOffice, OpenDocument, Microsoft Word/Excel/Powerpoint/Publisher/Visio/Works/XPS, Microsoft Outlook saved messages, Apple iWork, Word Perfect, AbiWord, RTF, DVI, Perl POD documentation, CSV, SVG, reStructured text, markdown, MAFF, MHTML, ATOM feeds, dejavu, RFC822 mail messages (.eml), vCard, RPM packages, Debian packages, and plain text. Adding support for indexing other formats is easy where conversion filters are available. This indexer works using the filing system, but we also provide a script to allow the htdig web crawler to be hooked in, allowing remote sites to be searched using Omega.
You can also index data from any SQL or other RDBMS supported by the Perl DBI module. That includes MySQL, PostgreSQL, SQLite, Oracle, DB2, MS SQL, LDAP, and ODBC.
CGI search front-end supplied with highly customisable appearance. This can also be customised to output results in JSON, XML or CSV, which is useful if you just want raw search results which you can process in your own page layout code for dynamically generated pages, or for integrating search into an AJAX front-end.
If you're after a packaged search engine for your website, you should take a look at Omega: an application we supply built upon Xapian. Unlike most other website search solutions, Xapian's versatility allows you to extend Omega to meet your needs as they grow.
The latest stable version is 1.4.19, released on 2021-12-31.
The latest old stable version is 1.2.25, released on 2017-09-26.