Foswiki project blog

Extensions Archive

You are here: Blog » Extensions

Extensions

419970 33314492.jpgFor a long time we've talked about stitching a relational database into Foswiki, mainly to speed up query searches. To keep things simple Foswiki uses a text database, which stores topics in lots of individual files. By default searches are implemented by searching every one of these files for matching text. This isn't as bad as it sounds; searching is quite fast, but doesn't scale up terribly well, so people have coupled in various full-text search engines, such as Kinosearch, to accelerate text searches. However there's been very little successful work yet on speeding up query searches – which is a shame, because query searches are heavily used in most wiki applications. The problem has always been that it's difficult to map from a Foswiki search query to any kind of structured search language, such as SQL.

One approach (championed by WikiRing member Sven Dowideit) has been to support a wider range of queries in the text search engine. Using the code I wrote to map query searches to regular expressions, Sven has been able to couple Foswiki to the MongoDB database, with some pretty impressive results. This is already available as the MongoDBPlugin. Other approaches have tried to map Foswiki queries to other database search models, such as XQuery.

SQL is still the most mature query language, and the RDBMS that support it are among some of the most mature and robust software on the planet. It's important that Foswiki has solutions that fall into that comfort zone. To that end I have implemented a new module, the DBIStoreContrib, that maps from the Foswiki query language to SQL. This allows us to use many of the existing RDBMS, such as MySQL, SQLite and PostgreSQL, to accelerate queries. By caching topics into the database in parallel to the text database, we avoid the risk of data loss that has made people nervous of other wiki solutions that use RDBMS, such as Mediawiki, while still maximising search performance.

It's early days for this module, and there's still lots to do to optimise how it works, so watch this space! If you are interested, and have a development environment, then you can check out the DBIStoreContrib from the Foswiki Subversion trunk.

Foswiki – the very best wiki application platform – is taking another giant leap forward….

Tonight I have released the new and improved Kino Search extensions for Foswiki.

Kino Search is a Perl port of the Lucene indexed based search engine from Apache. It provides a very fast full text search across large amounts of data, plus the ability to use Google style operators.

The Foswiki implementation is made up of the KinoSearchContrib and the KinoSearchPlugin. Not only does this provide indexed search for topics, it also has the ability to index a number of attachments, including MS Office 2003 and 2007 documents and PDF's.

The KinoSearchContrib is the workhorse, providing the indexer and a script for the searching. The KinoSearchPlugin is a thin wrapper around it that hooks into Foswiki to provide a %KINOSEARCH% macro and support for indexing on immediately when a topic has been saved.

There are a number of improvements over the previous version of Kino Search for Foswiki, including:

  • Fast CGI and mod_perl support
  • Indexing MS Office 2007 files
  • Much more resilient indexing
    • In particular when indexing password protected files or when an incorrect file type is supplied (such as attaching a image and calling it myfile.txt)
  • Using more of the official Foswiki API
  • Improved documentation and templates

If your Foswiki install has grown to a size where the performance of the default search is starting to suffer then you will need this extension. Of course smaller wikis will also benefit from the attachment search.

The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License
This page was cached on 23 Sep 2017 - 05:27.