Official Foswiki Weblog

One of the great strengths of Foswiki is that it stores topics in plain text files. This feature has proven its worth time and again, as Foswiki is able to quickly adapt to organisational change, easily support external processes that write to topics, and is quickly accessible for emergency measures. For an admin who has login access to the server, it’s very easy to use the command-line to make bulk changes to many topics simultaneously.

In Foswiki 1.1.4 we addressed a nasty bug that was inherited from the very earliest days of TWiki. This bug could result in serious history errors such as the wrong username being attributed to edits, incorrect revision numbers, and broken revision histories. Unfortunately there was a side-effect of this fix that anyone who modifies topics on the server needs to be aware of, and take action on.

Any process (or person) that changes a .txt file on the server must make sure that the corresponding .txt,v file is left with a file date that is equal to or more recent than the .txt file.

If you don’t do this, Foswiki will be forced to constantly re-read the .txt,v for the correct revision history, which can affect performance. Also, when such a topic is edited, Foswiki will create a new revision attributed to the “UnknownUser” to record that the topic was changed outside of Foswiki’s control. This is the correct behaviour in the general case, and will save less experienced users a lot of pain. However more experienced users may know that their changes are ’safe’ and that no change to the topic history is required. In this case, it is trivial to update the file date on the .txt,v file. In summary,

• After making any manual change to a .txt file on the server make sure that the corresponding .txt,v is ‘touched’ (use the UNIX ‘touch’ command, or simply text-edit and save the file)
• Scripts that make bulk changes to files – and external processes that write .txt files – should be modified to ‘touch’ the .txt,v files after writing.
• If you have already made such a change but can’t remember which files you changed, you can use the following linux command line to ‘touch’ all .txt,v files (this is perfectly safe)

find /path/to/foswiki/data -name '*.txt,v' -exec touch \{\} \;

Crawford Currie Development

We’ve carried on with the Foswiki Camp today, and wow, has it been a productive camp!

Andrew Jones has done an amazing job leading the effort on using PSGI / Plack to replace the low-level Foswiki engine, for simplified installation and ease of support across a wide range of platforms.

Andrew has also taken significant steps towards simplifying the installation documentation for new users.

Michael Daum has led the development of a super new plugin that helps admins find out what extensions can be upgraded on their installs. A bit of documentation and it’ll be ready for prime-time.

Olivier Raginel has led the move to git – we’ve managed to convert all the hooks, set up the DNS and otherwise get lots of preparation done.

Arthur Clemens has implemented support for displaying each search result differently, based on any attribute of the topic the result was found in. This makes the presentation of search results much sexier.

Arthur has also done a power of work on the awesome new base skin.

Padraig Lennon has led the development of a new “WikiApps” web on foswiki.org that helps people share and explore wiki applications in a simple and effective way.

Everyone at the camp has been fully engaged and productive – even to the extent of turning down free beers in order to keep on working!

Crawford Currie Development

We’ve been having fun at the 2011 Foswiki Camp! As well as some great discussions (and surprisingly few arguments) we have turned our hands to some hardcore hacking. Here’s some of what we’ve been working on:

• A new skin that will provide a strong base for the default install, and a good platform to build more sophisticated skins on. We’re applying learnings from many different skin developments to create a modern, HTML5-ready, easy to extend, look and feel.
• An exciting new methodology for packaging wiki applications, including a “play pen” which allows end users to try out applications in their own safe environment on foswiki.org, before deciding whether to install or not.
• Simplified and refined installation documentation for the core.
• First steps in replacing the Foswiki::Engine and low level CGI code, with the modern web ‘Plack’ framework (the Perl equivalent of Ruby’s excellent ‘Rack’).
• A clever new plugin that will notify foswiki admins when extension updates become available, that affect their own installations, saving them from missing out on important releases.
• A nice mind-map

What we haven’t started yet, but hope to get to tomorrow:

• Help people to share their own wiki applications on foswiki.org, in an easily accessible way.
• Ease the handling of attachments, supporting multi-upload using drag and drop, pagination through large lists, and download of zip archives of multiple attachments.
• Attack the “unicode” branch with interactive testing, to see how big the hole is.

Other messages that have come from the last couple of days include:

• We want to make Foswiki something that our users never grow out of, and that means keeping up with the technology curve. Store2, unicode, and Plack are all exciting advances that help us enormously.
• Continuity – ensuring safety of data and applications – always has been, and always will be, incredibly important to us.
• We really need to have more of these camps! The ideas exchanged, and the work done, have been incredible. The more the merrier!

Crawford Currie Development

The Foswiki Association is pleased to announce its 3rd General Assembly and Birthday Celebration, 19th-20th November 2011, at CERN Geneva.

• http://foswiki.org/Community/AgendaThirdGeneralAssembly

This is a great opportunity to meet some of the faces behind the IRC handles, to talk about Foswiki, and an excuse to visit the lovely Geneva area.

All association members are very welcome to attend. We’d love to meet you in person, but if you can’t make it physically we’ll be setting up channels to help you join in virtually. If you’re not a member of the Foswiki association yet, see http://foswiki.org/Community/CurrentFoswikiMembers – all are welcome.

We’re also excited to be holding our first one-day FoswikiCamp on the 18th November 2011 at CERN Geneva (the day before the GA). The plan is to have one or more fun “sprints”to scratch some itches that have been bothering us. See

• http://foswiki.org/Community/FoswikiCamp

The GA is open to all association members, and the camp is open to anyone. Due to access controls at CERN we need to know in advance if you are coming, so please add your name to the relevant lists in the Foswiki topics linked above.

(We might even have a Foswiki release 1.1.4 to celebrate too!)

Update: Every year on the General Assembly people have the opportunity to become a Foswiki Member. This is a great way to help Foswiki to grow and gives you the right to vote on any decisions to be made on a General Assembly as well. To apply for membership add yourself to http://foswiki.org/Community/CurrentFoswikiMembers#Membership_Applications and contact two of the current Foswiki Members to support your application.

Crawford Currie Event

For a long time we’ve talked about stitching a relational database into Foswiki, mainly to speed up query searches. To keep things simple Foswiki uses a text database, which stores topics in lots of individual files. By default searches are implemented by searching every one of these files for matching text. This isn’t as bad as it sounds; searching is quite fast, but doesn’t scale up terribly well, so people have coupled in various full-text search engines, such as Kinosearch, to accelerate text searches. However there’s been very little successful work yet on speeding up query searches – which is a shame, because query searches are heavily used in most wiki applications. The problem has always been that it’s difficult to map from a Foswiki search query to any kind of structured search language, such as SQL.

One approach (championed by WikiRing member Sven Dowideit) has been to support a wider range of queries in the text search engine. Using the code I wrote to map query searches to regular expressions, Sven has been able to couple Foswiki to the MongoDB database, with some pretty impressive results. This is already available as the MongoDBPlugin. Other approaches have tried to map Foswiki queries to other database search models, such as XQuery.

SQL is still the most mature query language, and the RDBMS that support it are among some of the most mature and robust software on the planet. It’s important that Foswiki has solutions that fall into that comfort zone. To that end I have implemented a new module, the DBIStoreContrib, that maps from the Foswiki query language to SQL. This allows us to use many of the existing RDBMS, such as MySQL, SQLite and PostgreSQL, to accelerate queries. By caching topics into the database in parallel to the text database, we avoid the risk of data loss that has made people nervous of other wiki solutions that use RDBMS, such as Mediawiki, while still maximising search performance.

It’s early days for this module, and there’s still lots to do to optimise how it works, so watch this space! If you are interested, and have a development environment, then you can check out the DBIStoreContrib from the Foswiki Subversion trunk.

Foswiki – the very best wiki application platform – is taking another giant leap forward….

Crawford Currie Extensions database, Development, sql, storage