Home > Development > Foswiki 1.1.4 heads-up on manual changes to .txt files

Foswiki 1.1.4 heads-up on manual changes to .txt files

January 25th, 2012

One of the great strengths of Foswiki is that it stores topics in plain text files. This feature has proven its worth time and again, as Foswiki is able to quickly adapt to organisational change, easily support external processes that write to topics, and is quickly accessible for emergency measures. For an admin who has login access to the server, it’s very easy to use the command-line to make bulk changes to many topics simultaneously.

In Foswiki 1.1.4 we addressed a nasty bug that was inherited from the very earliest days of TWiki. This bug could result in serious history errors such as the wrong username being attributed to edits, incorrect revision numbers, and broken revision histories. Unfortunately there was a side-effect of this fix that anyone who modifies topics on the server needs to be aware of, and take action on.

Any process (or person) that changes a .txt file on the server must make sure that the corresponding .txt,v file is left with a file date that is equal to or more recent than the .txt file.

If you don’t do this, Foswiki will be forced to constantly re-read the .txt,v for the correct revision history, which can affect performance. Also, when such a topic is edited, Foswiki will create a new revision attributed to the “UnknownUser” to record that the topic was changed outside of Foswiki’s control. This is the correct behaviour in the general case, and will save less experienced users a lot of pain. However more experienced users may know that their changes are ‘safe’ and that no change to the topic history is required. In this case, it is trivial to update the file date on the .txt,v file. In summary,

  • After making any manual change to a .txt file on the server make sure that the corresponding .txt,v is ‘touched’ (use the UNIX ‘touch’ command, or simply text-edit and save the file)
  • Scripts that make bulk changes to files – and external processes that write .txt files – should be modified to ‘touch’ the .txt,v files after writing.
  • If you have already made such a change but can’t remember which files you changed, you can use the following linux command line to ‘touch’ all .txt,v files (this is perfectly safe)
find /path/to/foswiki/data -name '*.txt,v' -exec touch {} ;

Development

  1. January 25th, 2012 at 11:57 | #1

    For those interested, we're tracking the performance concern at http://foswiki.org/Tasks/Item11454

  2. January 25th, 2012 at 14:00 | #2

    If you want to simply use the .txt date as a base for the .txt,v, you can use the -r option of touch, if your system supports it. So for a single files, that is:

    touch -r $file ${file},v where $file is the path to YourTopic.txt

    To do all at once, easiest is to use perl:

    perl -MFile::Find=find -wle'find(sub{/^(.*.txt),vz/&&-f&&system("echo touch -f $1 $_")},@ARGV)' /path/to/foswiki/data

  3. WilliamMorgan
    February 7th, 2012 at 04:05 | #3

    Thanks for the update. works great on my system.

    I saw an old post referring to the idea of creating a database for foswiki – one which would retain the simplicity of text files while utilising indexing capabilities. Is this idea progressing?

  4. February 22nd, 2012 at 13:48 | #4

    Hi William, http://trin.org.au is using the experimental (see http://wiki.trin.org.au/System/MongoDBPlugin ) already. This relies on trunk-only features, though, so won't work on Foswiki 1.1. We are running with over 200,000 topics and SEARCHes in webs of over 60,000 topics are quite snappy. I think the largest web to demonstrate is a SEARCH-powered jqgrid of 25,000 topics at http://wiki.trin.org.au/Lauries/ – keep in mind that filtering uses regex-searching, which is much, much slower than boolean type searchers.

    There are plans to freeze trunk soon so we can concentrate on releasing the work in trunk properly, http://foswiki.org/Blog/Release_1_1_5_planned__Br

  1. No trackbacks yet.