When community really matters – security report turned into release 1.1.2 in only 14 hours
The Foswiki project has just been through one of the rare situations where one of our users discover a serious security issue in our software.
It was one of those moments where many developers were thinking: “Why didn’t I see this?”. Many of us felt embarrassed.
It is a fact of life that when you have humans developing technology, things can and will go wrong. You can do a lot of minimize the risk, and the Foswiki project has conducted serious security reviews on the extensive rewrite of our code that took place from 1.0 to 1.1. And we have found and fixed issues before we even got close to releasing any code to the public, and the Foswiki project has had am impressive security record.
But the 1.1.0 release we missed a small issue where one code line had been moved down a few lines too many and we ended up not authenticating the user properly in a specific situation.
This could be a sad story but it isn’t. This situation showed what difference it makes to have a large and strong community behind an open source project.
First I want to give the timeline of what the events on the 9th and 10th of November.
- 09 Nov 2010 a little before noon CET a user asks a very good question on the Foswiki support web. He could not understand what he did wrong in setting up access rights because no matter how he did it, he was able to edit and save preferences in topics he did not have access rights to.
- Within a few hours a Foswiki developer read the support question. And to his horror the report was correct. The minute he realized the nature of the problem the support question was changed so it could only be viewed by the reporter and the Foswiki Security Task Team
- The next few hours the Foswiki security mailing list, which only the security team has access to, is glowing with emails. Five hours after the issue was reported a code fix has been made and tested by the security team members. The time is now near 18:00.
- At 18:00 I become aware of the situation and the security team quickly assess the issue and conclude that it is a security level 2 issue. This means that we have a goal to respond within 48 hours and will provide a fix and a security alert which will be provided to the people that subscribe to our low traffic announcement mailing list.
- It is decided that we will release a 1.1.2 as fast as we possibly can. We assess which urgent bugs we want to include in addition to the security fix.
- The next 8 hours are amazing. Many members of the security team works all night fixing and testing a short list of important bugs. A test of a new version of the Wysiwyg editor is conducted and merged in. Code is checked in and reviewed and tested at a rapid pace. Code is even thrown out again because it was decided to be too risky. Additional developers are pulled into the IRC channel we have created for the event to get the last review on code fixes.
- At 19:30 I request a CVE number from MITRE. And they are fantastic at MITRE. We have the number already 20:00.
- At 23:00 the security announcement and response plan is written and is being reviewed and agreed by the security team.
- At 02:00 the 10 Nov 2010 the 1.1.2 release is built and uploaded to our servers for download.
- At 02:30 the release announcement is sent to the Foswiki announcement mailing list
- At 02:35 the security announcement is sent to the Foswiki announcement mailing list
So we actually managed to react, assess, fix, finish a release, build release, and announce within 14 hours.
As I write this, I have just sent the the security announcement to the public security sites (48 hours after we did the announcement on the Foswiki mailing lists).
When I think back now on what happened I feel proud.
Proud that we managed to act like professionels even though we all work as volunteers on the project.
Proud because I am part of such a strong development team that care about the security of our users, and care about each other.
Proud because as a release manager located in Denmark I had people in USA, Mexico, Germany, France, UK, Australia, Netherlands being available helping, testing, coding, reviewing, and encouraging. I had a team backing me up getting the release out and getting the announcements out.
It shows the power of open source projects when they are supported by a committed and dedicated community.
Kenneth Lavrsen
Foswiki Release Manager
Leader of the Foswiki Security Task Team
Member of the board of the Foswiki Association
