Blog

On February 8th, 2011 Six Feet Up applied more than 70 software patches to help organizations address the recent critical security vulnerability announced in the open source content management system (CMS) Plone.

The security issue unveiled on February 3rd was one of the most critical issues ever found in Plone, a mature CMS that has been around for more than 10 years. According to the Plone Security Team, who found and reported the issue, anonymous users could elevate their privileges to the manager role, the equivalent of giving someone complete administrative access to the site. This allowed anonymous users to have access to viewing unpublished content, creating new content and modifying a site's look and feel.

Because Plone sites were at a high risk of getting compromised as soon as the patch was made available, Six Feet Up project managers carefully planned the "Patch Up" effort for 2 days prior to the work to minimize organizations' exposure to the vulnerability and site downtime. Our PMs scheduled and coordinated activities using both a Gantt chart in OmniPlan and Trac tickets assigned to each developer, sysadmin and tester.

Patching activities were divided into two rounds of work: 1) disable authentication as late as possible (i.e. as close to the patch release time as possible), 2) apply the fix as fast as possible after the fix is made public. Both rounds included multiple verification steps and a pass by our Quality Assurance Team. Six Feet Up developers and sysadmin worked together on creating a wiki page listing the 20+ steps to follow to apply and test the patch, and a training session was held two days prior to the event with the entire team to make sure to address questions proactively.

The effort was considerable because the patching activities spanned all the various versions of Plone (2.5x, 3.0x, 3.1x, 3.2.x, 4). The team addressed various deployment styles ( buildout and old style Zope installations) on multiple operating systems ranging from FreeBSD, to CentOS, Debian, and Red Hat. Deployments were obviously trickier for hosting clients who do their own development, yet the team addressed all configurations, from one basic instance all the way to a setup of 24 instances across 4 servers.

Thanks to the careful planning, the team was able to apply 70+ patches successfully, including some for highly visible sites such as Discover Magazine, and addressing some last minute client requests for assistance. A few roadblocks were hit when a server wouldn't restart, or when funkload tests wouldn't pass, but, altogether, the process went smoothly and our clients were very happy.