Revision 4 as of 2013-01-25 19:23:49

Clear message

Wiki Attack January 2013

Summary

The python.org wikis for Python, Jython and the Python Software Foundation (PSF) were subject to a security breach and later attack which caused all of the wiki data to be destroyed on January 5 2013.

An analysis of the incident revealed that an exploit had been planted on our servers possibly as early as July 25 2012, which allowed arbitrary execution of code under the user running the MoinMoin wiki.

It is likely that the password information was downloaded from the server in the course of the security breach, so we recommend changing your passwords immediately, if you have used the same password for other services as well.

During the recovery in the weeks after the attack, we reset all passwords on the wiki server to make sure that users do change their passwords. You can use the password recovery function of the resp. wiki to reset your password. The function is shown when clicking on the login link.

Moving forward, we will no longer allow changing wiki pages without login and have put additional security measures in place to prevent attacks like the one we came under. The PSF has also funded the change of the MoinMoin code base to include support for the passlib library, which provides much better means of protecting password information on the server than the SHA-1 based hash scheme used before MoinMoin 1.9.6. This new support will be available in MoinMoin 1.9.7.

Please report any problems you find with the wiki to the mailto:pydotorg-www@python.org mailing list.

Attack Analysis

In the week of January 14 2013, MarcAndreLemburg ran a longer analysis of the attack on the wiki VM. This is a summary of the things he found.

The attack used on the wiki was apparently the same as the one which hit Debian:

http://wiki.debian.org/DebianWiki/SecurityIncident2012

Someone used the https://security-tracker.debian.org/tracker/CVE-2012-6081 vulnerability to upload an action plugin to the server called moinexec.py, which was then subsequently used to run commands on the moin user account.

The few logs that were still accessible showed that someone tried sudo on Dec 28 2012, but without success.

On Jan 5 at 00:10 GMT, someone else ran the "rm -r *" which resulted in all files owned by the moin user to get deleted.

The VM was rebooted on Jan 7, apparently in an attempt to get things working again. The result of this was that the file system was remounted, clearing most of the transaction log, which could have been used to recover the deleted files. The VM image he was working off was created after the reboot.

Of course, without the moin user files available, getting a better picture of what happened was difficult. With the knowledge of the Debian analysis, he found a bytecode file of the uploaded plugin code in the VM image and this showed a PYC time stamp of Wed Jul 25 16:08:14 2012 GMT, closely matching the date given in the Debian analysis.

We were subsequently approached by the person who ran the rm -r *, so we know now that the original attack was performed by different people, most probably the same that attacked the Debian wiki. It is also obvious that the people who installed the plugin, had different intentions than causing easy to detect damage on the system.

Since the logs on the VM only go back 5 (!) days for server web logs and 7 (!) days for system logs, it was impossible to determine the amount of information leakage caused by the attack.

It is likely that the passwords and user configuration details were inspected in the same way as was done for the Debian wiki.

Content Recovery

Since we were not in the comfortable position to use a backup for restoring the wiki content (our most recent backup dates back to June 24 2012), Marc-Andre tried to get as much information from archive.org, the Google cache and the Yahoo/Bing cache as possible.

It turned out that the Google cache was unusable for the task due to a surge protection on their site. Yahoo/Bing worked and results in quite a few more recent edits/updates. The archive.org pages included snapshots from end of Nov 2012, but they apparently don't run complete dumps of the sites they archive (and since moin has a surge protection built-in as well, it's not easy for such archives to crawl the wiki).

Overall, we have been able to recover around 200 pages that were created/edited after the June 2012 backup date. Not really all that much, given the amount of data in the wiki, but still better than nothing.

He then ran ext4image and photorec on the VM image in several combinations and was able to restore a number of files. Unfortunately, important meta data such as the file date/time and name was lost. This made restoring from the files very difficult.

Moin stores the wiki pages in plain text files and keeps all revisions in the file system. Furthermore, there is no meta information inside the text files, so only the headers from the file content can be used as indication of which page it belongs to and even then, you get multiple copies of the same page, with no indication as to which of those to regard as most current version.

Still, the restored files do contain pages that were not available in the archive web sites, so if there are important pages that need to be restored, he can manually try to extract the relevant data and re-add it to the wikis. For the PSF wiki, this is the only way to recover files, since there are no web archives available for it.

Additionally, none of the newer attachments to wiki pages could be restored.

Content Restoration

In the week of January 21, Marc-Andre then worked together with Reimar Baur to prepare the archive downloads for reintegration into the wiki.

Reimar adapted a script he had written earlier, which is able to convert HTML pages into wiki markup, to turn the HTML archive dumps back into wiki markup. He also write a script which uses the MoinMoin API to readd the pages with proper timestamps and editor information back into the MoinMoin database.

In the meantime, Thomas Waldmann, one of the MoinMoin developers, was working on adding passlib support to moin 1.9. The work was funded by the Python Software Foundation (PSF).

Noah Kantrowitz had setup a new VM for the wiki, which Reimar and Marc-Andre then configured to use the newly added passlib support in moin. After an additional security audit on the configuration and changes to harden the installation, Reimar's script was used to readd the archive dumps on top of the June 24 2012 backup.

-- Python Software Foundation

Unable to edit the page? See the FrontPage for instructions.