The Python Package Index (PyPI), home to more than 455,000 Python code repositories, caged itself to new users and their projects over the weekend because it could not deal with a rush of efforts to create malicious accounts and code libraries.
“The volume of malicious users and malicious projects being created on the index in the past week has outpaced our ability to respond to it in a timely fashion, especially with multiple PyPI administrators on leave,” the package registry said in a status update on Saturday.
Software developers routinely rely on package registries to download modular code packages that perform useful functions. These registries, like PyPI, npm, and RubyGems, have become popular targets for software supply chain attacks that aim to compromise widely used packages and the applications and users that depend upon them.
Essentially, you really don’t want malicious users to get their malware and fake libraries into popular registries, as that would lead to unsuspecting developers poisoning their apps and users with bad dependencies. Someone has to filter out the nasty code from the good stuff.
The problem at PyPI was not so much a surge of fake accounts and subverted packages, though the tide of dubious stuff did rise from the typical rate of about 20-30 reports per day to about 40 per day over the weekend. Rather, the staff who usually vet suspect submissions had ebbed to a single person who felt unable to adequately respond.
Once again we’re reminded of XKCD.
Ee Durbin, director of infrastructure at the Python Software Foundation, told The Register in a phone interview that what happened had more to do with reduced resources than increased malware.
“What was different is that there’s a team of four PyPI Admins,” said Durbin. “Three of us take part in responding to malware reports, and we’re fairly diligent and pretty quick about those. Our goal is generally to take them down within 24 hours. But more realistically, it’s generally within one to six hours. The reason for this is that the longer they sit out there, the more of a threat they are, and just generally, we want to be responsive.”
Over the past two weeks, two of the three people who respond to incidents were on leave at some point. That left Durbin and occasionally another admin to field every security report.
“During that time, I noticed a lot more automation was going on,” explained Durbin, referring to both automated account creation and automated package submission.
“And it was just getting to the point where I didn’t feel confident that I as an individual was going to be sitting here all weekend watching that inbox. So you know, effectively it was I was burnt out after two weeks of doing it. I did a quick check with the rest of the team to make sure they felt like it was okay. And then I pulled that lever so that I wouldn’t feel personally responsible.
“The issue was that literally with the automations they had in place, as soon as I took something down, they would replace it with something else. And so it was just like, ‘I’m not gonna I’m not gonna sit here and play Whac-a-Mole.'”
Speaking of software supply-chain shenanigans, security firm Check Point last week flagged up the Microsoft Visual Studio Code Extension Marketplace – a repository for official and third-party add-ons for the code editor – for hosting a handful of malicious extensions.
One, named “Theme Darcula dark,” an apparent info stealer that purported to offer a way to adjust the differently named Dracula color scheme, was found to have more than 45,000 installations. Another, named “python-vscode,” was found to have a suspicious code injection pattern but couldn’t conclusively be determined to be hostile.
The Microsoft Visual Studio Code team reportedly removed the suspect extensions last week.
Maintainer burnout is a long-standing problem in the open source community, one generally dealt with by recognizing that more resources – in terms of people and often funding – need to be directed at affected projects.
As of Monday, there are once again three people fielding community reports, which is why PyPI has now resumed letting people create new accounts and upload new packages.
Durbin said there’s some good news to report. There’s a security-developer-in-residence coming to the Python Software Foundation (PSF) soon, for a year, thanks to funding from the OpenSSF and the Linux Foundation. That job offer, we’re told, is supposed to go out today.
And the PSF aims to fill another position focused specifically on security concerns related to PyPI. That’s to be funded by AWS and another organization that haven’t officially been announced as negotiations have yet to be completed.
“One of the projects they’re going to be working on is building us out to the point where we have automation-friendly ways of responding to these [malware reports],” said Durbin, who explained that the system needs to be able to handle scenarios like deletion rollbacks so that the consequences of incorrect reports can be undone if needed.
“I don’t think we’ll get to the point where it will be fully automatic for everything, just because that is just a recipe for bad days.” ®
