Packaging improvements that could be funded

This page lists specific things that

  1. the Python packaging community wants
  2. are fairly well-scoped
  3. would happen much faster if the Packaging Working Group got funding to achieve them (through donations or grants/directed gifts)

Foundational tool improvements

Finish dependency resolver for pip

We're partway through a next-generation rewrite of the dependency resolver within pip, Python's package download and installation tool. The project ran into massive technical debt, but the refactoring is nearly finished and prototype functionality is in alpha now. (In-depth explanation by Sebastian Awwad of the problem & our approach, lead developer Pradyun Gedam's initial plan, 2017 status updates, and June 2019 status update, GitHub issue #988 tracking progress and issue #6536 for planning rollout.)

Funding would support user experience, communications/publicity, and testing work (including developing robust testing/CI infrastructure) as well as core feature development and review.

We need to finish the resolver because so many other improvements are blocked on it:

and it would fix so many dependency issues for our users:

And in our larger ecology, this causes installation problems for:

Better specifications, toolchain, and services for building distributions

PyTorch, TensorFlow, and many other Python packages (especially science packages) suffer from cross-platform installability problems, which affect both users and developers. Packagers and users prefer using built distributions (usually in the wheel format); publishing built distributions increases convenience for end users because source code is pre-compiled, which significantly reduces install time (e.g., from 10+ minutes to several seconds).

Supporting the multifarious Linux platforms is something we've been lagging on; we are still finishing up the rollout of manylinux2010 and recently approved the new standard manylinux2014. But even so, packagers will have to build their own wheels to release packages, which can be fiddly, brittle, and time-consuming.

We'd like help to:

We need funding for specification research and writing, backend and frontend development, testing, DevOps/infrastructure/platform services, user experience work, technical writing for end users, project management, and community outreach.

Robust interoperability testing

We need funding to ensure core packaging tools work well with each other; currently they aren't seamlessly interoperable. See the integration-test project. This will help us get faster at testing and rolling out bugfixes and features for all Python packaging and distribution tools: well-known projects like pip, virtualenv, and wheel, but also all the downstream projects that depend on them.

Make setuptools the reference implementation of the distutils API

There is a part of the Python standard library called distutils, and some users directly use it. We want users to instead switch to the supported toolchain, which uses setuptools. This requires backend development work, technical writing, and publicity work within Python's community.

Improve pip user experience

pip's user experience needs to become more consistent across features, fit the user's mental model better, reduce unintended data loss, and provide better error messages and prompts, logs, output, and reporting. pip's maintainers have a list of TODOs and need funding so that user experience researchers, UX designers, developers, and technical writers can spend dedicated time addressing them.

Improve user experience of packaging

User experience research and UX and development implementation work would make it easier for packagers to create configuration files.

Audit and update package metadata

If we audit and update PyPI metadata for existing projects based on already-uploaded artifacts, we can publish information about what packages depend on each other and on certain environments, and ensure a high-quality API for many tools to reuse and build upon. The current PyPI upload API relies on the upload client extracting the metadata and supplying it with the first upload request, and that isn't a valid assumption for older upload clients. Currently, our constraint is a combination of developer time, compute resources, and privileged backend database access; funding would break this bottleneck.

Improve specificity of license classifiers

Our packaging ecosystem relies on a particular structured data format (classifiers) to indicate a package's legal license. However, our current system allows for ambiguity that makes some downstream data display incoherent or very difficult, and doesn't allow for some license specificity that downstream consumers need ( and similar projects). Fixing this is a fairly small project, involving Python development, public communications, project management, and potentially a few hours of legal counsel for review.

Add pipfile support to pip

pip currently uses requirements.txt to specify dependencies; it can specify versions of packages but not hashes. The newer pipfile format can include hashes, but pip doesn't yet support it, so many users are blocked from using pipfile to better secure their Python runtimes. We'd need Python engineering work and project management to develop and deploy this.

Package preview feature for PyPI

Right now, there are ways for package maintainers to test and share draft versions of their upcoming releases, but they cause friction and confusion. So we want to add staged releases -- a temporary state that a release can be in, where PyPI ''has'' it and can evaluate it, but hasn't ''published'' it yet.

This will:

We'll need database support for understanding the release state ("is this published or not"), user experience and developer support, and testing, security, infrastructure, and project management support.

Feature flag system on PyPI

It's difficult to roll out new features gradually to PyPI's test site or to selected test users. A feature flag system would help us do targeted outreach to particular groups of users, deploy more confidently, and roll back changes when needed. We'd need user experience, front and backend engineer, data analytics, and project management support to develop and deploy this.

User support ticket system

Python packagers who need help currently create Sourceforge and GitHub tickets, email mailing lists, tweet at maintainers, and so on. A unified user support ticket system, integrated into Warehouse, would:

We need funding for backend and frontend development, testing and security checks, DevOps/infrastructure/platform services (including API/email integration), user experience work, technical writing for end users, project management, and community outreach.

Security improvements and prerequisites

System to label projects on PyPI with administrative statuses/attributes

To scale up our anti-abuse moderation and help package maintainers with security response, we need to be able to, for instance, mark a release as deprecated or a project as unsupported. This means we need a generic system to add, edit, and remove administrative attributes ("flags" or "statuses") to individual projects and releases. We need support to do the architectural design to implement this. (See notes from this meeting.)

Security notifications for vulnerable packages

To keep PyPI's users secure, we want to give them an opt-in communication channel to hear about security vulnerabilities for the packages they use. Implementing this would also give us architectural support to warn or prevent pip users who try to install a PyPI package that's been found to be broken or malware. We need funding for user experience work, development, testing, infrastructure, potentially platform services (e.g., SMS), and community outreach.

Fundable Packaging Improvements (last edited 2019-08-22 19:07:30 by SumanaHarihareswara)

Unable to view page? See the FrontPage for instructions.