Dev Syncup

(June 3rd?)

- Remaining failing tests

• - Requirement already satisfied vs Requirement already up-to-date
• - test_install_distribution_union_with_versions
• - test_hashed_install_failure_later_flag

YAML test discussions

Participants:

• - Ilan - Tzu-ping

- Each case is a test. Tests in a file share the same base

• - It can have all the same fields in case
• - Each case inherits the base in the same file
• - Same-named fields override (not extend)

- Requests and responses correspond by index (requests[0] results in responses[0], etc.) - State in response must entirely match - No way to pass complex packages to pip (e.g. editable), but can use options: (hack)

• - Options are space-delimited

- Ilan added yesterday a better check for error messages, so it checks as a regex - The test goes through a linter first, so you can see the errors faster

• - tests/yaml/linter.py <path-to.yml>

• - Checks the structure of the YAML, and type of each value
• - Will also need to update this file if we add new fields to the YAML test declarations

- No stdout field yet, but should be easy to add - That's all!

pip Resolver development and User Experience meeting notes

Dev Syncup

- Deprecate stuff from pip 20.2

• - things that won't work in the new resolver.
• - because they're bad?
• - Add a master issue in the tracker for this
  • - constaints with extras, look at other stuff too?
  • - #egg=wrong_name

- issue yesterday #8339 - That escape hatch for "let me do what I want"

• - https://github.com/pypa/pip/issues/8307

• - TP thinks this is out of scope
• - Pradyun thinks it is.
• - Paul thinks it's borderline.
  • - It's needed and possible to implement.
  • - The UI for it is the tricky to figure out -- this is what pushes it out of scope.
    • - We shouldn't make this until we have a UI.
• - We should flag "we need to design a UI" to the UX folks (everyone agrees)
  • - digress: similar to the situation on the "next" stage of --unstable-feature=resolver (roll out flag)
• - Action: Someone (Paul, OK, then) describe what we want from the UX team, and pass it to them (and Sumana) for action.
  • - Dev team needs to give input on "what's possible", and "what's easy".

- pip's new resolver's output: https://github.com/pypa/pip/issues/8220

• - Treating these like regular bug reports would make the workflow easier.
• - Existing error on #8220 looks "OK" to us experts.
• - Could be improved, but we don't know if the improvements would help actual users.
• - We need the UX team to start the ball rolling by saying what *they* want to have included.
• - We can then refine that by looking at what's possible and what's information that the resolver doesn't have

  • Pradyun found https://github.com/pypa/pip/issues/8346 concerning.

- pip freeze showing "appdirs @ https://files.pythonhosted.org/packages/3b/00/2344469e2084fb287c2e0b57b72910309874c3245463acd6cf5e3db69324/appdirs-1.4.4-py2.py3-none-any.whl#sha256=a841dacd6b99318a741b166adb07e19ee71a274450e68237b4650ca1055ab128" with new resolver

• - TP: Likely due to PEP 610. Will need to review the logic after the wheel cache PR
• - Got fixed in the wheel cache PR.

- Requirements with (potentially conflicting) options

• - https://github.com/pypa/pip/issues/7846

• - Let’s make it an error, we don’t know how yet
• - Paul can't remember where the code is that prompted the question

- --upgrade-strategy=never

• - Note: This is a feature request

• - Paul worked on upgrade-strategy: https://github.com/pypa/pip/pull/8117

• - https://github.com/pypa/pip/issues/4745

• - No clear "this is the input, this is the output" description.

- Already Installed packages

• - https://github.com/pypa/pip/issues/7744

• We sort of reached agreement that we just do what we have at the moment.

  • - Pradyun wasn't 100% convinced by the arguments (and TBH, Paul understands they were a bit weak

  • - But we decided to deal with improving things as a later feature request.

There are 2 conflicting requirements on the resolver: want overrides on my environment, don't want broken environments at all.

People who install stuff into system just to do stuff are a *real* and important group of users, so broken environments are 100% necessary.

• TP made this point
• Paul 100% agrees.

New resolver reports previous installation failure when building multiple sdists for the same project

• https://github.com/pypa/pip/issues/8294

- Next up

• - Pradyun: Try to convince Travis not making the whole build red + Rename parent to template + Look into Travis not reporting
• - TP: Fix a test.
• - Paul: Fix tests (i.e., find a failing test and fix it - I'm doing test_install_unsupported_wheel_file right now)

Teamwide meeting

Teamwide meeting Wednesday, 3 June 2020

Participants:

• Sumana
• Bernard
• Georgia
• Paul
• Nicole
• Tzu-ping
• Pradyun
• Ernest

Agenda:

• Ilan and testing
  • his work: let's finish it and merge it. Useful parts in that PR
    • figure out who will finish and merge it ... migrate some stuff having to do with legacy resolver:
      • Pradyun: No preference.
      • TP: not sure if I'm the best person, I'd like to port some tests though.
      • Paul: I'm OK if TP wants to work on it. I don't have any specific goals. I can work on it, but not looking for that functionality.
    • TODO: Tzu-Ping to finish and merge this.
• Invoices for May
  • need from Changeset and Simply Secure
  • if Sumana gets them this week, then:
• Paul
  • a few more days in June
  • later in June, let's have a conversation about transition to volunteer
• Tzu-Ping
  • We're finding out whether there's money and availability for a few more days' work in June.

  • TP will be busy this weekend & next -- should have some availability in last week of June (correction: last two weeks, I misread the calendar - TP)

• Blockers/status
  • Tzu-Ping: just went through what we're working on, cleaning up remaining pip failures. We are mostly done! There's really nothing left on the list - no blockers!
  • Paul: there is the odd 1-2 test failures he can tidy up ... after that, am blocked by lack of next thing to do.

  • Pradyun: No blockers for me [other than THE CYCLONE]. Mainly want to pull together the loose ends on the issue tracker. Been slowly working through the trashfire that is my GitHub Notifications.

  • Georgia: nothing in particular.
  • Bernard: seeking some direction (see Zulip thread)
  • Nicole [see Zulip re availability dates]: have started on docs for the resolver. Not blocked yet, but probably going to need some dev review on this, hopefully between now and next week. Hope that by the end of today, have an outline/rough structure of doc to write, so that Tues she can continue.
  • Sumana: the blocker on the issue w.r.t budget (invoices!). There is a blocker that I hope to resolve later in this call (coordinating, later in this call) + decision making.
    • TODO: set up next 4 weekdays of optional collab time

• What flag will users use to call the new resolver? https://github.com/pypa/pip/issues/8371

  • ahhh -- [clarification re Bernard comment -- was "what is your concern that Nicole did not already address"?]
  • who will decide?
    • Sumana's suggestion: we have a couple of these different proposals (Paul's, Nicole's). And I saw that Paul... Nicole's proposal was partly a response. There are some concerns that have been brought up as questions. I think that I'd like for it to be Paul/TP/Pradyun who makes the decision what flag we'll use for the resolver, informed by all the others (Sumana / Bernard / Georgia / Nicole and everyone else). Pradyun makes the final choice if need be.
      • a basic idea for who decides.
    • Sumana: how I originally though about how to make a decision be: I thought we'd ask UX folks how our users would respond. However, there's a lot of fundamental research (users, users' needs, user's mental models). We have some data points, but we still don't know enough to make research backed decisions.
      • For the next little while/this month, it'll make sense to pull back to advice based on general UX best practices etc, instead of research backed advice (due to how long it'd take).
    • Georgia: I think we've been struggle overall with a decision framework that factors in a user/ux frame, which is hard to answer that when we don't have a base level of user research -- user /use case diversity. not enough context to place responses in. making it hard to make directed advice from the research.
    • Bernard: it is helpful..... 
    • Nicole: agree that it's helpful -- we can provide input on time-sensitive issues based on our general knwoledge of best practice, would like to continue to do that. would be useful for us to take a step back and do more discovery work. might be useful on UX side - G, B, and N need to talk about this - what form the discovery is going to be. What will output be? Python packaging community would find that useful, shared for other projects other than pip .... [comes into] the documentation work we talked about, Phase III
    • Georgia: agreed. We have some start on from earlier this year. Got put to the side to focus more on this specific resolver stuff.
    • Sumana: initial project plan was overambitious in what we thought we could do in the first few months
    • Bernard: Agree with Nicole and Georgia. Recommendations we can give will be a blend of best practices + actual qualitative/quantitiative user research, because it'll be difficult to get a large enough sample of "what the pip user base needs".
    • Nicole: for Pradyun to consider: part of the scope of this is to inform the team: who pip users are, what they want, etc. All the research we can gather. For later: what format would this be most useful to share this knowledge? how can we engage the developers working on the project? and get most value out of that work as well?
      • TODO: Sumana to follow up on this with examples
• How we coordinate (espc. with other maintainers)
  • Sumana: how to avoid regrettable "wait what?" with other maintainers?

  • Paul: making sure we are public about decisions we make is important. Difficult to be sure how public we're being, because of semipublic usage of things like Zulip -- need to be conscious of summarizing where the other developers look. Getting other developers to confirm they are ok .... we can be reasonably relaxed over that. A lot of pip developers have patchy availability. If we try to get everyone to say they're happy, we can wait weeks for straightforward decisions. Xavier and Stephan are most active other than Pradyun & Paul. They are participating in some conversations. I wouldn't try to block ourselves too hard on getting visible consensus. But we should be careful.

  • Pradyun: Agree with Paul. Add: as long as we are giving the other maintainers a chance to speak up, a fair chance (not "next 2 hrs") but "next few days", it's fine. More urgent than that, I am sure they will understand. Zulip is OK. GitHub issue and keeping it open for a few days so people chime in would be nice. Haven't hit anything where we need to do that except for rollout flag stuff.

  • Paul: I think the rollout is where we will need to be careful about communication - in terms of timing. Putting out extra beta, which we did the other week, in hindsight, perhaps should have given other developers a heads-up about it. Publicity and timing pushes for releases -- make sure other devs know it is happening
    • Therefore: Sumana: what I feel like I'm hearing is

      • a) make sure that if we seem to be having a conv in Zulip, we should put that in a GitHub issue -- and document the discussion/decision. If it's something we can wait on, explicitly note that we're waiting on inputs.

        • Sounds about right - Paul
        • Sounds good, yea - Pradyun

      • b) rollout/publicity/timing: we need to make sure that those are conversations that happen in GitHub, and we @mention other maintainers, and try to get an explicit OK.

        • Sounds reasonable. - Paul
      • Paul: we are making an assumption that 20.2 release will be "The New Resolver Release". We should ensure this framing is ok with the other developers. There's also the in-place build work that got added in 20.1, but backed out. If someone wants to put that back in, might be swamped by publicity for new resolver. Let's collaborate on what the headline issue is./headlines are.
      • Pradyun: I don't know if we will have only 1 headline issue. We would want to include that..... special instructions .... 
        • TODO: separate call about this.
• UX work/workflow/priorities

  • if UX people need to work on ResolutionTooDeep error message

    • ResolutionTooDeep will now only happen after about 2 million tries -- if it is very unlikely to happen, how high-priority is it to work on this?

    • this change was made shortly after the last 20.1 release - it's in 20.2b1. From late May. Not worth investing a lot of time in trying to explain it to people now.
    • We can leave it with the current good-enough, or we can get some best practice suggestion from UX
    • TP: given the current value is 2 million, ok to leave it as is, as a bare exception. If they hit this mark, I want them to report it [inaudible] so I can take a look at it

    • TODO: Nicole & TP: agree to add a "please report this to the pip issue tracker" prompt to the error message

  • But ResolutionImpossible people will see frequently

    • TP: it would be suboptimal for them to report it to *pip*, but we'd like it reported to https://github.com/pypa/packaging-problems/issues/

    • TODO: add "please report this to https://github.com/pypa/packaging-problems/issues/ "

    • should we tentatively plan to revisit and refactor later
    • Nicole: I think we could make some small changes that would be low-cost and radically improve the UX -- better explanation and formatting
    • Bernard: agreed. There are some hopefully reasonably easy formatting and info layout changes we can recommend without having to reformat version signifiers in human-readable versions, etc.
    • TODO: team to finish this change within the next week.

    • Bernard: we have Iteration 2 -- in the ticket 8377 ... next step would be to test with users a bit. Will do that Thursday and Friday https://github.com/pypa/pip/issues/8377

    • Do we know what code to change to implement the improved error message?
      • Pradyun: yes.
      • TP: has an idea
      • Paul: knows the code. wants to make sure we don't cross wires on re what we change it to
      • Nicole: iteration 2 is the "Iteration 2: Error message contains - what the error is, what caused it, and a link to documentation for possible ways to solve it"
        • TODO: by next week, we'll have some basic docs published. We'll have a placeholder up soon
    • Bernard: this is Iteration 2, our current thinking. Then we test this with users, then, assuming the bindings are not drastically different, it will follow that general structure ..... 

      • Paul: it's an actionable coding task. The quoted message https://github.com/pypa/pip/issues/8377 he can do, and then he can update with a followup with that. 

  • so: no new UX research on these 2 issues, and instead the UX group will concentrate on fundamental research and will give advice to the dev team based on general best practices and principles

  • Pradyun: ResolutionImpossible or ResolutionTooDeep - which error message should direct users to packaging-problems? (Nicole: ResolutionImpossible)

  • workflow:
    • how do we pass tasks to UX folks? that has been a problem Nicole/Bernard: is there a preference in how to handle these -- UX questions + outputs?

      • Sumana: prefers GH repo & project board, for public/transparent nature of it

      • Bernard: light weight, frameworky thing.

      • Nicole: from past tickets -- "I don't know how to prioritise UI/UX of <thing>" for the devs, about how important a task is in relation to the other tasks.

      • TODO: follow up, Sumana - Bernard - Georgia - Nicole
    • async standup Zulip thread re what people are working on and waiting for
    • Nicole: re any task how much effort is required to implement this, and how much will user benefit.... 
  • Sumana: We've been discovering this as we go. We didn't even had rough answers earlier.
  • Paul: the problem I had was "fixing tests" we know we need to address, vs discussions where my opinion was being asked for -- wanted a clear [??]. that's not always possible. people need input, and I gues the question is how far we can let... when things become general discussions without an obvious answer, I guess... it becomes how hard is this problem, and then the discussions become quite time consuming.
  • Nicole: if we can have a simple way of saying "UX team asks for this, we rate it as a 7/10 for importance for the user," dev team can say "we rate this as 7/10 for complexity for implementation" then we can decide whether to prioritize and build.... if we say "2/3" for user and dev says "9/10" then it's clear not to do it. but right now, there is no scale, no official way of doing that
    • Pradyun: me likey this "put a number out of 10" idea.
    • Paul was feeling pressured re time limit, doesn't want to leave tasks unfinished, hence some of the management of questions
  • Sumana: will eb setting up collaborative work times. Taking some time to work on this during that would be possible/nice. folks are welcome to put notes in this etherpad.
• Beta timeline
  • Given what we just said, about the work that needs to happen, w.r.t improving the error messages, there's no way we'd wanna put a beta before that.
    • Sumana: Earliest next beta: June 10.
    • Bernard: wants a few more days in case it's hard to get the testing in?
    • We will have Paul making the code change based on Iteration 2 tomorrow (Thursday) and then there will be a pause in his working time till Wednesday next week. He will throw the Iteration 2 text into a WIP PR tomorrow, and then hold off on the actual wording till we have validated text.
    • Bernard: there's lead time between contacting ~200 people and getting the testing set up. So, Bernard will work on testing Thursday through Tuesday.
  • So: Fri the 12th or Thurs the 11th ... pushing to Mon the 15th if the user testing of the error message gooes slower than we hope
  • (Pradyun: back now, taking notes)
    • We have between now and Thurs [June 11th] to get publicity for the release set up
    • manual testing instructions/guide, changelogs, release notes -- more readable prose summary.
    • Sumana to work on this, with help from UX + Dev inputs/help.
    • Hopefully, that'll get us enough input, before the July release, when we do the flag change (one of the headlines: new resolver to try!)
  • Who will be the release manager?
    • Pradyun

TODO: Sumana: followup on ballooning meeting time

Followups and TODO:

• TODO: Tzu-Ping to finish and merge Ilan's testing work.
• TODO: (Not clear who assigned, maybe Sumana?) set up next 4 weekdays of optional collab time
• TODO: Sumana to follow up on this (inform the team: who pip users are, what they want, etc. all research we can cather/what format would this be most useful to share this) with examples
• TODO: separate call about this (planning for rollout).

• TODO: Nicole & TP: agree to add a "please report this to the pip issue tracker" prompt to the error message

• TODO: Re: ResolutionImpossible - add "please report this to https://github.com/pypa/packaging-problems/issues/ "

• TODO: team to finish this change ("some small changes that would be low-cost and radically improve the UX"/"easy formatting and info layout changes" recommended by UX) within the next week.
• TODO: by next week, we'll have some basic docs published. We'll have a placeholder up soon
• TODO: Re: "how do we pass tasks to UX folks?" - follow up, Sumana - Bernard - Georgia - Nicole
• TODO: Sumana: followup on ballooning meeting time