|
⇤ ← Revision 1 as of 2005-10-04 12:06:07
Size: 1316
Comment:
|
Size: 3330
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| This page explains how to review proposals for PyCon. | #acl All:read |
| Line 3: | Line 3: |
| Each proposal is randomly assigned three reviewers when it's submitted. However, those assignments are just suggestions: you don't have to review all the papers you're assigned, and you might not be able to review them because of a conflict of interest, not being competent in the field, etc.. Therefore, you can review proposals that aren't assigned to you. As long as every proposal gets three reviews from three different people, we're happy. |
This page explains the duties of reviewers on the program committee. Committee members can add questions and observations to this page. '''Do not refer to specific proposals on this page''', because this page is publicly visible. |
| Line 11: | Line 5: |
| Once submission is closed and all the proposals have three reviews, we make the accept/reject decisions. Last year we did this in a 2- or 3-hour IRC session where perhaps half the committee was present. |
Reviewers have logins on http://submit.python.org, just like proposal authors. Reviewers have the additional privilege of being able to view other proposals and to record comments and votes on them. |
| Line 15: | Line 9: |
| Reviewers vote +1, -1, +0, -0. Proposals that get only +1, or +1 with a +0, are usually accepted with little discussion. Papers that get only -1 or -0s usually get rejected. The bulk of the IRC discussion was about papers with mixed reviews. Another IRC session a few days later was used to assemble the scheduling, which was done outside the proposal system. (AMK wants to add scheduling to the proposal system this year... not a pressing need right now, though.) |
Proposals will be assessed using the [ http://www.iam.unibe.ch/~oscar/Champion/ Identify the Champion] process. Basically, reviewers vote using a +/- 1, +/- 0 scale. To be accepted, a paper must have at least one +1 vote; each person who votes +1 on a paper is a paper's champion. |
| Line 23: | Line 14: |
| The general process came from the paper at http://www.iam.unibe.ch/~oscar/Champion/ . |
When reviewers log in, their web page will list their own proposals as well as a bunch of hyperlinks to various specialized lists of proposals. The most important list is the one of proposals still needing reviews, at http://submit.python.org/reviews_needed. Reviewers can then view those proposals and record their votes along with additional comments. Be sure to read the most recent version of the proposal; check the date and time of the file. Users can upload multiple revisions of their proposal. When you mark a review as publicly visible, it gets e-mailed to the author the same day (not immediately -- it's a cron job that runs every so often), so the author can update the proposal, fix a problem, etc. After the reviewing is over, authors are mailed all comments on their proposal, whether marked as publicly visible or not. If you want your comments to be e-mailed to the proposal's author right away, you MUST uncheck the "keep comments private" box in the review form. If you have suggestions that would improve a proposal you should normally make them visible, allowing the author to respond before the acceptance decisions are made (hopefully with an updated proposal). In previous years, authors were e-mailed the comments on their proposal after the final decisions were made. You should write your comments with this in mind. Please try to avoid unnecessarily negative comments, and don't be rude. Reviewers should join [http://mail.python.org/mailman/listinfo/pycon-pc/ the pycon-pc mailing list], which is for program committee discussions. When a proposal is submitted, three reviewers are randomly assigned to it. Those assignments aren't enforced, so reviewers should feel free to skip proposals assigned to them that they don't feel competent to review. Conversely, reviewers should be reviewing proposals assigned to other people, and doing so is greatly encouraged. ''QUESTION: Is there a way for a reviewer to find out which proposals he or she was assigned to?'' Once you've logged in to submit.python.org, under the 'Your assigned reviews' header is a list of proposals assigned to you that you haven't voted on yet (if you just posted a comment with no vote, the proposal will remain on this list). There's currently no page that lists everything that was assigned to you, whether you voted on it or not. |
This page explains the duties of reviewers on the program committee. Committee members can add questions and observations to this page. Do not refer to specific proposals on this page, because this page is publicly visible.
Reviewers have logins on http://submit.python.org, just like proposal authors. Reviewers have the additional privilege of being able to view other proposals and to record comments and votes on them.
Proposals will be assessed using the [ http://www.iam.unibe.ch/~oscar/Champion/ Identify the Champion] process. Basically, reviewers vote using a +/- 1, +/- 0 scale. To be accepted, a paper must have at least one +1 vote; each person who votes +1 on a paper is a paper's champion.
When reviewers log in, their web page will list their own proposals as well as a bunch of hyperlinks to various specialized lists of proposals. The most important list is the one of proposals still needing reviews, at http://submit.python.org/reviews_needed. Reviewers can then view those proposals and record their votes along with additional comments.
Be sure to read the most recent version of the proposal; check the date and time of the file. Users can upload multiple revisions of their proposal.
When you mark a review as publicly visible, it gets e-mailed to the author the same day (not immediately -- it's a cron job that runs every so often), so the author can update the proposal, fix a problem, etc. After the reviewing is over, authors are mailed all comments on their proposal, whether marked as publicly visible or not.
If you want your comments to be e-mailed to the proposal's author right away, you MUST uncheck the "keep comments private" box in the review form.
If you have suggestions that would improve a proposal you should normally make them visible, allowing the author to respond before the acceptance decisions are made (hopefully with an updated proposal).
In previous years, authors were e-mailed the comments on their proposal after the final decisions were made. You should write your comments with this in mind. Please try to avoid unnecessarily negative comments, and don't be rude.
Reviewers should join [http://mail.python.org/mailman/listinfo/pycon-pc/ the pycon-pc mailing list], which is for program committee discussions.
When a proposal is submitted, three reviewers are randomly assigned to it. Those assignments aren't enforced, so reviewers should feel free to skip proposals assigned to them that they don't feel competent to review. Conversely, reviewers should be reviewing proposals assigned to other people, and doing so is greatly encouraged.
QUESTION: Is there a way for a reviewer to find out which proposals he or she was assigned to?
Once you've logged in to submit.python.org, under the 'Your assigned reviews' header is a list of proposals assigned to you that you haven't voted on yet (if you just posted a comment with no vote, the proposal will remain on this list). There's currently no page that lists everything that was assigned to you, whether you voted on it or not.