Differences between revisions 4 and 9 (spanning 5 versions)
Revision 4 as of 2007-07-18 19:08:43
Size: 6752
Editor: AlanKennedy
Comment:
Revision 9 as of 2008-11-15 09:15:59
Size: 6483
Editor: localhost
Comment: converted to 1.6 markup
Deletions are marked like this. Additions are marked like this.
Line 3: Line 3:
[[TableOfContents(4)]] <<TableOfContents(4)>>
Line 7: Line 7:
There is now select support in the [http://www.jython.org/Project/download.html jython distribution], as of version 2.2rc1. There is now select support in the [[http://www.jython.org/Project/download.html|jython distribution]], as of version 2.2rc1.
Line 9: Line 9:
The new module presents an API which is as close as possible to the [http://www.python.org/doc/lib/module-select.html cpython select module], Jython supports both the The new module presents an API which is as close as possible to the [[http://www.python.org/doc/lib/module-select.html|cpython select module]], Jython supports both the
Line 12: Line 12:
   1. [http://www.python.org/doc/lib/poll-objects.html select.poll objects].    1. [[http://www.python.org/doc/lib/poll-objects.html|select.poll objects]].
Line 14: Line 14:
When using the select module, you should be guided by the cpython documentation: any deviation from the behaviour described in that documentation, except for the the considerations mentioned below, should be considered a bug and [http://www.jython.org/bugs reported] as such. When using the select module, you should be guided by the cpython documentation: any deviation from the behaviour described in that documentation, except for the the considerations mentioned below, should be considered a bug and [[http://www.jython.org/bugs|reported]] as such.
Line 18: Line 18:
   1. Poll objects are the most efficient mechanism to multiplex sockets on jython, being fairly much a direct mapping to [http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/Selector.html java.nio.channel.Selector objects].    1. Poll objects are the most efficient mechanism to multiplex sockets on jython, being fairly much a direct mapping to [[http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/Selector.html|java.nio.channel.Selector objects]].
Line 52: Line 52:
However, [http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/SelectableChannel.html#register(java.nio.channels.Selector,%20int) java will only permit multiplex operations on sockets that are in non-blocking mode]; any attempt to pass a socket in blocking mode to either select.select or select.poll().register will fail with an exception. However, [[http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/SelectableChannel.html#register(java.nio.channels.Selector,%20int)|java will only permit multiplex operations on sockets that are in non-blocking mode]]. Because jython must respect this java restriction, any attempt in jython to pass a socket in blocking mode to either select.select or select.poll().register will fail with an exception.
Line 61: Line 61:
This issue could be problematic if you want to use a cpython module which relies on select, e.g. telnetlib. Such cpython modules do not set their sockets in non-blocking mode before passing them to select.select. In order to support such cpython modules, the following fix is proposed This issue could be problematic if you want to use a cpython module which relies on select. Such cpython modules do not set their sockets in non-blocking mode before passing them to select.select. In order to support such cpython modules, the following workaround is available.
Line 63: Line 63:
==== Possible solution for supporting cpython-compatible select ==== ==== The cpython-compatible select function ====
Line 65: Line 65:
One possible solution to this problem is to write a special version of the jython select function which provides compatibility with the cpython version. This would be achieved by A special version of the select function is provided, to provide cpython compatible select functionality. This function works by
Line 69: Line 69:
   1. Calling the normal select.select function (which requires non-blocking sockets)    1. Calling the normal jython select.select function (which requires non-blocking sockets)
Line 73: Line 73:
However, there is a clear problem with this approach: The called select function may not be the only thread of execution examining the socket. If the socket is being operating upon in a different thread, then it is very likely that changing the blocking mode of the socket will cause a failure of the operation in the other thread.

==== Solution for jython 2.2 ====

Based on the following points

   1. It is unacceptable for a standard jython library function to change the blocking mode of a socket with the explicit consent of the user
   1. The problem only arises when the user wishes to make use of select-dependent cpython modules, e.g. telnetlib, without modification.

The following is the proposed solution for jython 2.2

A special version of the select function will be written to provide cpython compatibility. This function will

   1. Change the blocking mode of all sockets to non-blocking before passing to the "real" select function
   1. Will restore the original blocking mode of all sockets before return to the user
   1. Will be named in such a way that the name does not interfere with existing classes and functions
   1. Will require explicit action from the user before it is enabled
This function should only be used when you are working with code that has been written for cpython select, i.e. when the code is passing blocking-sockets to the select function.
Line 93: Line 77:
If you want to use cpython compatible select function, follow these steps

   1. Import the select module, i.e. '''import select'''
   1. Reassign the select function to the cpython compatible version, i.e. '''select.select = select.cpython_compatible_select'''
   1. Then import whatever cpython module you wish to use, e.g. '''import telnetlib'''

Here is an example
Here is a code sample which shows how to use the cpython-compatible select function as a replacement for the standard jython select function
Line 102: Line 80:
import select
select.select = select.
cpython_compatible_select
import
telnetlib
from select import cpython_compatible_select as select
Line 106: Line 82:
# Make use of telnetlib here # Make use of the select function here
Line 111: Line 87:
   1. If using the cpython_compatible_select function, you must be aware of that the function will modify the blocking mode of your sockets for the duration of the call. If you are carrying out socket operations on that socket in another thread, then those socket operation may fail or raise exceptions.    1. If using the cpython_compatible_select function, you must be aware that the function will modify the blocking mode of your sockets for the duration of the call. If you are carrying out socket operations on that socket in another thread, then those socket operations may fail or raise exceptions.
Line 113: Line 89:


== Known Issues ==

=== Receiving urgent data can cause select to lie (Windows implementation) ===

Bug 1773955 describes the scenario. It is caused by a bug in the Sun Java implementation and there is no known Jython fix. A partial workaround is to call setOOBInline(True) on the jsocket within the Jython socket. This will cause the urgent data to be merged into the regular stream and will keep from confusing select() however you will have to ignore the urgent data at the application level.

'''Addendum''': Support for socket options has been checked into the repo at [[http://fisheye3.atlassian.com/changelog/jython/?cs=4494|revision 4494]]. See NewSocketModule for more details on the options available.

With the new support, you don't need to call the java API on the underlying socket; you can set the OOBINLINE flag using python syntax, like this
{{{
mysock.setsockopt(socket.SOL_SOCKET, socket.SO_OOBINLINE, 1)
}}}
But there is still no workaround for the lack of support for TCP Urgent Data on java.

New Select Module

Introduction

There is now select support in the jython distribution, as of version 2.2rc1.

The new module presents an API which is as close as possible to the cpython select module, Jython supports both the

  1. select.select function
  2. select.poll objects.

When using the select module, you should be guided by the cpython documentation: any deviation from the behaviour described in that documentation, except for the the considerations mentioned below, should be considered a bug and reported as such.

If you are starting a new project, it is recommended that you use select.poll objects, because

  1. Poll objects are the most efficient mechanism to multiplex sockets on jython, being fairly much a direct mapping to java.nio.channel.Selector objects.

  2. The select.select function is implemented using a poll object, which means that sockets are de/registered every single time, which is slightly less efficient.

Usage notes

Always close poll objects

When a socket has been registered with a select.poll object, it remains registered until explicitly deregistered. This has the following implications

  1. Sockets cannot be placed in blocking mode while they are still registered with poll objects
  2. The reference from the poll object to the socket might interfere with garbage collection

Therefore, it is recommended that you always explicitly close poll objects, using an idiom such as

def my_polling_func(sockets):
    poll_object = select.poll()
    try:
        for s in sockets:
            poll_object.register(s)
    finally:
        poll_object.close()

Closing a poll object cancels all registrations of sockets.

Differences between cpython and jython

Due to fundamental differences in the behaviour of java and C on various platforms, there are differences between the cpython and jython select modules which are not possible to code around. Those differences will be listed here.

Only sockets in non-blocking mode can be multiplexed

On cpython, when a socket is passed to select.select or select.poll, it can either be in blocking or non-blocking mode.

However, java will only permit multiplex operations on sockets that are in non-blocking mode. Because jython must respect this java restriction, any attempt in jython to pass a socket in blocking mode to either select.select or select.poll().register will fail with an exception.

To summarise

  1. You must set a socket in non-blocking mode before passing it to the select function or registering it with a poll object
  2. Any attempt to register a blocking socket for multiplex will raise a select.error exception, with an error code of errno.ESOCKISBLOCKING
  3. If a socket is currently registered with a select.poll object, an attempt to change it to blocking mode will give rise to the same exception.
  4. A socket can only be placed in blocking mode if it is not registered with a select.poll object.

This issue could be problematic if you want to use a cpython module which relies on select. Such cpython modules do not set their sockets in non-blocking mode before passing them to select.select. In order to support such cpython modules, the following workaround is available.

The cpython-compatible select function

A special version of the select function is provided, to provide cpython compatible select functionality. This function works by

  1. Recording the blocking mode of all sockets that are passed to the function
  2. Setting non-blocking mode on all sockets
  3. Calling the normal jython select.select function (which requires non-blocking sockets)
  4. Restoring the previously-saved blocking mode for all sockets
  5. Returning the results of the wrapped select function

This function should only be used when you are working with code that has been written for cpython select, i.e. when the code is passing blocking-sockets to the select function.

How to enable cpython compatible select processing

Here is a code sample which shows how to use the cpython-compatible select function as a replacement for the standard jython select function

from select import cpython_compatible_select as select

# Make use of the select function here

WARNING!

  1. If using the cpython_compatible_select function, you must be aware that the function will modify the blocking mode of your sockets for the duration of the call. If you are carrying out socket operations on that socket in another thread, then those socket operations may fail or raise exceptions.
  2. This function will still not cover all possible uses of select.select by cpython modules. If a cpython module tries to multiplex the sys.stdin or sys.stdout streams, then registration of the channels will fail, because the InputStream and OutputStream representing sys.stdin and sys.stdout are not SelectableChannels.

Known Issues

Receiving urgent data can cause select to lie (Windows implementation)

Bug 1773955 describes the scenario. It is caused by a bug in the Sun Java implementation and there is no known Jython fix. A partial workaround is to call setOOBInline(True) on the jsocket within the Jython socket. This will cause the urgent data to be merged into the regular stream and will keep from confusing select() however you will have to ignore the urgent data at the application level.

Addendum: Support for socket options has been checked into the repo at revision 4494. See NewSocketModule for more details on the options available.

With the new support, you don't need to call the java API on the underlying socket; you can set the OOBINLINE flag using python syntax, like this

mysock.setsockopt(socket.SOL_SOCKET, socket.SO_OOBINLINE, 1)

But there is still no workaround for the lack of support for TCP Urgent Data on java.

SelectModule (last edited 2009-01-28 13:10:09 by AlanKennedy)