|
Size: 2918
Comment: link to some diffs
|
← Revision 33 as of 2010-01-02 16:58:16 ⇥
Size: 4615
Comment: Binary spam
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| Setuptools requirements: | === setuptools includes Jython trunk support as of version 0.6c8! === Setuptools trunk and its 0.6 branch have had [[http://svn.python.org/view?rev=60062&view=rev|a couple of changes]] (discussed [[http://mail.python.org/pipermail/distutils-sig/2008-January/008617.html|here on distutils-sig]]) made to fix incompatibilities with Jython. Even as of these changes, there are a couple improvements that can be made to running setuptools on Jython: * Most importantly, the lack of os.chmod. chmod can be implemented with JNA. setuptools simply avoids chmod if it does not exist as of '''''[[http://svn.python.org/view?rev=60062&view=rev|r60062]])''''' * setuptools' use of marshal: setuptools uses marshal.load in two places -- to get a code object from a .pyc file, so it can scan the code object's co_names and co_consts (basically to read a module's variables without importing it). This isn't portable: Jython's compiled .pys (actually Java .class files) can't be read via marshal, nor do Jython's code objects support co_names and co_consts the two places this is used: 1. when a package doesn't mark itself as zip_safe, setuptools will scan it for special variable names (like '__file__') to determine zip_safetyness. '''''(setuptools defaults to zip_safe=False for archives that don't set it as of [[http://svn.python.org/view?rev=60062&view=rev|r60062]])''''' We could get around the lack of marshal compatibility here by using the tokenize module to read the source code. This wouldn't work in the rare case of a byte-code only egg. 2. setuptools.depend.get_module_constant: a generic way to grab the value of a module variable. currently only used by Require.get_version '''''setuptools.depend is experimental, unsupported functionality that we don't need to worry about''''' * We should consider making the -E command line argument disable the registry The Jython changes made for setuptools: |
| Line 7: | Line 27: |
| We don't have: | * file descriptor support for tempfile.mkstemp, os.open and tarfile '''''Added in r3711''''' * tempfile.mkstemp '''''Added in r3712''''' |
| Line 9: | Line 31: |
| * the kicker, a working os.chdir. I hacked together [http://hg.underboss.org/jython-pjenvey/rev/51246729a616 a really quick version of this ], and setuptools was able to get pretty far at installing Pylons. Pylons itself and a number of its dependencies installed fine, but the process died installing the decorator module due to marshal raising a KeyError | * os.chdir (also needed for distutils) '''''Added in r3844''''' |
| Line 11: | Line 33: |
| * working marshal. setuptools uses marshal.load, and there's some bugs with our current marshaller | * tarfile module '''''Added in r3850''''' |
| Line 13: | Line 35: |
| * tarfile module. This requires [http://hg.underboss.org/jython-pjenvey/rev/3507a9b953fa a small patch to PyString] to work for setuptools. I believe it may require some additions to the os module for test_tarfile.py to pass | * a valid sys.executable. required by setuptools and distutils for spawning subprocesses. '''''Added in r3867, enabled via -Dpython.executable (see http://article.gmane.org/gmane.comp.lang.jython.devel/4068 )''''' |
| Line 15: | Line 37: |
| * a valid sys.executable. required by setuptools and distutils for spawning subprocesses. It's impossible to determine argv[0] from java: I'm thinking the jython executable should pass this information along to Jython (java -Dpython.executable=$0) | * a site-packages dir (jython needs to include it on sys.path by default). required by setuptools and distutils '''''Added in r3886''''' |
| Line 17: | Line 39: |
| * distutils: requires [http://hg.underboss.org/jython-pjenvey/rev/1026fe32c01c a number of small patches]: | * distutils: requires [[http://hg.underboss.org/jython-pjenvey/rev/1026fe32c01c|a number of small patches]]: |
| Line 19: | Line 41: |
| - distutils metadata for the os.name == 'java' (like path names, default bdist type, etc) | * a valid sys.executable '''''Added in r3867''''' * getpass module '''''Added in r3876''''' * distutils metadata for the os.name == 'java' (like path names, default bdist type, etc) |
| Line 21: | Line 47: |
| - a jython spawn-like function: "error: don't know how to spawn programs on platform 'java'". os.system works as a replacement | * a jython spawn-like function: "error: don't know how to spawn programs on platform 'java'". os.system works as a replacement |
| Line 23: | Line 49: |
| - a valid sys.executable * tempfile.mkstemp - I have [http://jython.org/patches/1783692 patch #1783692] implementing mkstemp (without a valid file descriptor) and NamedTemporaryFile with Java's secure File.createTempFile '''* See below''' * imp.acquire/release_lock This is just a simple lock, but may have to be used by internal imp operations. I'm not sure |
* compile .py files to $py.class instead of .pyc '''''distutils and these last 3 modifications were added in r3888''''' * imp.acquire/release_lock (and the accompanying lock_held which we don't actually need). Exposes the import machinery's lock via the imp module '''''Added in r3894''''' |
| Line 33: | Line 55: |
| -E codec : Use a different codec the reading from the console. * a site-packages dir (jython needs to include it on sys.path by default) |
-E codec : Use a different codec the reading from the console. '''''-E codec renamed to -C codec in r4013. -E is now a noop (but valid) command line arg''''' |
| Line 37: | Line 57: |
| * A few small maybe upstream patches to setuptools, this includes so far: - use tempfile.NamedTemporaryFile instead of tempfile.mkstemp (Java/Jython doesn't allow usage of file descriptors). This will also remove a usage of os.open '''* See below''' - wrap uses of os.chmod in hasattr(os, 'chmod') '''* See below''' - sandbox may assume os.open (and maybe others) exist '''* See below''' - setuptools cut and pasted shutil's rmtree from CPython 2.3, which uses lstat to check for directories. This may need to change to os.path.isdir (Jython might be able to do lstat, though) |
* os.lstat. setuptools cut and pasted shutil's rmtree from CPython 2.4, which uses lstat to check for a directory vs a symlink to a directory. We now support an os.stat that fills in st_mode's directory bit, but can we support lstat? (We definitely can with JNA)? Can setuptools avoid lstat completely (requiring a patch)? I don't think Java can safely determine a sym link yet. Java 1.7's JSR 203 (nio part 2) should address this as well as chmod, but we obviously need a solution for Java 1.5 '''''Turned out we can implement a simple lstat in pure Java, added in r4014''''' |
| Line 47: | Line 59: |
| - '''*I'm looking into faking file descriptors in Jython, allowing the os.open and mkstemp APIs to work -pjenvey''' | * for Jython on Windows: distutils.filelist.translate_pattern required ntpath to work correctly, and setuptools required chdir to not canonicalize the pathname via java.io.File '''''both issues fixed with usage of ntpath, in r4171''''' |
setuptools includes Jython trunk support as of version 0.6c8!
Setuptools trunk and its 0.6 branch have had a couple of changes (discussed here on distutils-sig) made to fix incompatibilities with Jython.
Even as of these changes, there are a couple improvements that can be made to running setuptools on Jython:
Most importantly, the lack of os.chmod. chmod can be implemented with JNA. setuptools simply avoids chmod if it does not exist as of r60062)
- setuptools' use of marshal: setuptools uses marshal.load in two places -- to get a code object from a .pyc file, so it can scan the code object's co_names and co_consts (basically to read a module's variables without importing it). This isn't portable: Jython's compiled .pys (actually Java .class files) can't be read via marshal, nor do Jython's code objects support co_names and co_consts
- the two places this is used:
when a package doesn't mark itself as zip_safe, setuptools will scan it for special variable names (like 'file') to determine zip_safetyness. (setuptools defaults to zip_safe=False for archives that don't set it as of r60062) We could get around the lack of marshal compatibility here by using the tokenize module to read the source code. This wouldn't work in the rare case of a byte-code only egg.
setuptools.depend.get_module_constant: a generic way to grab the value of a module variable. currently only used by Require.get_version setuptools.depend is experimental, unsupported functionality that we don't need to worry about
- We should consider making the -E command line argument disable the registry
The Jython changes made for setuptools:
PEP 302 style zipimport module Added in r3463
number of misc. Jython bug fixes most already committed
file descriptor support for tempfile.mkstemp, os.open and tarfile Added in r3711
tempfile.mkstemp Added in r3712
os.chdir (also needed for distutils) Added in r3844
tarfile module Added in r3850
a valid sys.executable. required by setuptools and distutils for spawning subprocesses. Added in r3867, enabled via -Dpython.executable (see http://article.gmane.org/gmane.comp.lang.jython.devel/4068 )
a site-packages dir (jython needs to include it on sys.path by default). required by setuptools and distutils Added in r3886
distutils: requires a number of small patches:
a valid sys.executable Added in r3867
getpass module Added in r3876
- distutils metadata for the os.name == 'java' (like path names, default bdist type, etc)
- a jython spawn-like function: "error: don't know how to spawn programs on platform 'java'". os.system works as a replacement
compile .py files to $py.class instead of .pyc distutils and these last 3 modifications were added in r3888
imp.acquire/release_lock (and the accompanying lock_held which we don't actually need). Exposes the import machinery's lock via the imp module Added in r3894
- setuptools runs a test to ensure .pth files work by running "sys.executable -E -c pass". -E is ignore environment on CPython, but is different on Jython (takes an argument):
-E codec : Use a different codec the reading from the console. -E codec renamed to -C codec in r4013. -E is now a noop (but valid) command line arg
os.lstat. setuptools cut and pasted shutil's rmtree from CPython 2.4, which uses lstat to check for a directory vs a symlink to a directory. We now support an os.stat that fills in st_mode's directory bit, but can we support lstat? (We definitely can with JNA)? Can setuptools avoid lstat completely (requiring a patch)? I don't think Java can safely determine a sym link yet. Java 1.7's JSR 203 (nio part 2) should address this as well as chmod, but we obviously need a solution for Java 1.5 Turned out we can implement a simple lstat in pure Java, added in r4014
for Jython on Windows: distutils.filelist.translate_pattern required ntpath to work correctly, and setuptools required chdir to not canonicalize the pathname via java.io.File both issues fixed with usage of ntpath, in r4171