|
Size: 38
Comment:
|
Size: 10538
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| '''Gaps between Jython and Django''' | [http://pipes.yahoo.com/pipes/pipe.info?_id=jroxOD1f3BGgvtiIqGIyXQ RSS Aggregator for blog posts about Django on Jython] == Summary of Django on Jython: The Gaps == The most important thing to know about Django on Jython is that we are almost there, and with clean code. End-to-end functionality is demonstrated by the admin tool running in full CRUD, along with a substantial number of unit tests. But this has been achieved by so far requiring only 6 lines of code in changes to Django trunk. (There will be more, however, see below.) == Running on Jython == To run Django on Jython, with a PostgreSQL backend, the following steps are necessary: * Use the [https://jython.svn.sourceforge.net/svnroot/jython/branches/modern/ Modern branch] of Jython. This consolidated the bugs, workarounds, and patches found below -- plus a bunch more -- in a stable, almost-ready-to-be-merged-into-trunk version of Jython. The most important aspect is that we have tried to make Jython conform more to CPython, using Django as our guide, although there are some gaps - especially if Django already had incorporated fixes. Our driving goal is to converge on these gaps over time. Please note that is intended to be stable, performant code. * Use the Django trunk (tested with rev 6992, later should be OK too). * Apply these two patches, [https://jython.svn.sourceforge.net/svnroot/jython/trunk/sandbox/jbaker/django/dispatch/robustapply.py django.dispatch.robustapply] ([https://hg.leosoto.com/django.jythonport/rev/bb2b9048ed14f72e99c5c133c7ecaaeb0a5c425c diff]) and [https://jython.svn.sourceforge.net/svnroot/jython/trunk/sandbox/jbaker/django/views/debug.py django.views.debug] ([https://hg.leosoto.com/django.jythonport/rev/663bcf5efc31409899fcdc10009bc92df3e1f6b9 diff]) due to Leo Soto. These should be in Django trunk soon. * Copy these three files from CPythonLib to Lib: gettext.py, locale.py, optparse.py. Please note that these files are only partially working on Jython, that's why they haven't been promoted yet (gettext.py actually works, as verified by test_gettext.py, but depends on still failing locale.py). But they are very close, and they appear to be fine for Django. Certainly fine for this round of development! * Use the database backend [https://jython.svn.sourceforge.net/svnroot/jython/trunk/sandbox/jbaker/django/db/backends/postgresql_zxjdbc/ zxjdbc_postgresql], which was contributed by Leo Soto. Frank Wierzbicki has an experimental backend for !MySQL, this should be incorporated soon. == Status == Here's what works: syncdb and the very cool Django admin run; many unit tests pass. You can run with internationalization enabled. You do need to run the dev server with --noreload for now. We need to document here how to run with [http://www.xhaus.com/modjy/ modjy], which is Alan Kennedy's servlet container for WSGI apps. In running the model unit tests, here are the things we seem to be missing, accounting for most of the approximately 75 failures: * Many doctests are fragile, because they depend on the dict traversal ordering; in Jython, this is different that CPython, and if we adopt {{{ConcurrentHashMap}}}, it's not even repeatable. This would seem to be a pervasive bug in Django. * We still have some encoding problems, again seen in doctests. An example where output is expected to be lower case hex, not upper case. I fixed the problem in {{{PyUnicode}}}, but there are more places. * Problem with the {{{ManagerDescriptor}}} handling, in {{{django.db.models.manager}}}. * No decorators yet! (But they are coming soon, and are now available experimentally for Jython.) There may be some other rough categories, we need to look at the failures more systematically. All that doctest noise is certainly annoying! == Next Steps == On the Django front, get more of the unit tests running! Before we can push modern into trunk, the following needs to be done: * The test_extcall unit test currently fails. This appears to be a dependency on {{{dict traversal}}} being repeatable. However, it's a mind bending test. The 2.3 version is particularly problematic because it's not modular at all. Google's GHOP has just produced an improved version for Python 2.6 - we will look at this as a starting point. * Tristan King provided a near complete subset of the functionality for {{{time.strptime}}}, as implemented in {{{org.python.modules.time.Time}}}. This needs to be enhanced. All unit tests in the CPythonLib version of test_time now pass except for {{{strptime}}} -- specifically the conversion specifier {{{'%c'}}} -- so we can also move to that, and discard our Jython version, when this is completed. That should be soon! * Decide whether we should use {{{ConcurentHashMap}}} or not as the backing hash map for {{{dict}}} and {{{__dict__}}}. CHM introduces creation overhead, but it should prove to be far more scalable on multicore systems. The programming model is also far nicer with respect to Jython. ---- == Historical Notes == (Preserved to show how much we had to go through...) === Formed Major Problems === * Jython does not support non-str keys in {{{__dict__}}} right now. Jim Baker replaced the custom dictionary in {{{PyStringMap}}} with a {{{HashMap}}} (along Groves' earlier work). This change should resolve this problem. Try the [https://jython.svn.sourceforge.net/svnroot/jython/branches/modern/ modern] branch. Problem encountered in {{{django/utils/_threading_local.py}}}. * Something goes wrong with {{{re.compile(u"([\u0080-\uffff])")}}} in {{{django/utils/text.py}}} I think it is a lacking in our array implementation but this needs more investigation. * See [http://sourceforge.net/tracker/index.php?func=detail&aid=1544953&group_id=12867&atid=112867 bug 1544953] * {{{django/views/debug.py}}} wants to know a function's {{{__module__}}}. functions don't have a {{{__module__}}} attribute in jython yet. [http://sourceforge.net/tracker/index.php?func=detail&aid=1800472&group_id=12867&atid=312867 patch] * {{{inspect.getargs()}}} doesn't work. * That's because {{{co_code}}} isn't supported by {{{PyTableCode}}}. * '''But''', {{{co_code}}} is used by {{{getargs()}}} only to discover anonymous tuple arguments. Commenting such section seems to be enough to get the other cases working. * workaround [https://dev.archer.edu.au/projects/kepler/svn/jython-django/trunk/jython-patches/co_code-workaround.patch patch]. just sets {{{co_code = ""}}}. i haven't come across any problems for this inside django yet, but i'm sure it'll break other python code -- TristanKing * {{{time}}} module lacks {{{strptime()}}}. * workaround [https://dev.archer.edu.au/projects/kepler/svn/jython-django/trunk/jython-patches/time-strptime.patch patch]. this is incomplete, untested, and not optimal but works for the django tutorial -- TristanKing * inspect.isclass(time) shouldn't return True [http://sourceforge.net/tracker/index.php?func=detail&aid=1786009&group_id=12867&atid=112867 bug] * Django [http://code.djangoproject.com/changeset/6194 works around this] until it's fixed. === Former Minor annoyances === * slice should be a type, not a function. [http://sourceforge.net/tracker/index.php?func=detail&aid=1791931&group_id=12867&atid=112867 bug] [http://sourceforge.net/tracker/index.php?func=detail&aid=1798544&group_id=12867&atid=312867 patch] * {{{PyBoolean.__tojava__}}} returns an {{{Integer}}} * It affects zxJDBC unless a DataHandler sets the right type of boolean parameters. Fortunately, JDBC30DataHandler does this thing, at least on my limited tests -- LeonardoSoto * Jython does not have {{{__builtins__}}} as a module. Django adds "_" as a convenience method for calling gettext in {{{django/conf/__init__.py}}}. * According to the Python docs, {{{__builtins__}}} is a implementation detail and shouldn't be used by portable applications, so it's a Django problem. (See note about {{{__builtins__}}} here: http://docs.python.org/ref/naming.html) * Jython methods are not descriptors. It seems a implementation detail. [http://code.djangoproject.com/ticket/5664 django patch] * Jython classes which extend dict cannot be compared to a dict. [https://sourceforge.net/tracker/index.php?func=detail&aid=1804011&group_id=12867&atid=112867 bug] [https://dev.archer.edu.au/projects/kepler/changeset/149?format=diff&new=149 django patch] * Jython doesn't recognize {{{__unicode__}}} methods assigned to instances. * But, this is the only {{{__*__}}} method where CPython allows this, and it's not documented anywhere. So, it's a implementation detail and should be fixed on Django. [http://code.djangoproject.com/ticket/5560 django patch] === Already Fixed === (And many more!) * Jython does not have {{{sys.stdout.isatty}}}. Jython should get added to the exclusions around that call in {{{django/core/management.py}}} (win32 and 'Pocket PC' are already excluded). * Fixed in [http://code.djangoproject.com/changeset/6032 Django changeset 6032] * stringmap (or the {{{__dict__}}} implementation) needs a {{{pop(o)}}} method. * Jython lists and tuples don't have {{{__iter__}}} . * Fixed in [http://code.djangoproject.com/changeset/6211 Django changeset 6211] * Jython does not have {{{os.getpid()}}}. * Django workarounds it now. * Jython does not have {{{os.W_OK}}} needed in {{{django/core/management.py}}}. This probably will not be hard to add this since {{{java.io.File}}} has a {{{canWrite()}}}. * {{{os.chmod}}} should be implemented too (After checking for write permissions, Django will try to change them if necessary). * That's hard. Maybe Django shouldn't check for permissions if running on Jython. * Django don't check or change file permissions when running on Jython now * The {{{U}}} (Universal newlines) flag for {{{open()}}} isn't recognized on Jython * Unicode string interpolation doesn't work * {{{object}}} has a {{{__unicode__}}} attribute (it doesn't on CPython) [https://sourceforge.net/tracker/?func=detail&atid=112867&aid=1800378&group_id=12867 bug] [https://sourceforge.net/tracker/?func=detail&atid=312867&aid=1800381&group_id=12867 patch] -- TristanKing * metaclasses need to know the {{{__module__}}} of its classes [http://sourceforge.net/tracker/index.php?func=detail&aid=1781500&group_id=12867&atid=112867 bug] [http://sourceforge.net/tracker/index.php?func=detail&aid=1789137&group_id=12867&atid=312867 patch] |
[http://pipes.yahoo.com/pipes/pipe.info?_id=jroxOD1f3BGgvtiIqGIyXQ RSS Aggregator for blog posts about Django on Jython]
Summary of Django on Jython: The Gaps
The most important thing to know about Django on Jython is that we are almost there, and with clean code. End-to-end functionality is demonstrated by the admin tool running in full CRUD, along with a substantial number of unit tests. But this has been achieved by so far requiring only 6 lines of code in changes to Django trunk. (There will be more, however, see below.)
Running on Jython
To run Django on Jython, with a PostgreSQL backend, the following steps are necessary:
Use the [https://jython.svn.sourceforge.net/svnroot/jython/branches/modern/ Modern branch] of Jython. This consolidated the bugs, workarounds, and patches found below -- plus a bunch more -- in a stable, almost-ready-to-be-merged-into-trunk version of Jython. The most important aspect is that we have tried to make Jython conform more to CPython, using Django as our guide, although there are some gaps - especially if Django already had incorporated fixes. Our driving goal is to converge on these gaps over time. Please note that is intended to be stable, performant code.
- Use the Django trunk (tested with rev 6992, later should be OK too).
Apply these two patches, [https://jython.svn.sourceforge.net/svnroot/jython/trunk/sandbox/jbaker/django/dispatch/robustapply.py django.dispatch.robustapply] ([https://hg.leosoto.com/django.jythonport/rev/bb2b9048ed14f72e99c5c133c7ecaaeb0a5c425c diff]) and [https://jython.svn.sourceforge.net/svnroot/jython/trunk/sandbox/jbaker/django/views/debug.py django.views.debug] ([https://hg.leosoto.com/django.jythonport/rev/663bcf5efc31409899fcdc10009bc92df3e1f6b9 diff]) due to Leo Soto. These should be in Django trunk soon.
- Copy these three files from CPythonLib to Lib: gettext.py, locale.py, optparse.py. Please note that these files are only partially working on Jython, that's why they haven't been promoted yet (gettext.py actually works, as verified by test_gettext.py, but depends on still failing locale.py). But they are very close, and they appear to be fine for Django. Certainly fine for this round of development!
Use the database backend [https://jython.svn.sourceforge.net/svnroot/jython/trunk/sandbox/jbaker/django/db/backends/postgresql_zxjdbc/ zxjdbc_postgresql], which was contributed by Leo Soto. Frank Wierzbicki has an experimental backend for !MySQL, this should be incorporated soon.
Status
Here's what works:
syncdb and the very cool Django admin run; many unit tests pass. You can run with internationalization enabled. You do need to run the dev server with --noreload for now. We need to document here how to run with [http://www.xhaus.com/modjy/ modjy], which is Alan Kennedy's servlet container for WSGI apps.
In running the model unit tests, here are the things we seem to be missing, accounting for most of the approximately 75 failures:
Many doctests are fragile, because they depend on the dict traversal ordering; in Jython, this is different that CPython, and if we adopt ConcurrentHashMap, it's not even repeatable. This would seem to be a pervasive bug in Django.
We still have some encoding problems, again seen in doctests. An example where output is expected to be lower case hex, not upper case. I fixed the problem in PyUnicode, but there are more places.
Problem with the ManagerDescriptor handling, in django.db.models.manager.
- No decorators yet! (But they are coming soon, and are now available experimentally for Jython.)
There may be some other rough categories, we need to look at the failures more systematically. All that doctest noise is certainly annoying!
Next Steps
On the Django front, get more of the unit tests running!
Before we can push modern into trunk, the following needs to be done:
The test_extcall unit test currently fails. This appears to be a dependency on dict traversal being repeatable. However, it's a mind bending test. The 2.3 version is particularly problematic because it's not modular at all. Google's GHOP has just produced an improved version for Python 2.6 - we will look at this as a starting point.
Tristan King provided a near complete subset of the functionality for time.strptime, as implemented in org.python.modules.time.Time. This needs to be enhanced. All unit tests in the CPythonLib version of test_time now pass except for strptime -- specifically the conversion specifier '%c' -- so we can also move to that, and discard our Jython version, when this is completed. That should be soon!
Decide whether we should use ConcurentHashMap or not as the backing hash map for dict and __dict__. CHM introduces creation overhead, but it should prove to be far more scalable on multicore systems. The programming model is also far nicer with respect to Jython.
Historical Notes
(Preserved to show how much we had to go through...)
Formed Major Problems
Jython does not support non-str keys in __dict__ right now. Jim Baker replaced the custom dictionary in PyStringMap with a HashMap (along Groves' earlier work). This change should resolve this problem. Try the [https://jython.svn.sourceforge.net/svnroot/jython/branches/modern/ modern] branch. Problem encountered in django/utils/_threading_local.py.
Something goes wrong with re.compile(u"([\u0080-\uffff])") in django/utils/text.py I think it is a lacking in our array implementation but this needs more investigation.
django/views/debug.py wants to know a function's __module__. functions don't have a __module__ attribute in jython yet. [http://sourceforge.net/tracker/index.php?func=detail&aid=1800472&group_id=12867&atid=312867 patch]
inspect.getargs() doesn't work.
That's because co_code isn't supported by PyTableCode.
But, co_code is used by getargs() only to discover anonymous tuple arguments. Commenting such section seems to be enough to get the other cases working.
workaround [https://dev.archer.edu.au/projects/kepler/svn/jython-django/trunk/jython-patches/co_code-workaround.patch patch]. just sets co_code = "". i haven't come across any problems for this inside django yet, but i'm sure it'll break other python code -- TristanKing
time module lacks strptime().
workaround [https://dev.archer.edu.au/projects/kepler/svn/jython-django/trunk/jython-patches/time-strptime.patch patch]. this is incomplete, untested, and not optimal but works for the django tutorial -- TristanKing
inspect.isclass(time) shouldn't return True [http://sourceforge.net/tracker/index.php?func=detail&aid=1786009&group_id=12867&atid=112867 bug]
Django [http://code.djangoproject.com/changeset/6194 works around this] until it's fixed.
Former Minor annoyances
slice should be a type, not a function. [http://sourceforge.net/tracker/index.php?func=detail&aid=1791931&group_id=12867&atid=112867 bug] [http://sourceforge.net/tracker/index.php?func=detail&aid=1798544&group_id=12867&atid=312867 patch]
PyBoolean.__tojava__ returns an Integer
It affects zxJDBC unless a DataHandler sets the right type of boolean parameters. Fortunately, JDBC30DataHandler does this thing, at least on my limited tests -- LeonardoSoto
Jython does not have __builtins__ as a module. Django adds "_" as a convenience method for calling gettext in django/conf/__init__.py.
According to the Python docs, __builtins__ is a implementation detail and shouldn't be used by portable applications, so it's a Django problem. (See note about __builtins__ here: http://docs.python.org/ref/naming.html)
Jython methods are not descriptors. It seems a implementation detail. [http://code.djangoproject.com/ticket/5664 django patch]
Jython classes which extend dict cannot be compared to a dict. [https://sourceforge.net/tracker/index.php?func=detail&aid=1804011&group_id=12867&atid=112867 bug] [https://dev.archer.edu.au/projects/kepler/changeset/149?format=diff&new=149 django patch]
Jython doesn't recognize __unicode__ methods assigned to instances.
But, this is the only __*__ method where CPython allows this, and it's not documented anywhere. So, it's a implementation detail and should be fixed on Django. [http://code.djangoproject.com/ticket/5560 django patch]
Already Fixed
(And many more!)
Jython does not have sys.stdout.isatty. Jython should get added to the exclusions around that call in django/core/management.py (win32 and 'Pocket PC' are already excluded).
Fixed in [http://code.djangoproject.com/changeset/6032 Django changeset 6032]
stringmap (or the __dict__ implementation) needs a pop(o) method.
Jython lists and tuples don't have __iter__ .
Fixed in [http://code.djangoproject.com/changeset/6211 Django changeset 6211]
Jython does not have os.getpid().
- Django workarounds it now.
Jython does not have os.W_OK needed in django/core/management.py. This probably will not be hard to add this since java.io.File has a canWrite().
os.chmod should be implemented too (After checking for write permissions, Django will try to change them if necessary).
- That's hard. Maybe Django shouldn't check for permissions if running on Jython.
- Django don't check or change file permissions when running on Jython now
The U (Universal newlines) flag for open() isn't recognized on Jython
- Unicode string interpolation doesn't work
object has a __unicode__ attribute (it doesn't on CPython) [https://sourceforge.net/tracker/?func=detail&atid=112867&aid=1800378&group_id=12867 bug] [https://sourceforge.net/tracker/?func=detail&atid=312867&aid=1800381&group_id=12867 patch] -- TristanKing
metaclasses need to know the __module__ of its classes [http://sourceforge.net/tracker/index.php?func=detail&aid=1781500&group_id=12867&atid=112867 bug] [http://sourceforge.net/tracker/index.php?func=detail&aid=1789137&group_id=12867&atid=312867 patch]