Differences between revisions 1 and 2
Revision 1 as of 2012-06-24 06:38:27
Size: 2244
Editor: JeffAllen
Comment: Start describing a buffer protocol
Revision 2 as of 2012-06-24 07:37:52
Size: 3789
Editor: JeffAllen
Comment: References and design
Deletions are marked like this. Additions are marked like this.
Line 53: Line 53:

== References ==

 1. [[http://python.org/dev/peps/pep-3118/|PEP 3118]] Revising the buffer protocol
 1. [[http://docs.python.org/library/stdtypes.html#memoryview-type|memoryview type]] (in Python 2.7 docs)
 1. [[http://docs.python.org/py3k/library/stdtypes.html#memoryview-type|memoryview type]] (in Python 3 docs)
 1. [[http://docs.python.org/c-api/typeobj.html#buffer-object-structures|Buffer Object Structures]] (in Python 2.7 docs)
 1. [[http://docs.python.org/py3k/c-api/buffer.html|Buffer Protocol]] (in Python 3 docs)
 1. [[http://bugs.python.org/issue10181| CPython issue 10181]] Problems with Py_buffer management in `memoryobject.c` (and elsewhere?)

http://docs.python.org/py3k/c-api/buffer.html



== Design ==
=== Considerations ===

 * We would benefit from something in the role of the CPython buffer API.
 * It should be as familiar as possible to people who know the CPython buffer API.
 * It should be a feasible basis for `memoryview`.
 * Java is not C:
   * some CPython buffer API idioms have no equivalent, e.g. {{{((float*)ap->ob_item)[i] = x;}}}
   * and some may be done in a better way (e.g. polymorphism in place of if statements, object lifetime).
 * 99% (maybe 100%) of core development only accesses the buffer as a one-dimensional array of bytes.
 * Some significant applications (NumPy, PIL) use the richer facilities (>1 dimension, strided access, indirection).


=== Sketch interface to bytes ===


=== Provision for things other than bytes ===


=== Combined proposal ===

Buffer Protocol

This page proposes (later documents) a design for a Jython equivalent to the CPython buffer protocol.

Introduction (Situation in June 2012)

In CPython, certain objects are based on an underlying memory array or buffer. The CPython designers judged it desirable to be able to access that buffer directly, without intermediate copying. CPython provides this at the C level in the form of the buffer protocol. This is used heaviliy in the implementation of some core types and standard library modules. And it is the basis of the type memoryview.

Although the buffer API and memoryview are Python 3k features, they were backported into CPython 2.7. In C the buffer protocol the exporting object gives consumers a pointer to memory and some information about how it is structured.

Jython does not (yet) have an equivalent of buffer protocol or support memoryview. There is a stub for each, but no access to data through it. In the recent implementation of PyByteArray absence of a buffer protocol implemented by incoming arguments was a complicating factor. In CPython, the majority of methods start by getting a buffer view of their arguments: acceptable types are all those that implement the API. Other types and modules present a similar challenge in reaching 2.7 compliance. But we cannot directly emulate the C buffer protocol in Java, since it does not allow pointers to memory like:

   1     self_size = _getbuffer(self, &self_bytes);
   2     other_size = _getbuffer(other, &other_bytes);
   3         ...
   4     cmp = memcmp(self_bytes.buf, other_bytes.buf, minsize);

or casts like:

   1         ((float*)ap->ob_item)[i] = x;

Yet, without something filling the role, core implementation is made more difficult and always falls short of compatibility.

This raises two questions:

  • Can we create a buffer API that provides an equivalent facility in Java?
  • Can we go on from there to implement memoryview?

This page looks at the first of these, arguing for a particular approach. In due course, it ought to change into documentation of the approach, preserving the rationale of the final design.

References

  1. PEP 3118 Revising the buffer protocol

  2. memoryview type (in Python 2.7 docs)

  3. memoryview type (in Python 3 docs)

  4. Buffer Object Structures (in Python 2.7 docs)

  5. Buffer Protocol (in Python 3 docs)

  6. CPython issue 10181 Problems with Py_buffer management in memoryobject.c (and elsewhere?)

http://docs.python.org/py3k/c-api/buffer.html

Design

Considerations

  • We would benefit from something in the role of the CPython buffer API.
  • It should be as familiar as possible to people who know the CPython buffer API.
  • It should be a feasible basis for memoryview.

  • Java is not C:
    • some CPython buffer API idioms have no equivalent, e.g. ((float*)ap->ob_item)[i] = x;

    • and some may be done in a better way (e.g. polymorphism in place of if statements, object lifetime).
  • 99% (maybe 100%) of core development only accesses the buffer as a one-dimensional array of bytes.
  • Some significant applications (NumPy, PIL) use the richer facilities (>1 dimension, strided access, indirection).

Sketch interface to bytes

Provision for things other than bytes

Combined proposal

BufferProtocol (last edited 2014-06-14 14:38:51 by JeffAllen)