Differences between revisions 1 and 3 (spanning 2 versions)
Revision 1 as of 2004-10-12 21:27:59
Size: 4162
Comment: Copy of topics/scicomp/storage.html
Revision 3 as of 2006-08-04 05:45:14
Size: 5109
Editor: MaryHaley
Comment: Added entry for PyNIO.
Deletions are marked like this. Additions are marked like this.
Line 21: Line 21:
          + [http://www.pyngl.ucar.edu/Nio.shtml PyNIO] - A Numeric-based Python package that allows read and/or write access to a variety of data formats (NetCDF, HDF 4, GRIB) using an interface modelled on Konrad Hinsen's [http://starship.python.net/crew/hinsen/scientific.html NetCDF interface].
Line 24: Line 25:
     * HDF5 interfaces
       Interface to the [http://hdf.ncsa.uiuc.edu/HDF5/ HDF5] format (hierachically organised datasets).

     HDF5 is a general purpose library and file format for storing scientific data.
     It's more complex and powerful than NetCDF, and the forthcoming NetCDF-4 is based on it.
          + [ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/contrib/hl-hdf5/README.html PyHL interface] - A High Level Interface to the HDF5 File Format. (Anders Henja and Daniel B. Michelson)

          + [http://pytables.sf.net/ PyTables interface] - HDF5 interface with full support of 64-bit data address and data indexing. (Carabos Coop. V.)
Line 62: Line 71:
  • Scientific applications are known for their ability to generate huge amounts of data which are sometimes hard to manage. This page lists some of the tools which have been made available for interfacing with standard scientific file formats, as well as Python-specific tools for manipulating arrays and text files.

Interfaces to Standard Formats

Python-specific Tools

  • [http://php.iupui.edu/~mmiller3/python/#TableIO TableIO] by Mike Miller.

    • "When I first started using Python, I wanted to read lots of numbers into

      NumPy arrays. This can be done with the standard Python file reading methods, but I found that to be prohibitively slow for largish data sets. So I wrote TableIO (_tableio.c and TableIO.py), which lets me start with a file containing a rectangular array of ASCII data (a `table') and read it into Python so I can manipulate it. For example, if I have a file containing an table in a file with 10 columns and 50 rows, I can use

        >>> d = TableIO.readTableAsArray(file)
      to get an array with shape (50,10). If I only want to read a couple of columns,

      say the first and ninth and tenth, I can use

        >>> [x, y, dy] = TableIO.readColumns(file, [0, 8, 9])
      to read the first column in to the 1D array x and the eigth and ninth into yand dy."
  • [http://starship.python.net/crew/hinsen/scientific.html FortranFormat.py] by Konrad Hinsen.

    • "This module provides two classes that aid in reading and writing Fortran-formatted text files. Only a subset of formatting options is supported: A, D, E, F, G, I, and X formats, plus string constants for output. Repetition (e.g. 4I5 or 3(1X,A4)) is supported. Complex numbers are not supported; you have to treat real and imaginary parts separately."
  • [http://starship.python.net/~da/Travis/numpyio/ numpyio] by Travis Oliphant.

    • "Once compiled, numpyio is a loadable module that can be used in python for reading and writing arbitrary binary data to and from Numerical Python arrays. I work in Medical Imaging and often have large data sets to manipulate. I do much of my interactive data analysis with MATLAB, however, only having doubles to work with really puts a crimp on the sizes of the data sets I can manipulate. The fact that Numerical Python has more data types defined than doubles encouraged me to try it out. I have been very impressed with its speed and utility, but I needed some way to read large data sets from an arbitrary binary file into Numerical Python arrays. I didn't see any obvious way to do this so I wrote an extension module. Although there is not much documentation, having the sources available is ultimately better than documentation. But, as this is my first extension module, my style may not be elegant as I may not be using the correct APIs. Feel free to send me corrections."

NumericAndScientific/Formats (last edited 2008-11-15 14:00:18 by localhost)

Unable to edit the page? See the FrontPage for instructions.