Differences between revisions 31 and 32
Revision 31 as of 2004-11-12 18:53:37
Size: 13935
Editor: VinaySajip
Comment:
Revision 32 as of 2004-11-27 03:27:13
Size: 16322
Editor: CPE-24-211-7-145
Comment:
Deletions are marked like this. Additions are marked like this.
Line 247: Line 247:
== ConfigParser, OptParse Marriage ==

A marriage of the two would make a lot of sense to me. My thought is for the user (script) to instantiate a parser and add options to it much like OptParse does today. When the parser processes the command line args (or args passed to it) it should also look at the configuration file (possibly using the long setting name) for the option settings. Its first priority would be command line args.

A hierarchical “key” or “namespace” scheme should exist so that multiple settings may be stored in the user configuration file. I would propose a standard -k, –key option always present in the parser so the user can pass a list of keys to control the group of settings to use (I’d also like some of the other ways to control keys that I use in my config.py module). In addition I’d propose a -c, –config option that specifies a configuration file to use instead of the user’s default configuration file.

My last proposal for today is to change the “options” interface a bit from OptParse. Instead of returning a static object of options it should be an object with a get() method so that additional keys can be used to get at settings in the configuration “on the fly”. One application where this is important is in test frameworks. “Test specification” input files may have additional “keys” to be used and aren’t known when the parser is instantiated.

One benefit of this marriage is that the help messages available from the command line would also apply to what can be set in the configuration file. Also there is a lot of flexibility for the user with this scheme. For options they always use they can hard code them in their default configuration as settings. If they want to temporarily override them they can use command line options. For groups of settings that are always used together, they can be used by simply passing in a key with the -k option to select the group.

The part that isn’t addressed here that bothers me yet is how to weave in the ability to pass in real python objects as settings. It would be good to have both a secure (restricted to strings and simple settings) and a second less than secure parser for those applications that need the flexibility of python. I’d like to see the API’s of the same for both the parser and the configuration file. -- dan.gass@gmail.com

There has been some recent debate about [http://python.org/doc/current/lib/module-ConfigParser.html ConfigParser] on the Python mailing lists. For more, see these threads:

This page serves as a place to record alternatives, and discuss (in a semi-permanent way) the features such a library might have. A new configuration parsing library could go into the Python standard library, probably in addition to the current ConfigParser module (perhaps with that module being deprecated).

Broken Out Sections

  • ConfigParserGoals: what is ConfigParser really for? (But note that the Features section is still relevant.) Also note that there has been some confusion between "In memory storage of configuration data" and "Simple persistent storage of configuration data". Part of the problem is that almost every configuration storage system (including ConfigParser, optparse, and getopt) comes with its own in-memory API. There should be some uniform means of accessing data from configuration files and command line options parsed via optparse or getopt, including the ability to override options in configuration files with command line options. Ideally, the programmer API should not normally care whether an option was set in the configuration or the command line.

Implementations

Please list interesting implementations of config parsers here.

[http://www.voidspace.org.uk/atlantibots/configobj.html ConfigObj - A simple to use config file parser] Not really an alternative to ConfigParser, but a very easy to use config file parser. Dictionary like syntax with the ability to save modified config files. Preserves comments but not indentation.

M. Chermside's candidate

[http://www.mcherm.com/publish/2004-10-17/config.py code] and [http://www.mcherm.com/publish/2004-10-17/configTest.py test cases]. Currently allows files in either str or unicode, with sensible defaults. Allows dictionary or dotted-name access (though dotted-name can fail in some cases). Allows subsections of arbitrary length. For example,

    x.y = True 
    x.y.z = 47
    x.y.a = prime

Would allow x.y to be viewed as either a value ("True") or a section.

    x.y == "True"

but also

    x.y['a'] = 'prime'
    x.y['z'] = '47'

Note that keys and values are always strings or unicode -- no autoconversion to other types. Note that this focuses on storage and API -- reading and writing is left out at the moment, and might reasonably be in a separate module for each format supported.

.ini file parser and schema

Now contains two modules -- iniparser is a parser for .ini files. It needs to be subclassed to produce useful output. By subclassing the parser you can implement complex or simple config parsers, while allowing for context-sensitive error messages.

inischema allows schema definitions, with type specifications, attribute access, etc. It is still quite young, and missing some major features (like handling sections).

Available at:

Skip's Idea

In my use of INI files I've always been annoyed that I couldn't nest sections to an arbitrary depth and had to resort to baroque XML APIs to accomplish that sort of task. I also figured a structure defined by indentation would be a good way to go, though YAML always seemed too complex. I worked up a little [http://www.musi-cal.com/~skip/python/cfgparse.py config file parser] that reads and writes files like

empty section1:
level1 = new val
section1:
# this is a comment for section1.item1:
    item1 = item 1
          # this is another comment
    subsection:  
        item2 = item 2
section2:
   subsection:
       item3 = item 3
very last = 7

Dan Gass'

[https://sourceforge.net/projects/config-py/ config-py] allows either strings (safe) or evaluated code (powerful) with the same API. It needs python 2.3, or at least dict.pop()

Vinay Sajip's implementation

The [http://www.red-dove.com/python_config.html config] module allows a hierarchical configuration scheme with support for mappings and sequences, cross-references between one part of the configuration and another, the ability to flexibly access real Python objects without full-blown eval(), an include facility, simple expression evaluation and the ability to change, save, cascade and merge configurations. It has been developed on python 2.3 but should work on version 2.2 or greater.

A simple example - with the example configuration file:

messages:
[
  {
    stream : `sys.stderr`
    message: 'Welcome'
    name: 'Harry'
  }
  {
    stream : `sys.stdout`
    message: 'Welkom'
    name: 'Ruud'
  }
  {
    stream : $messages[0].stream
    message: 'Bienvenue'
    name: Yves
  }
]

a program to read the configuration would be::

from config import Config

f = file('simple.cfg')
cfg = Config(f)
for m in cfg.messages:
    s = '%s, %s' % (m.message, m.name)
    try:
        print >> m.stream, s
    except IOError, e:
        print e

which, when run, would yield the console output::

Welcome, Harry
Welkom, Ruud
Bienvenue, Yves

cfgparse

[http://www.cs.wisc.edu/~param/software/cfgparse/ cfgparse] - cfgparse is a Python module that provides mechanisms for managing configuration information. It is backward compatible with ConfigParser, in addition to having the following features:

  • Preserves structure of INI files: Order of sections & options, indentation (to some extent), comments, and blank lines are preserved when data is updated.

  • More convenient than ConfigParser: Values can be accessed using dotted notation, or using container syntax (cfg[key]).

  • Backward compatibility: Backward compatible implementations of ConfigParser, RawConfigParser, and SafeConfigParser are included that are API-compatible with the Python standard library. They pass all the unit tests in Python-2.3.4.

  • Extensible: It is possible to add other configuration formats, and to convert between different formats (as long as the data models are compatible).

ZConfig

[http://www.zope.org/Members/fdrake/zconfig/ ZConfig] - This Python package is a bit larger than some of the others, but provides for schema-based development of configuration structures. The schema language uses XML, but the configuration language is more like Apache's. Sections are typed and completely nestable.

Features

This is a list of features that should be taken into account. Certainly not all these features are required; maybe some aren't even desired.

  • Uses a human-editable source file format. Everyone seems fairly happy with INI files as a basis, but

    [http://cvs.zope.org/Zope3/doc/zcml/ ZCML] is an example of a different syntax.

  • Has a simple API, especially for the simplest case (read a bunch of key-value string pairs from a file).
  • Can be round-tripped, i.e. a program can modify the configuration, then write the new configuration out.
    • Bonus if you can keep comments intact when writing the file. Or at least keep track of the comments in some way, so it can be extended by someone else later (throwing away comments entirely puts up brick walls).
    • Keeps track of the order of items in the configuration file.
    • Is intelligent about default values; writes out a minimal config file.
  • Supports some notion of a config schema. You might define a certain key as being an integer, and it would be automatically converted when the file was loaded. This becomes a bit more difficult when you have more complex structures, multiple sections, etc. Defaults fit in here as well. Obviously the types available should not be fixed.
  • Supports repeating values in some fashion. Maybe a key can appear multiple times in the file. Maybe nested structures are allowed otherwise. Repeating values are common.
  • Allows multiple config files to be combined, e.g., a site config, user config, custom config. Consideration should be given to repeating values; sometimes a user config file may want to add to a site config without repeating all items in the site config.
  • Allows dynamic nesting. E.g., some configuration values may be modified only in a small context, or for a single request in a long-running server. The nesting needs to be undone later. Maybe this simply requires the config to be easily copiable (i.e., copy the config, modify that copy, throw it away when you are done); maybe something more sophisticated is possible.
  • Supports multiple syntaxes; e.g., an XML plist syntax, an INI syntax, a ZCML syntax.
  • Gives good error messages. Error messages should include file and line number. If we define types in a schema, invalid values should give good errors.
  • Supports (maybe through utility functions or otherwise) more advanced configuration directives, like including other files or string

    interpolation. But a lot of people don't like ConfigParser's current string interpolation.

  • Consider the general case of configuration settings, not just config.ini. While round-trip to human readable is important, it is also important that the in-memory version play well with other ways of setting parameters -- including some that set arbitrary python objects.

To be more explicit, it should work well with at least optparse (Optik), .ini files, .xml files, and computed-at-runtime values. The interface to the various storage mechanisms can be different, but developers shouldn't have to repeat information across the various formats; adding an option (and default value/help message/restrictions) should only need to be done once.

  • Backward compatible; at least to Python 2.2, best if portable to Python 2.1.
  • Implements a true parser that can be subclassed and specialized. If it only has a method that parses the config file into a dictionary, then any attempts to extend or specialize the parser won't have access to line information. The parser need not be a high-profile part of the module, so long as it is available.

Discussion

Discuss. Please sign your name.

What exactly is the goal? A new API to access configuration info? Or a specific file format itself? Or both? I don't have much problem with the current ConfigParser but ideally I would like use a 'simpler' API. This would allow attribute access (a.b.c) to values, provide default values, convert some types, and do some constraint checking (xyz is required) etc. It's very possible to get this functionality through a wrapper on top of ConfigParser. IMO that is the best approach, as long as there is a way to map the same API over a different underlying file format, such as XML. I think the 'dynamic nesting' point above is outside the scope of the config access API. -- Shalabh

Three features I want in a config parser are 1) keyed settings 2) pulling in settings from multiple configurition files, and 3) ability for user to pass in real python objects through the settings. The "keyed" settings can be thought of as namespaces and I need an arbitrary number of key nestings. For certain applications I EXPECT the user to pass in python objects that meet a specified API. This allows the user to customize a certain operation however they would like with the full power and flexibility of python. I then don't need my tool to be tailered with a switch statement having custom solutions for each user type. This would require the configuration file (or parts of it) to be able to be executed as a python script and I realize this is a security hole that would be unacceptable to many. What I would propose is the config parser module support two methodologies, both sharing the same API and configuration file syntax. One would parse the config file and prevent security issues, the other would either execute the whole config file or be a combination parse/execute but would support attaching real python objects to configuration settings. These features have been implemented in a configuration parser https://sourceforge.net/projects/config-py/ (needs python 2.3 unless the dict pops are reimplemented) and is available for experimentation/use. -- dan.gass@gmail.com 28oct04

I think it is reasonable to ask the setting code to create the object (possibly by running a random string); the config system just needs to accept objects that have already been made. A round-trip is useful, but I'm not sure source code is the best way to do that; editing will probably require an external tool anyhow. Maybe just use pickle to save arbitrary objects? (And avoid storing them *within* the config, as much as possible.) -- Jim J Jewett

ConfigParser, OptParse Marriage

A marriage of the two would make a lot of sense to me. My thought is for the user (script) to instantiate a parser and add options to it much like OptParse does today. When the parser processes the command line args (or args passed to it) it should also look at the configuration file (possibly using the long setting name) for the option settings. Its first priority would be command line args.

A hierarchical “key” or “namespace” scheme should exist so that multiple settings may be stored in the user configuration file. I would propose a standard -k, –key option always present in the parser so the user can pass a list of keys to control the group of settings to use (I’d also like some of the other ways to control keys that I use in my config.py module). In addition I’d propose a -c, –config option that specifies a configuration file to use instead of the user’s default configuration file.

My last proposal for today is to change the “options” interface a bit from OptParse. Instead of returning a static object of options it should be an object with a get() method so that additional keys can be used to get at settings in the configuration “on the fly”. One application where this is important is in test frameworks. “Test specification” input files may have additional “keys” to be used and aren’t known when the parser is instantiated.

One benefit of this marriage is that the help messages available from the command line would also apply to what can be set in the configuration file. Also there is a lot of flexibility for the user with this scheme. For options they always use they can hard code them in their default configuration as settings. If they want to temporarily override them they can use command line options. For groups of settings that are always used together, they can be used by simply passing in a key with the -k option to select the group.

The part that isn’t addressed here that bothers me yet is how to weave in the ability to pass in real python objects as settings. It would be good to have both a secure (restricted to strings and simple settings) and a second less than secure parser for those applications that need the flexibility of python. I’d like to see the API’s of the same for both the parser and the configuration file. -- dan.gass@gmail.com

Complexity killed the cat!

Look at what happened to urllib2 - the entry level for new users was greatly enhanced, but the added functionality was really little more than what you could already do more simply with urllib. With that in mind, I think the new ConfigParser should in it's simplest form act very closely like the original ConfigParser.

ConfigParserShootout (last edited 2014-04-02 18:08:02 by SkipMontanaro)

Unable to edit the page? See the FrontPage for instructions.