In 2004 there was a debate about ConfigParser on the Python mailing lists. But since that initial discussion there hasn't been any strong push to resolve the situation. For more, see these threads:

This page serves as a place to record alternatives, and discuss (in a semi-permanent way) the features such a library might have. A new configuration parsing library could go into the Python standard library, probably in addition to the current ConfigParser module (perhaps with that module being deprecated).

See the "Status of Shootout" section (from about 2006) for a summary of the major period of discussion which lasted on the order of a year.

Goals

Discussion of people's goals for revising ConfigParser have been broken out to the page ConfigParserGoals: what is ConfigParser really for?

Also note that there has been some confusion between "In memory storage of configuration data" and "Simple persistent storage of configuration data". Part of the problem is that almost every configuration storage system (including ConfigParser, optparse, and getopt) comes with its own in-memory API. There should be some uniform means of accessing data from configuration files and command line options parsed via optparse or getopt, including the ability to override options in configuration files with command line options. Ideally, the programmer API should not normally care whether an option was set in the configuration or the command line.

Implementations

Please list interesting implementations of config parsers here.

ConfigObj 4

ConfigObj - A simple to use config file parser

This is now a very powerful config file parser that addresses many of the issues raised here.

As of version 4 it reads and writes sections nested to *any* depth. It uses square brackets round section markers to denote nesting. this means it is compatible with most files written for ConfigParser : ::

    key = value
    key2 = member1, member2, memebr3
    [section name]
        key = value
        key2 = value2
        [[sub-section]]
            key = value
            key2 = value2

Each section is acessed as a dictionary.

It has various features (e.g. list values), and can be initialised from a variety of sources (list of lines, StringIO instance, filename). Because of the straightforward write method it is also very useful for data persistence.

e.g.

    config = ConfigObj()
    config['key'] = value
    config['section'] = {'key': 'value', 'key2': ['val1', 'val2']}
    config.filename = filename
    config.write()

ConfigObj also has a feature I *think* is unique - in the shape of a tightly integrated type checking/conversion system. This is (allegedly) substantially simpler than the type system of ZConfig and *doesn't* involve storing type info in the config file.

The type specification is kept in a separate schema (which has a simple key = checkname(parameters) syntax and also allows for default values. The validation process checks that the config file matches the schema and fills in any default values. It also converts all values that pass into the required type.

This means a ConfigObj is an abstraction of *config data* - not just the config file, at no cost to the *user*. The validation system is simple and extendable.

M. Chermside's candidate

code and test cases. Currently allows files in either str or unicode, with sensible defaults. Allows dictionary or dotted-name access (though dotted-name can fail in some cases). Allows subsections of arbitrary length. For example,

    x.y = True
    x.y.z = 47
    x.y.a = prime

Would allow x.y to be viewed as either a value ("True") or a section.

    x.y == "True"

but also

    x.y['a'] = 'prime'
    x.y['z'] = '47'

Note that keys and values are always strings or unicode -- no autoconversion to other types. Note that this focuses on storage and API -- reading and writing is left out at the moment, and might reasonably be in a separate module for each format supported.

INITools

INITools is factored into a parser that conforms to the ConfigParser sense of what an INI file is (initools.iniparser), and a couple of implementations based on that. One of those implementations is initools.configparser, which is compatible with the standard library ConfigParser.

It includes several features that can normally be built on to ConfigParser, but enables these through simple class variables per feature. These include features like:

Because it preserves all the information from the file you could write a view on top of this structure, to present any reasonable API -- the ConfigParser API is awkward, but given a few additions it is at least a reasonably complete description.

Skip's Idea

In my use of INI files I've always been annoyed that I couldn't nest sections to an arbitrary depth and had to resort to baroque XML APIs to accomplish that sort of task. I also figured a structure defined by indentation would be a good way to go, though YAML always seemed too complex. I worked up a little config file parser that reads and writes files like

empty section1:
level1 = new val
section1:
# this is a comment for section1.item1:
    item1 = item 1
          # this is another comment
    subsection:
        item2 = item 2
section2:
   subsection:
       item3 = item 3
very last = 7

Dan Gass'

I've just released a new configuration parser (http://cfgparse.sourceforge.net/). It has many of the features outlined as desireable:

The documentation is complete and is available in HTML and PDF. The home page (http://cfgparse.sourceforge.net/) is the HTML version online. The documentation and the module functionality is quite complete and useful as it is. But I am looking forward to feedback so that it may be improved. The distribution contains the module, help documentation (including source) and a fairly extensive test suite (you'll need Python2.4 to run the tests).

I figure I will make an announcement after any dust settles from this posting. By the way, I apologize for using the same name (cfgparse) as another entry but since this is modelled after optparse, that was the most logical choice.

Enjoy, Dan Gass -- dan.gass@gmail.com

Vinay Sajip's implementation

The config module allows a hierarchical configuration scheme with support for mappings and sequences, cross-references between one part of the configuration and another, the ability to flexibly access real Python objects without full-blown eval(), an include facility, simple expression evaluation and the ability to change, save, cascade and merge configurations. It has been developed on python 2.3 but should work on version 2.2 or greater.

A simple example - with the example configuration file:

messages:
[
  {
    stream : `sys.stderr`
    message: 'Welcome'
    name: 'Harry'
  }
  {
    stream : `sys.stdout`
    message: 'Welkom'
    name: 'Ruud'
  }
  {
    stream : $messages[0].stream
    message: 'Bienvenue'
    name: Yves
  }
]

a program to read the configuration would be::

from config import Config

f = file('simple.cfg')
cfg = Config(f)
for m in cfg.messages:
    s = '%s, %s' % (m.message, m.name)
    try:
        print >> m.stream, s
    except IOError, e:
        print e

which, when run, would yield the console output::

Welcome, Harry
Welkom, Ruud
Bienvenue, Yves

One problem I have with this implementation is the configuration file syntax. I respect the need for a syntax to handle dictionaries and lists but why invent yet another language? If one wants a Python like syntax make it the Python syntax. I don't think everyone is on board with a Python syntax for configuration files though. My biggest concern is to have a syntax that supports heirarchies and being able to construct Python objects for configuration settings. I lean toward using Python as the syntax (and even the parser) because of the flexibility offered but if there was another way that makes sense I'm ok with that. -- dan.gass@gmail.com

My reasons for not using Python itself for the syntax and parser are:

  1. I couldn't see how to just parse the required subset of Python - lists and dicts - while preventing arbitrary code from being executed.
  2. I wanted to be more forgiving of missing commas.
  3. I wanted to allow easy cross-referencing, inclusion and evaluation, using a more compact notation, which precludes the use of standard Python.

If someone could show me a way to meet the above desires whilst using Python syntax and parser, I'll gladly revisit the issue. -- VinaySajip

cfgparse

cfgparse - cfgparse is a Python module that provides mechanisms for managing configuration information. It is backward compatible with ConfigParser, in addition to having the following features:

ZConfig

ZConfig - This Python package is a bit larger than some of the others, but provides for schema-based development of configuration structures. The schema language uses XML, but the configuration language is more like Apache's. Sections are typed and completely nestable. The basic implementation does have some limitations that are tedious to work around if you run into them. One that can bite quickly is that names in the configuration language are case-insensitive by default; for versions before 2.3.1 this was terribly difficult to work around without copying lots of code, and even with 2.3.1 it takes more than it should.

tconfpy

OK, I'll Toss My Hat Into The Ring. I just found this discussion tonight for the first time. A fascinating topic and one very much dear to my heart. So much so that, ahem, uh, ... http://www.tundraware.com/Software/tconfpy

configparse

configparse is an extension that is built on top of the command line parsing library optparse. It provides the same interface and is intended to be uses as a drop-in-replacement for optparse. configparse is very limited in its abilities, there's no support for sections, recursiveness or sophisticated value checking etc. Its advantage is that it takes only a few modifications to existing code to add simple support for config files which can be quite handy sometimes.

plistlib

plistlib plistlib is a small module for generating and parsing Mac OS X .plist files. It supports:

As the default config format on OS X, plist files are already used by hundreds of apps. Though popular on the Mac, the format can be used from any platform or language since it's a subset of XML. Graphical editors are available from Apple as part of its free developer tools, as well as from third party developers

PyOptionTree

PyOptionTree is a hierarchical parameter parser that I wrote with the goal being to allow the user to both specify parameters *and* modify, control, structure, copy, and record them in a efficient and intuitive way. I've been using and refining it for over a year, and it has many of the options that people seem to find desirable, plus a few more, so I thought it'd be worthwhile to post here. If people have any comments, please let me know.

It supports:

Here's a simple example:

Tasks = [exercises/jog, eating/eatcereal, eating/eattoast]   # links to the following subtrees

exercises = {
  jog = {
    action = "jog"
    minutes = 30
  }
  # etc.
}

eating = {
  eatcereal = {
    action = "eat"
    food = "cereal"
  }
  
  eattoast = copy(eatcereal)  # this creates a copy of the eatcereal subtree
  eattoast/food = "toast"     # this changed the food option of that copy
}

On the programmer's end, each of the braches behaves like a full tree, e.g.:

def runTests(opttreefile):
    ot = PyOptionTree(opttreefile)
    for t in ot('Tasks'):   # According to the above definition,
        runTest(t)         # t is a subtree (branch).

def runTest(ot):
    print 'Current Action: ', ot("name")

configparser2

A slightly cleaned up version of the original ConfigParser here.

Features

This is a list of features that should be taken into account. Certainly not all these features are required; maybe some aren't even desired.

To be more explicit, it should work well with at least optparse (Optik), .ini files, .xml files, and computed-at-runtime values. The interface to the various storage mechanisms can be different, but developers shouldn't have to repeat information across the various formats; adding an option (and default value/help message/restrictions) should only need to be done once.

Discussion

Discuss. Please sign your name.

What exactly is the goal? A new API to access configuration info? Or a specific file format itself? Or both? I don't have much problem with the current ConfigParser but ideally I would like use a 'simpler' API. This would allow attribute access (a.b.c) to values, provide default values, convert some types, and do some constraint checking (xyz is required) etc. It's very possible to get this functionality through a wrapper on top of ConfigParser. IMO that is the best approach, as long as there is a way to map the same API over a different underlying file format, such as XML. I think the 'dynamic nesting' point above is outside the scope of the config access API. -- Shalabh

Three features I want in a config parser are 1) keyed settings 2) pulling in settings from multiple configurition files, and 3) ability for user to pass in real python objects through the settings. The "keyed" settings can be thought of as namespaces and I need an arbitrary number of key nestings. For certain applications I EXPECT the user to pass in python objects that meet a specified API. This allows the user to customize a certain operation however they would like with the full power and flexibility of python. I then don't need my tool to be tailered with a switch statement having custom solutions for each user type. This would require the configuration file (or parts of it) to be able to be executed as a python script and I realize this is a security hole that would be unacceptable to many. What I would propose is the config parser module support two methodologies, both sharing the same API and configuration file syntax. One would parse the config file and prevent security issues, the other would either execute the whole config file or be a combination parse/execute but would support attaching real python objects to configuration settings. These features have been implemented in a configuration parser https://sourceforge.net/projects/cfgparse/ (needs python 2.3 unless the use of the textwrap module is removed) and is available for experimentation/use. -- dan.gass@gmail.com

I think it is reasonable to ask the setting code to create the object (possibly by running a random string); the config system just needs to accept objects that have already been made. A round-trip is useful, but I'm not sure source code is the best way to do that; editing will probably require an external tool anyhow. Maybe just use pickle to save arbitrary objects? (And avoid storing them *within* the config, as much as possible.) -- Jim J Jewett

ConfigParser, optparse Marriage

A marriage of the two would make a lot of sense to me. My thought is for the user (script) to instantiate a parser and add options to it much like optparse does today. When the parser processes the command line args (or args passed to it) it should also look at the configuration file (possibly using the long setting name) for the option settings. Its first priority would be command line args.

Assume that the config file is simply

#This file is config.cfg
verbose : False

and suppose you want to be able to override this using the command line. One possible program is:

#This is test.py
from config import Config, ConfigList
from optparse import OptionParser

def getMergedConfig(filename):
    optcfg = Config()
    filecfg = Config(filename)
    parser = OptionParser()
    parser.add_option('-v', '--verbose', action='store_true', dest='verbose', help='Produce verbose output')
    args = parser.parse_args(None, optcfg)[1]
    cfglist = ConfigList()
    cfglist.append(optcfg)
    cfglist.append(filecfg)
    return cfglist, args

def main():
    cfg, args = getMergedConfig('config.cfg')

    print "verbose output? %r" % cfg.getByPath('verbose')
    print "args: %s" % args

main()

If you run this with

python test.py an_arg

the output is

verbose output? False
args: ['an_arg']

whereas if you run this with

python test.py -v an_arg

the output is

verbose output? True
args: ['an_arg']

This admittedly simple example points to how simple the actual application code - i.e. main() could be. I'm assuming that getMergedConfig() is a utility function which could be shared across multiple scripts, but even so it is pretty simple, in my view. Notice that main() does not know where the configured value of 'verbose' came from.

To forestall the complaint that the optparse specification appears to require some duplication, you could change the config file to:

#This file is config.cfg
verbose : False
optparse:
[
  { name: verbose, short: '-v', long: '--verbose', action: 'store_true', default: None, help: 'Produce verbose output'}
]

and the code which sets the parser options to (indent appropriately):

for optspec in filecfg.optparse:
    parser.add_option(optspec.short, optspec.long, action=optspec.action, dest=optspec.name, help=optspec.help)

Remember, this is just one way of doing it - not the only way and perhaps not the best way for your needs, but a simple enough way.

-- VinaySajip

A hierarchical “key” or “namespace” scheme should exist so that multiple settings may be stored in the user configuration file. I would propose a standard -k, –key option always present in the parser so the user can pass a list of keys to control the group of settings to use (I’d also like some of the other ways to control keys that I use in my config.py module). In addition I’d propose a -c, –config option that specifies a configuration file to use instead of the user’s default configuration file.

My last proposal for today is to change the “options” interface a bit from optparse. Instead of returning a static object of options it should be an object with a get() method so that additional keys can be used to get at settings in the configuration “on the fly”. One application where this is important is in test frameworks. “Test specification” input files may have additional “keys” to be used and aren’t known when the parser is instantiated.

path = sys.platform + ".database.connection"
connstr = cfg.getByPath(path)

One benefit of this marriage is that the help messages available from the command line would also apply to what can be set in the configuration file. Also there is a lot of flexibility for the user with this scheme. For options they always use they can hard code them in their default configuration as settings. If they want to temporarily override them they can use command line options. For groups of settings that are always used together, they can be used by simply passing in a key with the -k option to select the group.

The part that isn’t addressed here that bothers me yet is how to weave in the ability to pass in real python objects as settings. It would be good to have both a secure (restricted to strings and simple settings) and a second less than secure parser for those applications that need the flexibility of python. I’d like to see the API’s of the same for both the parser and the configuration file. -- dan.gass@gmail.com

Extension to configuration editing

The configuration parsers mentioned so far may handle textual configuration files and maybe extensions into options parsing, but there is a need sometimes to graphically change options . Maybe an amalgamation of the above ideas with the traits package of scipy http://old.scipy.org/site_content/traits would allow options to be set in all three places - command line, GUI and options file.

My implementation (http://cfgparse.sourceforge.net/) could be useful for implementing a GUI option editor by using the round trip capability. I'm sure some of the other parsers would as well. A GUI editor seems so application specific that I don't think we would want a GUI editor built into the standard library. -- dan.gass@gmail.com

Status of Shootout

So where are we? The wiki hasn't seen much activity since January of 2005. Are any of these on track for 2.5? As strictly a user, most of these seem over-engineered for the standard library (to me anyway). cfgparse (dan gass) seemed the most approachable to me. XML, or some new configuration 'language' in the file is way too much. Keeping to a similar (or even matching) API to optparse is important to me. But where are we? -- HunterMatthews (showing up as 82 in the edit log)

GuidoVanRossum has discussed possible changes to ConfigParser in a thread on python-dev. He doesn't see a great deal of need to improve on ConfigParser. He dislikes *most* of the suggestions on this page because he doesn't want to see nested sections in configuration files.

His reason for this is that he wants to discourage developers from mixing configuration options with application data more suited to a data persistence format. For this he recommends an XML type solution.

I personally think that complex configuration use cases are reasonably common, and that nested sections is a reasonable requirement. Editing/reading XML is not fun, whether it's for configuration *or* data persistence. For this reason ConfigObj is listed as an XML Alternative.

That aside, it seems like he wouldn't be averse to improvements on ConfigParser in the following ways :

-- MichaelFoord

I think we have to distinguish between configurations primarily intended for editing by users and configurations primarily intended for programmatic editing. For example, XML makes a lot of sense as an interchangeable language for systems configurations (e.g., SuSE Linux AutoYaST). But XML is a horrid format for manual editing. The delimiter cruft makes it hard to comprehend for all but the simplest cases.

I also do not understand the allergy to nesting, conditionals, and hierarchical namespaces. I implemented all three of these in tconfpy precisely because they make the resultant configuration file simpler - both to understand by the human reader and to subsequently maintain.

I realize there is always the danger of backing into a configuration language that becomes Turing-complete in its own right, but there is a balance to be struck between utility and simplicity. (One tconfpy tester kiddingly said I was on my way to a reimplementation of m4 <shudder> ;) My general view on this tradeoff is that configuration "languages" should be more-or-less entirely about lexical replacement and not carry a lot of deep semantics around. There is, of course, some semantic content when you introduce nesting, conditionals, and hierarchical namespaces, but even these really serve just a lexical purpose.

ConfigParser is fine for simple day-to-day configurations but I still think there is room for a more full-featured configuration "language". One does not preclude the other.

-- Tim Daneliuk


With an eye to "simple configuration scripts and storing data in a user-friendly way", Babar K. Zafar worked out his idea of a safe evaluator.

ConfigParserShootout (last edited 2014-04-02 18:08:02 by SkipMontanaro)

Unable to edit the page? See the FrontPage for instructions.