I have just released ConfigTree. It is the longest project of mine. It took more than two and a half years from the first commit to the release. But the history of the project is much longer.

The idea came from “My Health Experience” project. It was a great project I worked on, unfortunately it is closed now. My team started from a small forum and ended up with a full featured social network. We got a single server at the start and a couple of clusters at the end. A handful of configuration files grew up to a directory with dozens of ones, which described all subsystems in all possible environments. Each module of the project had dozens of calls to the configuration registry. And we developed a special tool to manage the settings.

This is how it worked. An environment name was a dot-separated string in format group.subgroup.environment. For instance, prod.cluster-1.server-1 was an environment name of the first server from the first cluster of the production environment; and dev.kr41 was the name of my development environment. The configuration directory contained a tree of subdirectories, where each of the subdirectory was named after a part of some environment name. For example:

config/
    prod/
        cluster-1/
            server-1/
    dev/
        kr41/

The most common configuration options were defined at the root of the tree, the most specific ones—at the leafs. For example, config/prod directory contained files with common production settings; config/prod/cluster-1—common settings for all servers of the first cluster; and config/prod/cluster-1/server-1—concrete settings for the first server. The files were merged by a loader on startup into a single mapping object using passed environment name. Some of the common settings were overridden by the concrete ones during the loading process. So that we did not use copy-paste in our configuration files. If there was an option for a number of environments, this option had been defined within group settings. There we also post-loading validation, that helped us to use safe defaults. For instance, when each server had to use its own cryptographic key, such key had been defined on the group level with an empty default value, which was required to be overridden. So that validator raised an exception on startup, when it had found this empty value in the result configuration. Because of this we never deployed our application on production with unsafe settings.

The tool was so useful, so when I started to use Python I had tried to find something similar. Yep, “My Health Experience” had been written on PHP, and it was the last PHP project I worked on. My search was unsuccessful, and I reinvented such tool working on each my project. So I eventually decided to rewrite and release it as an open-source project. And here it is.

I added some flexibility and extensibility to the original ideas. Each step of configuration loading process can be customized or replaced by your own implementation. It also comes with command line utility program, which can be used to build configuration as a single JSON file. So you can even use it within a non-Python project—JSON parser is all what you need. I hope, the tool is able to solve a lot of problems and can be useful for different kind of projects. Try it out and send me your feedback. As for me, I am going to integrate it into my current project right now.