PasteDeploy is a great tool for managing WSGI applications. Unfortunately, there is no support of configuration formats other than INI-files. Montague is going to solve the problem, but its documentation is unfinished and says nothing useful. Hope, it will be changed soon. But if you don’t want to wait, as me do, the following recipe is for you.

Using ConfigTree on my current project, I stumbled with the problem: how to serve Pyramid applications (I got three ones) from the custom configuration? Here is how it looks like in YAML:

app:
    use: "egg:MyApp#main"
    # Application local settings goes here
filters:
    -
        use: "egg:MyFilter#filter1"
        # Filter local settings goes here
    -
        use: "egg:MyFilter#filter2"
server:
    use: "egg:MyServer#main"
    # Server local settings goes here

The easy way is to build INI-file and use it. The hard way is to make my own loader. I chose the hard one.

PasteDeploy provides public functions loadapp, loadfilter, and loadserver. However, these functions don’t work, because they don’t accept local settings. Only global configuration can be passed into.

app = loadapp('egg:MyApp#main', global_conf=config)

But the most of PasteDeploy-based applications simply ignore global_conf. For example, here is the paste factory of Waitress:

def serve_paste(app, global_conf, **kw):
    serve(app, **kw)        # global_conf? Who needs this shit?
    return 0

I dug around the sources of PasteDeploy and found loadcontext function. It is kind of low level private function. But who cares? So here is the source of loader, that uses the function.

from paste.deploy.loadwsgi import loadcontext, APP, FILTER, SERVER


def run(config):

    def load_object(object_type, conf):
        conf = conf.copy()
        spec = conf.pop('use')
        context = loadcontext(object_type, spec)    # Loading object
        context.local_conf = conf                   # Passing local settings
        return context.create()

    app = load_object(APP, config['app'])
    if 'filters' in config:
        for filter_conf in config['filters']:
            filter_app = load_object(FILTER, filter_conf)
            app = filter_app(app)
    server = load_object(SERVER, config['server'])
    server(app)

But it is not the end. Pyramid comes with its own command pserve, that uses PasteDeploy to load and start up application from INI-file. And there is an option of the command that makes development fun. I mean --reload one. It starts separate process with a file monitor that restarts your application when its sources are changed. The following code provides the feature. It depends on Pyramid, because I don’t want to reinvent the wheel. But if you use another framework, it won’t be hard to write your own file monitor.

import sys
import os
import signal
from subprocess import Popen

from paste.deploy.loadwsgi import loadcontext, APP, FILTER, SERVER
from pyramid.scripts.pserve import install_reloader, kill


def run(config, with_reloader=False):

    def load_object(object_type, conf):
        conf = conf.copy()
        spec = conf.pop('use')
        context = loadcontext(object_type, spec)
        context.local_conf = conf
        return context.create()

    def run_server():
        app = load_object(APP, config['app'])
        if 'filters' in config:
            for filter_conf in config['filters']:
                filter_app = load_object(FILTER, filter_conf)
                app = filter_app(app)
        server = load_object(SERVER, config['server'])
        server(app)

    if not with_reloader:
        run_server()
    elif os.environ.get('master_process_is_running'):
        # Pass your configuration files here using ``extra_files`` argument
        install_reloader(extra_files=None)
        run_server()
    else:
        print("Starting subprocess with file monitor")
        environ = os.environ.copy()
        environ['master_process_is_running'] = 'true'
        childproc = None
        try:
            while True:
                try:
                    childproc = Popen(sys.argv, env=environ)
                    exitcode = childproc.wait()
                    childproc = None
                    if exitcode != 3:
                        return exitcode
                finally:
                    if childproc is not None:
                        try:
                            kill(childproc.pid, signal.SIGTERM)
                        except (OSError, IOError):
                            pass
        except KeyboardInterrupt:
            pass

That’s it. Wrap the code with a console script and don’t forget to initialize the logging.

Traversal is awesome thing, I believe that it is real killer feature of Pyramid web framework. However people usually don’t get it. They think it is too complicated. So I’m going to convince you in the opposite.

I assume, you know that Pyramid supports two URL handling methods: URL Dispatch and Traversal. And you familiar with the technical details of how them work (follow the links above if you don’t). So here I’m considering the benefits of Traversal, instead of how it actually works.

Pyramid is super-flexible framework, where you can do thing in the way you want to. Traversal is not an exception. To start working with Traversal, you just need to provide a root_factory callable, which accepts single argument request and returns a root resource of your web application. The root can be arbitrary object. However, to feel all power of traversal the root_factory should return a resource tree—a hierarchy of objects, where each one provides the following features:

• it knows its name, i.e. has __name__ attribute;

• it knows its parent, i.e. has __parent__ attribute;

• it knows its children, i.e. implements __getitem__ method in the following way:

  >>> root = root_factory(request)
  >>> child = root['child_resource']
  >>> child.__name__
  'child_resource'
  >>> child.__parent__ is root
  True
  

So that to build URL structure of your web site, you should build a resource tree—a bunch of classes, in fact. And that is exactly what usually confuses people. Is it overengineering? Why so complicated? Indeed, writing a dozen of routes will take exactly a dozen lines of code. Whereas writing a couple of classes will take much more ones.

However, the answer is “No”, it’s not overengineering. Traversal use resource tree for handling URLs, but the resource tree itself is not only used to represent URL structure. It is a perfect additional abstraction level which can encapsulate business logic. In that way the old holy war about fat models and skinny controllers (views in Pyramid terms) can be solved.

Resource also provides a unified interface between models and views. From one hand, you can build your models using different data sources: RDBMS, NoSQL, RPC, REST, and other terrifying abbreviations. And resources will make them work together. From other hand, you can use these resources in different interfaces: web (which Pyramid actually provides), RPC, CLI, even tests. Because test is just another interface of your application. And yes, using Traversal will make testing much more easier.

But what about URL structure? Using traversal is hard to start, you should build a resource tree. However these efforts will be rewarded in future. Because supporting traversal-based application is a walk in the park. For example, you have code that implements blog:

class Blog(Resource):

    def __getitem__(self, name):
        return BlogPost(name, parent=self)


class BlogPost(Resource):
    ...


@view_config(context=Blog)
def index(context, request):
    ...

@view_config(context=Blog, view_name='archive')
def archive(context, request):
    ...

@view_config(context=BlogPost)
def show(context, request):
    ...

Now, you can bind Blog resource to other ones, to add blogs into different places of your site. And it can be done with a couple of lines of code:

class User(Resource):

    def __getitem__(self, name):
        if name == 'blog':
            # Now each user has her own blog
            return Blog(name, parent=self)
        elif ...

From this point of view, resource with associated views can be considered as a reusable component, just like application in Django. You can also use mixin classes to create plugins:

class Commetable(object):
    """ Implements comment list """

class Likeable(object):
    """ Implements like/unlike buttons behavior """

class BlogPost(Resource, Commentable, Likeable):
    """ Blog post that can be commented and liked/unliked """

You can even make the trick, which I described in Obscene Python, i.e. constructing your resource classes on the fly using different set of mixins for each one.

And the last, but not least, Traversal is a right way for handling URLs, because it works with hierarchical structure which reflects URL. Whereas URL Dispatch uses flat list of regular expressions. So that, task like rendering breadcrumb navigation is trivial for traversal-based application, but it is hard as hell using URL Dispatch (in fact, it cannot be done without dirty hacks).

So if you are going to use Traversal, try also TraversalKit. This library is essential of my own experience of Traversal usage. I hope it will be useful for you too.

P.S. The article has been written in transfer zone of Moscow airport Domodedovo on my way from PyCon Finland 2014, Helsinki to Omsk.