Comment Example =============== .. contents:: Introduction ------------ This is an example of how to write WSGI middleware with WebOb. The specific example adds a simple comment form to HTML web pages; any page served through the middleware that is HTML gets a comment form added to it, and shows any existing comments. Code ---- The finished code for this is available in `docs/comment-example-code/example.py `_ -- you can run that file as a script to try it out. Instantiating Middleware ------------------------ Middleware of any complexity at all is usually best created as a class with its configuration as arguments to that class. Every middleware needs an application (``app``) that it wraps. This middleware also needs a location to store the comments; we'll put them all in a single directory. .. code-block:: python import os class Commenter(object): def __init__(self, app, storage_dir): self.app = app self.storage_dir = storage_dir if not os.path.exists(storage_dir): os.makedirs(storage_dir) When you use this middleware, you'll use it like: .. code-block:: python app = ... make the application ... app = Commenter(app, storage_dir='./comments') For our application we'll use a simple static file server that is included with `Paste `_ (use ``easy_install Paste`` to install this). The setup is all at the bottom of ``example.py``, and looks like this: .. code-block:: python if __name__ == '__main__': import optparse parser = optparse.OptionParser( usage='%prog --port=PORT BASE_DIRECTORY' ) parser.add_option( '-p', '--port', default='8080', dest='port', type='int', help='Port to serve on (default 8080)') parser.add_option( '--comment-data', default='./comments', dest='comment_data', help='Place to put comment data into (default ./comments/)') options, args = parser.parse_args() if not args: parser.error('You must give a BASE_DIRECTORY') base_dir = args[0] from paste.urlparser import StaticURLParser app = StaticURLParser(base_dir) app = Commenter(app, options.comment_data) from wsgiref.simple_server import make_server httpd = make_server('localhost', options.port, app) print 'Serving on http://localhost:%s' % options.port try: httpd.serve_forever() except KeyboardInterrupt: print '^C' I won't explain it here, but basically it takes some options, creates an application that serves static files (``StaticURLParser(base_dir)``), wraps it with ``Commenter(app, options.comment_data)`` then serves that. The Middleware -------------- While we've created the class structure for the middleware, it doesn't actually do anything. Here's a kind of minimal version of the middleware (using WebOb): .. code-block:: python from webob import Request class Commenter(object): def __init__(self, app, storage_dir): self.app = app self.storage_dir = storage_dir if not os.path.exists(storage_dir): os.makedirs(storage_dir) def __call__(self, environ, start_response): req = Request(environ) resp = req.get_response(self.app) return resp(environ, start_response) This doesn't modify the response it any way. You could write it like this without WebOb: .. code-block:: python class Commenter(object): ... def __call__(self, environ, start_response): return self.app(environ, start_response) But it won't be as convenient later. First, lets create a little bit of infrastructure for our middleware. We need to save and load per-url data (the comments themselves). We'll keep them in pickles, where each url has a pickle named after the url (but double-quoted, so ``http://localhost:8080/index.html`` becomes ``http%3A%2F%2Flocalhost%3A8080%2Findex.html``). .. code-block:: python from cPickle import load, dump class Commenter(object): ... def get_data(self, url): filename = self.url_filename(url) if not os.path.exists(filename): return [] else: f = open(filename, 'rb') data = load(f) f.close() return data def save_data(self, url, data): filename = self.url_filename(url) f = open(filename, 'wb') dump(data, f) f.close() def url_filename(self, url): # Double-quoting makes the filename safe return os.path.join(self.storage_dir, urllib.quote(url, '')) You can get the full request URL with ``req.url``, so to get the comment data with these methods you do ``data = self.get_data(req.url)``. Now we'll update the ``__call__`` method to filter *some* responses, and get the comment data for those. We don't want to change responses that were error responses (anything but ``200``), nor do we want to filter responses that aren't HTML. So we get: .. code-block:: python class Commenter(object): ... def __call__(self, environ, start_response): req = Request(environ) resp = req.get_response(self.app) if resp.content_type != 'text/html' or resp.status_code != 200: return resp(environ, start_response) data = self.get_data(req.url) ... do stuff with data, update resp ... return resp(environ, start_response) So far we're punting on actually adding the comments to the page. We also haven't defined what ``data`` will hold. Let's say it's a list of dictionaries, where each dictionary looks like ``{'name': 'John Doe', 'homepage': 'http://blog.johndoe.com', 'comments': 'Great site!'}``. We'll also need a simple method to add stuff to the page. We'll use a regular expression to find the end of the page and put text in: .. code-block:: python import re class Commenter(object): ... _end_body_re = re.compile(r'', re.I|re.S) def add_to_end(self, html, extra_html): """ Adds extra_html to the end of the html page (before ) """ match = self._end_body_re.search(html) if not match: return html + extra_html else: return html[:match.start()] + extra_html + html[match.start():] And then we'll use it like: .. code-block:: python data = self.get_data(req.url) body = resp.body body = self.add_to_end(body, self.format_comments(data)) resp.body = body return resp(environ, start_response) We get the body, update it, and put it back in the response. This also updates ``Content-Length``. Then we define: .. code-block:: python from webob import html_escape class Commenter(object): ... def format_comments(self, comments): if not comments: return '' text = [] text.append('
') text.append('

Comments (%s):

' % len(comments)) for comment in comments: text.append('

%s at %s:

' % ( html_escape(comment['homepage']), html_escape(comment['name']), time.strftime('%c', comment['time']))) # Susceptible to XSS attacks!: text.append(comment['comments']) return ''.join(text) We put in a header (with an anchor we'll use later), and a section for each comment. Note that ``html_escape`` is the same as ``cgi.escape`` and just turns ``&`` into ``&``, etc. Because we put in some text without quoting it is susceptible to a `Cross-Site Scripting `_ attack. Fixing that is beyond the scope of this tutorial; you could quote it or clean it with something like `lxml.html.clean `_. Accepting Comments ------------------ All of those pieces *display* comments, but still no one can actually make comments. To handle this we'll take a little piece of the URL space for our own, everything under ``/.comments``, so when someone POSTs there it will add a comment. When the request comes in there are two parts to the path: ``SCRIPT_NAME`` and ``PATH_INFO``. Everything in ``SCRIPT_NAME`` has already been parsed, and everything in ``PATH_INFO`` has yet to be parsed. That means that the URL *without* ``PATH_INFO`` is the path to the middleware; we can intercept anything else below ``SCRIPT_NAME`` but nothing above it. The name for the URL without ``PATH_INFO`` is ``req.application_url``. We have to capture it early to make sure it doesn't change (since the WSGI application we are wrapping may update ``SCRIPT_NAME`` and ``PATH_INFO``). So here's what this all looks like: .. code-block:: python class Commenter(object): ... def __call__(self, environ, start_response): req = Request(environ) if req.path_info_peek() == '.comments': return self.process_comment(req)(environ, start_response) # This is the base path of *this* middleware: base_url = req.application_url resp = req.get_response(self.app) if resp.content_type != 'text/html' or resp.status_code != 200: # Not an HTML response, we don't want to # do anything to it return resp(environ, start_response) # Make sure the content isn't gzipped: resp.decode_content() comments = self.get_data(req.url) body = resp.body body = self.add_to_end(body, self.format_comments(comments)) body = self.add_to_end(body, self.submit_form(base_url, req)) resp.body = body return resp(environ, start_response) ``base_url`` is the path where the middleware is located (if you run the example server, it will be ``http://localhost:PORT/``). We use ``req.path_info_peek()`` to look at the next segment of the URL -- what comes after base_url. If it is ``.comments`` then we handle it internally and don't pass the request on. We also put in a little guard, ``resp.decode_content()`` in case the application returns a gzipped response. Then we get the data, add the comments, add the *form* to make new comments, and return the result. submit_form ~~~~~~~~~~~ Here's what the form looks like: .. code-block:: python class Commenter(object): ... def submit_form(self, base_path, req): return '''

Leave a comment:

Name:
URL:
Comments:

''' % (base_path, html_escape(req.url)) Nothing too exciting. It submits a form with the keys ``url`` (the URL being commented on), ``name``, ``homepage``, and ``comments``. process_comment ~~~~~~~~~~~~~~~ If you look at the method call, what we do is call the method then treat the result as a WSGI application: .. code-block:: python return self.process_comment(req)(environ, start_response) You could write this as: .. code-block:: python response = self.process_comment(req) return response(environ, start_response) A common pattern in WSGI middleware that *doesn't* use WebOb is to just do: .. code-block:: python return self.process_comment(environ, start_response) But the WebOb style makes it easier to modify the response if you want to; modifying a traditional WSGI response/application output requires changing your logic flow considerably. Here's the actual processing code: .. code-block:: python from webob import exc from webob import Response class Commenter(object): ... def process_comment(self, req): try: url = req.params['url'] name = req.params['name'] homepage = req.params['homepage'] comments = req.params['comments'] except KeyError, e: resp = exc.HTTPBadRequest('Missing parameter: %s' % e) return resp data = self.get_data(url) data.append(dict( name=name, homepage=homepage, comments=comments, time=time.gmtime())) self.save_data(url, data) resp = exc.HTTPSeeOther(location=url+'#comment-area') return resp We either give a Bad Request response (if the form submission is somehow malformed), or a redirect back to the original page. The classes in ``webob.exc`` (like ``HTTPBadRequest`` and ``HTTPSeeOther``) are Response subclasses that can be used to quickly create responses for these non-200 cases where the response body usually doesn't matter much. Conclusion ---------- This shows how to make response modifying middleware, which is probably the most difficult kind of middleware to write with WSGI -- modifying the request is quite simple in comparison, as you simply update ``environ``.