WebOb/docs/jsonrpc-example.txt

JSON-RPC Example
================

.. contents::

:author: Ian Bicking

Introduction
------------

This is an example of how to write a web service using WebOb.  The
example shows how to create a `JSON-RPC <http://json-rpc.org/>`_
endpoint using WebOb and the `simplejson
<http://www.undefined.org/python/#simplejson>`_ JSON library.  This
also shows how to use WebOb as a client library using `WSGIProxy
<http://pythonpaste.org/wsgiproxy/>`_.

While this example presents JSON-RPC, this is not an endorsement of
JSON-RPC.  In fact I don't like JSON-RPC.  It's unnecessarily
un-RESTful, and modelled too closely on `XML-RPC
<http://www.xmlrpc.com/>`_.

Code
----

The finished code for this is available in
`docs/json-example-code/jsonrpc.py
<https://github.com/Pylons/webob/tree/master/docs/jsonrpc-example-code/jsonrpc.py>`_
-- you can run that file as a script to try it out, or import it.

Concepts
--------

JSON-RPC wraps an object, allowing you to call methods on that object
and get the return values.  It also provides a way to get error
responses.

The `specification
<http://json-rpc.org/wd/JSON-RPC-1-1-WD-20060807.html>`_ goes into the
details (though in a vague sort of way).  Here's the basics:

* All access goes through a POST to a single URL.

* The POST contains a JSON body that looks like::

   {"method": "methodName",
    "id": "arbitrary-something",
    "params": [arg1, arg2, ...]}

* The ``id`` parameter is just a convenience for the client to keep
  track of which response goes with which request.  This makes
  asynchronous calls (like an XMLHttpRequest) easier.  We just send
  the exact same id back as we get, we never look at it.

* The response is JSON.  A successful response looks like::

    {"result": the_result,
     "error": null,
     "id": "arbitrary-something"}

* The error response looks like::

    {"result": null,
     "error": {"name": "JSONRPCError",
               "code": (number 100-999),
               "message": "Some Error Occurred",
               "error": "whatever you want\n(a traceback?)"},
     "id": "arbitrary-something"}

* It doesn't seem to indicate if an error response should have a 200
  response or a 500 response.  So as not to be completely stupid about
  HTTP, we choose a 500 resonse, as giving an error with a 200
  response is irresponsible.

Infrastructure
--------------

To make this easier to test, we'll set up a bit of infrastructure.
This will open up a server (using `wsgiref
<http://python.org/doc/current/lib/module-wsgiref.simpleserver.html>`_)
and serve up our application (note that *creating* the application is
left out to start with):

.. code-block:: python

    import sys

    def main(args=None):
        import optparse
        from wsgiref import simple_server
        parser = optparse.OptionParser(
            usage="%prog [OPTIONS] MODULE:EXPRESSION")
        parser.add_option(
            '-p', '--port', default='8080',
            help='Port to serve on (default 8080)')
        parser.add_option(
            '-H', '--host', default='127.0.0.1',
            help='Host to serve on (default localhost; 0.0.0.0 to make public)')
        if args is None:
            args = sys.argv[1:]
        options, args = parser.parse_args()
        if not args or len(args) > 1:
            print 'You must give a single object reference'
            parser.print_help()
            sys.exit(2)
        app = make_app(args[0])
        server = simple_server.make_server(
            options.host, int(options.port),
            app)
        print 'Serving on http://%s:%s' % (options.host, options.port)
        server.serve_forever()

    if __name__ == '__main__':
        main()

I won't describe this much.  It starts a server, serving up just the
app created by ``make_app(args[0])``.  ``make_app`` will have to load
up the object and wrap it in our WSGI/WebOb wrapper.  We'll be calling
that wrapper ``JSONRPC(obj)``, so here's how it'll go:

.. code-block:: python

    def make_app(expr):
        module, expression = expr.split(':', 1)
        __import__(module)
        module = sys.modules[module]
        obj = eval(expression, module.__dict__)
        return JsonRpcApp(obj)

We use ``__import__(module)`` to import the module, but its return
value is wonky.  We can find the thing it imported in ``sys.modules``
(a dictionary of all the loaded modules).  Then we evaluate the second
part of the expression in the namespace of the module.  This lets you
do something like ``smtplib:SMTP('localhost')`` to get a fully
instantiated SMTP object.

That's all the infrastructure we'll need for the server side.  Now we
just have to implement ``JsonRpcApp``.

The Application Wrapper
-----------------------

Note that I'm calling this an "application" because that's the
terminology WSGI uses.  Everything that gets *called* is an
"application", and anything that calls an application is called a
"server".

The instantiation of the server is already figured out:

.. code-block:: python

    class JsonRpcApp(object):

        def __init__(self, obj):
            self.obj = obj

        def __call__(self, environ, start_response):
            ... the WSGI interface ...

So the server is an instance bound to the particular object being
exposed, and ``__call__`` implements the WSGI interface.

We'll start with a simple outline of the WSGI interface, using a kind
of standard WebOb setup:

.. code-block:: python

    from webob import Request, Response
    from webob import exc

    class JsonRpcApp(object):
        ...
        def __call__(self, environ, start_response):
            req = Request(environ)
            try:
                resp = self.process(req)
            except ValueError, e:
                resp = exc.HTTPBadRequest(str(e))
            except exc.HTTPException, e:
                resp = e
            return resp(environ, start_response)

We first create a request object.  The request object just wraps the
WSGI environment.  Then we create the response object in the
``process`` method (which we still have to write).  We also do some
exception catching.  We'll turn any ``ValueError`` into a ``400 Bad
Request`` response.  We'll also let ``process`` raise any
``web.exc.HTTPException`` exception.  There's an exception defined in
that module for all the HTTP error responses, like ``405 Method Not
Allowed``.  These exceptions are themselves WSGI applications (as is
``webob.Response``), and so we call them like WSGI applications and
return the result.

The ``process`` method
----------------------

The ``process`` method of course is where all the fancy stuff
happens.  We'll start with just the most minimal implementation, with
no error checking or handling:

.. code-block:: python

    from simplejson import loads, dumps

    class JsonRpcApp(object):
        ...
        def process(self, req):
            json = loads(req.body)
            method = json['method']
            params = json['params']
            id = json['id']
            method = getattr(self.obj, method)
            result = method(*params)
            resp = Response(
                content_type='application/json',
                body=dumps(dict(result=result,
                                error=None,
                                id=id)))
            return resp

As long as the request is properly formed and the method doesn't raise
any exceptions, you are pretty much set.  But of course that's not a
reasonable expectation.  There's a whole bunch of things that can go
wrong.  For instance, it has to be a POST method:

.. code-block:: python

    if not req.method == 'POST':
        raise exc.HTTPMethodNotAllowed(
            "Only POST allowed",
            allowed='POST')

And maybe the request body doesn't contain valid JSON:

.. code-block:: python

    try:
        json = loads(req.body)
    except ValueError, e:
        raise ValueError('Bad JSON: %s' % e)

And maybe all the keys aren't in the dictionary:

.. code-block:: python

    try:
        method = json['method']
        params = json['params']
        id = json['id']
    except KeyError, e:
        raise ValueError(
            "JSON body missing parameter: %s" % e)

And maybe it's trying to acces a private method (a method that starts
with ``_``) -- that's not just a bad request, we'll call that case
``403 Forbidden``.

.. code-block:: python

    if method.startswith('_'):
        raise exc.HTTPForbidden(
            "Bad method name %s: must not start with _" % method)

And maybe ``json['params']`` isn't a list:

.. code-block:: python

    if not isinstance(params, list):
        raise ValueError(
            "Bad params %r: must be a list" % params)

And maybe the method doesn't exist:

.. code-block:: python

    try:
        method = getattr(self.obj, method)
    except AttributeError:
        raise ValueError(
            "No such method %s" % method)

The last case is the error we actually can expect: that the method
raises some exception.

.. code-block:: python

    try:
        result = method(*params)
    except:
        tb = traceback.format_exc()
        exc_value = sys.exc_info()[1]
        error_value = dict(
            name='JSONRPCError',
            code=100,
            message=str(exc_value),
            error=tb)
        return Response(
            status=500,
            content_type='application/json',
            body=dumps(dict(result=None,
                            error=error_value,
                            id=id)))

That's a complete server.

The Complete Code
-----------------

Since we showed all the error handling in pieces, here's the complete
code:

.. code-block:: python

    from webob import Request, Response
    from webob import exc
    from simplejson import loads, dumps
    import traceback
    import sys

    class JsonRpcApp(object):
        """
        Serve the given object via json-rpc (http://json-rpc.org/)
        """

        def __init__(self, obj):
            self.obj = obj

        def __call__(self, environ, start_response):
            req = Request(environ)
            try:
                resp = self.process(req)
            except ValueError, e:
                resp = exc.HTTPBadRequest(str(e))
            except exc.HTTPException, e:
                resp = e
            return resp(environ, start_response)

        def process(self, req):
            if not req.method == 'POST':
                raise exc.HTTPMethodNotAllowed(
                    "Only POST allowed",
                    allowed='POST')
            try:
                json = loads(req.body)
            except ValueError, e:
                raise ValueError('Bad JSON: %s' % e)
            try:
                method = json['method']
                params = json['params']
                id = json['id']
            except KeyError, e:
                raise ValueError(
                    "JSON body missing parameter: %s" % e)
            if method.startswith('_'):
                raise exc.HTTPForbidden(
                    "Bad method name %s: must not start with _" % method)
            if not isinstance(params, list):
                raise ValueError(
                    "Bad params %r: must be a list" % params)
            try:
                method = getattr(self.obj, method)
            except AttributeError:
                raise ValueError(
                    "No such method %s" % method)
            try:
                result = method(*params)
            except:
                text = traceback.format_exc()
                exc_value = sys.exc_info()[1]
                error_value = dict(
                    name='JSONRPCError',
                    code=100,
                    message=str(exc_value),
                    error=text)
                return Response(
                    status=500,
                    content_type='application/json',
                    body=dumps(dict(result=None,
                                    error=error_value,
                                    id=id)))
            return Response(
                content_type='application/json',
                body=dumps(dict(result=result,
                                error=None,
                                id=id)))

The Client
----------

It would be nice to have a client to test out our server.  Using
`WSGIProxy`_ we can use WebOb
Request and Response to do actual HTTP connections.

The basic idea is that you can create a blank Request:

.. code-block:: python

    >>> from webob import Request
    >>> req = Request.blank('http://python.org')

Then you can send that request to an application:

.. code-block:: python

    >>> from wsgiproxy.exactproxy import proxy_exact_request
    >>> resp = req.get_response(proxy_exact_request)

This particular application (``proxy_exact_request``) sends the
request over HTTP:

.. code-block:: python

    >>> resp.content_type
    'text/html'
    >>> resp.body[:10]
    '<!DOCTYPE '

So we're going to create a proxy object that constructs WebOb-based
jsonrpc requests, and sends those using ``proxy_exact_request``.

The Proxy Client
----------------

The proxy client is instantiated with its base URL.  We'll also let
you pass in a proxy application, in case you want to do local requests
(e.g., to do direct tests against a ``JsonRpcApp`` instance):

.. code-block:: python

    class ServerProxy(object):

        def __init__(self, url, proxy=None):
            self._url = url
            if proxy is None:
                from wsgiproxy.exactproxy import proxy_exact_request
                proxy = proxy_exact_request
            self.proxy = proxy

This ServerProxy object itself doesn't do much, but you can call methods on
it.  We can intercept any access ``ServerProxy(...).method`` with the
magic function ``__getattr__``.  Whenever you get an attribute that
doesn't exist in an instance, Python will call
``inst.__getattr__(attr_name)`` and return that.  When you *call* a
method, you are calling the object that ``.method`` returns.  So we'll
create a helper object that is callable, and our ``__getattr__`` will
just return that:

.. code-block:: python

    class ServerProxy(object):
        ...
        def __getattr__(self, name):
            # Note, even attributes like __contains__ can get routed
            # through __getattr__
            if name.startswith('_'):
                raise AttributeError(name)
            return _Method(self, name)

    class _Method(object):
        def __init__(self, parent, name):
            self.parent = parent
            self.name = name

Now when we call the method we'll be calling ``_Method.__call__``, and
the HTTP endpoint will be ``self.parent._url``, and the method name
will be ``self.name``.

Here's the code to do the call:

.. code-block:: python

    class _Method(object):
        ...

        def __call__(self, *args):
            json = dict(method=self.name,
                        id=None,
                        params=list(args))
            req = Request.blank(self.parent._url)
            req.method = 'POST'
            req.content_type = 'application/json'
            req.body = dumps(json)
            resp = req.get_response(self.parent.proxy)
            if resp.status_code != 200 and not (
                resp.status_code == 500
                and resp.content_type == 'application/json'):
                raise ProxyError(
                    "Error from JSON-RPC client %s: %s"
                    % (self._url, resp.status),
                    resp)
            json = loads(resp.body)
            if json.get('error') is not None:
                e = Fault(
                    json['error'].get('message'),
                    json['error'].get('code'),
                    json['error'].get('error'),
                    resp)
                raise e
            return json['result']

We raise two kinds of exceptions here.  ``ProxyError`` is when
something unexpected happens, like a ``404 Not Found``.  ``Fault`` is
when a more expected exception occurs, i.e., the underlying method
raised an exception.

In both cases we'll keep the response object around, as that can be
interesting.  Note that you can make exceptions have any methods or
signature you want, which we'll do:

.. code-block:: python

    class ProxyError(Exception):
        """
        Raised when a request via ServerProxy breaks
        """
        def __init__(self, message, response):
            Exception.__init__(self, message)
            self.response = response

    class Fault(Exception):
        """
        Raised when there is a remote error
        """
        def __init__(self, message, code, error, response):
            Exception.__init__(self, message)
            self.code = code
            self.error = error
            self.response = response
        def __str__(self):
            return 'Method error calling %s: %s\n%s' % (
                self.response.request.url,
                self.args[0],
                self.error)

Using Them Together
-------------------

Good programmers start with tests.  But at least we'll end with a
test.  We'll use `doctest
<http://python.org/doc/current/lib/module-doctest.html>`_ for our
tests.  The test is in `docs/json-example-code/test_jsonrpc.txt
<https://github.com/Pylons/webob/tree/master/docs/jsonrpc-example-code/test_jsonrpc.txt>`_
and you can run it with `docs/json-example-code/test_jsonrpc.py
<https://github.com/Pylons/webob/tree/master/docs/jsonrpc-example-code/test_jsonrpc.py>`_,
which looks like:

.. code-block:: python

    if __name__ == '__main__':
        import doctest
        doctest.testfile('test_jsonrpc.txt')

As you can see, it's just a stub to run the doctest.  We'll need a
simple object to expose.  We'll make it real simple:

.. code-block:: python

    >>> class Divider(object):
    ...     def divide(self, a, b):
    ...        return a / b

Then we'll get the app setup:

.. code-block:: python

    >>> from jsonrpc import *
    >>> app = JsonRpcApp(Divider())

And attach the client *directly* to it:

.. code-block:: python

    >>> proxy = ServerProxy('http://localhost:8080', proxy=app)

Because we gave the app itself as the proxy, the URL doesn't actually
matter.

Now, if you are used to testing you might ask: is this kosher?  That
is, we are shortcircuiting HTTP entirely.  Is this a realistic test?

One thing you might be worried about in this case is that there are
more shared objects than you'd have with HTTP.  That is, everything
over HTTP is serialized to headers and bodies.  Without HTTP, we can
send stuff around that can't go over HTTP.  This *could* happen, but
we're mostly protected because the only thing the application's share
is the WSGI ``environ``.  Even though we use a ``webob.Request``
object on both side, it's not the *same* request object, and all the
state is studiously kept in the environment.  We *could* share things
in the environment that couldn't go over HTTP.  For instance, we could
set ``environ['jsonrpc.request_value'] = dict(...)``, and avoid
``simplejson.dumps`` and ``simplejson.loads``.  We *could* do that,
and if we did then it is possible our test would work even though the
libraries were broken over HTTP.  But of course inspection shows we
*don't* do that.  A little discipline is required to resist playing clever
tricks (or else you can play those tricks and do more testing).
Generally it works well.

So, now we have a proxy, lets use it:

.. code-block:: python

    >>> proxy.divide(10, 4)
    2
    >>> proxy.divide(10, 4.0)
    2.5

Lastly, we'll test a couple error conditions.  First a method error:

.. code-block:: python

    >>> proxy.divide(10, 0) # doctest: +ELLIPSIS
    Traceback (most recent call last):
       ...
    Fault: Method error calling http://localhost:8080: integer division or modulo by zero
    Traceback (most recent call last):
      File ...
        result = method(*params)
      File ...
        return a / b
    ZeroDivisionError: integer division or modulo by zero
    <BLANKLINE>

It's hard to actually predict this exception, because the test of the
exception itself contains the traceback from the underlying call, with
filenames and line numbers that aren't stable.  We use ``# doctest:
+ELLIPSIS`` so that we can replace text we don't care about with
``...``.  This is actually figured out through copy-and-paste, and
visual inspection to make sure it looks sensible.

The other exception can be:

.. code-block:: python

    >>> proxy.add(1, 1)
    Traceback (most recent call last):
        ...
    ProxyError: Error from JSON-RPC client http://localhost:8080: 400 Bad Request

Here the exception isn't a JSON-RPC method exception, but a more basic
ProxyError exception.

Conclusion
----------

Hopefully this will give you ideas about how to implement web services
of different kinds using WebOb.  I hope you also can appreciate the
elegance of the symmetry of the request and response objects, and the
client and server for the protocol.

Many of these techniques would be better used with a `RESTful
<http://en.wikipedia.org/wiki/Representational_State_Transfer>`_
service, so do think about that direction if you are implementing your
own protocol.