pyCurious

Using python-purple with the twisted mainloop

admin — Fri, 04 Jun 2010 02:26:53 +0000

libpurple is the instant messaging library underlying the Pidgin and Adium multi-protocol IM clients. Out of the box libpurple supports a wide variety of protocols accessible via a consistent programmatic interface. The Carman project provides a set of python bindings for libpurple. The source for the bindings can be checked out from SVN at http://vcs.maemo.org/svn/carman/trunk/python-purple/.

The python-purple bindings are provided as a set of .pyx files that must be compiled into a Python module using Cython. Instructions on compiling the module are available at http://briglia.net/wiki/tiki-index.php?page=Python-purple+Howto. Although the instructions mention Python 2.5, they work fine when Python 2.6 is used.

The trunk version of the python-purple bindings utilizes python-ecore (a python abstraction of the core libraries used by Enlightenment) to manage the main loop. If you wish to use the twisted mainloop instead simply apply the following patch (python-purple-twisted.patch) and recompile the module.

Index: python-purple/purple.pyx
===================================================================
--- python-purple/purple.pyx	(revision 2088)
+++ python-purple/purple.pyx	(working copy)
@@ -22,8 +22,8 @@
 cdef extern from "c_purple.h":
     glib.guint glib_input_add(glib.gint fd, eventloop.PurpleInputCondition condition, eventloop.PurpleInputFunction function, glib.gpointer data)
 
-import ecore
 import signal
+from twisted.internet import reactor
 
 cdef glib.GHashTable *c_ui_info
 
@@ -91,8 +91,8 @@
         if default_path:
             util.purple_util_set_user_dir(default_path)
 
-        # adds glib iteration inside ecore main loop
-        ecore.timer_add(0.001, self.__glib_iteration_when_idle)
+        # adds glib iteration inside twisted main loop
+        reactor.callLater(0.001, self.__glib_iteration_when_idle)
 
         # libpurple's built-in DNS resolution forks processes to perform
         # blocking lookups without blocking the main process.  It does not
@@ -171,7 +171,7 @@
 
     def __glib_iteration_when_idle(self):
         glib.g_main_context_iteration(NULL, False)
-        return True
+        reactor.callLater(0.001, self.__glib_iteration_when_idle)
 
     def purple_init(self):
         '''Initializes the purple.
Index: python-purple/conversation_cbs.pxd
===================================================================
--- python-purple/conversation_cbs.pxd	(revision 2088)
+++ python-purple/conversation_cbs.pxd	(working copy)
@@ -104,7 +104,7 @@
         message = None
 
     # FIXME: Maybe we need add more purple flags in the future
-    if flags & conversation.PURPLE_MESSAGE_SEND:
+    if int(flags) & conversation.PURPLE_MESSAGE_SEND:
         flag = "SEND"
     else:
         flag = "RECV"
Index: python-purple/nullclient.py
===================================================================
--- python-purple/nullclient.py	(revision 2088)
+++ python-purple/nullclient.py	(working copy)
@@ -17,7 +17,7 @@
 #  along with this program.  If not, see .
 #
 
-import ecore
+from twisted.internet import reactor
 import getpass
 import purple
 import sys
@@ -60,5 +60,7 @@
     # Enable account (connects automatically)
     account.set_enabled(True)
 
-    # Initializes ecore mainloop
-    ecore.main_loop_begin()
+    # Start twisted mainloop
+    reactor.run()
+
+

Using additional term structure interpolation methods with QuantLib’s Python bindings

admin — Tue, 02 Feb 2010 10:05:36 +0000

The amazingly comprehensive open-source quantitative finance library QuantLib supplies a set of Python bindings generated with SWIG. Unfortunately much of QuantLib’s adaptability is made available via C++ templates. With the current SWIG wrapper it’s difficult to expose the entirety of QuantLib’s functionality without compiling every permutation of template parameter.

I discovered this quirk whilst trying to apply cubic spline interpolation to a zero curve. It turns out that by default the SWIG interface only exposes a linearly interpolated zero curve class. Fortunately there are some nice macros within the SWIG interface that ease the exposure of additional interpolation schemes, albeit with a recompile of the Python module.

Zero curves with additional interpolation methods can be added to the end of QuantLib-SWIG-0.9.7/SWIG/zerocurve.i using the export_zero_curve macro as follows:

export_zero_curve(ZeroCurve,Linear);
export_zero_curve(CubicZeroCurve,Cubic);

After recompiling the Python bindings you’ll now have a CubicZeroCurve class that performs cubic spline interpolation between data points.

This approach can be used throughout much of the SWIG interface files to expose template customized QuantLib classes.

Solution to Mercurial repository hooks not being run when served over http

admin — Wed, 09 Dec 2009 03:32:13 +0000

At work we use trac for feature planning and defect management. Having recently moved from Subversion to Mercurial (Hg) we needed to install hooks in our Hg repositories so that trac tickets could be modified and closed via special commit messages commands.

Hooks are added to a repository’s configuration file (.hg/hgrc) as follows

[hooks]
incoming = /usr/bin/trac-admin /trac/project/directory added Reponame $HG_NODE

This repository is then served using hgwebdir running via Apache mod_wsgi. hgwebdir.config configures the repositories that are shown, their filesystem locations, permissions, etc.

For the life of me I couldn’t figure out why the trac incoming hook wasn’t running. After a lot of tail chasing and advice from #mercurial we discovered the solution.

It turns out that by default Hg will ignore .hg/hgrc files that aren’t owned by a trusted user. Since our repositories are 2775/664 root:apache and the server process only trusts .hg/hgrc files that it owns, all our repository specific config options are ignored.

Simple adding

[trusted]
users = root

to hgweb.config fixed the problem and our trac hooks now run as desired.

Passing Python file objects across a SWIG interface

admin — Fri, 04 Dec 2009 13:50:39 +0000

SWIG works great out of the box when you’re dealing with basic types. Once you start to wrap more complicated APIs the type conversion magic can only go so far.

I have been writing a Python SWIG interface for a library that accepts stdio FILE* pointers for input/output. Here I’ll present a minimal example where a Python file object is passed as a FILE* to a simple C function. All source is available here if you wish to follow along.

char buffer[1024];
 
char*
message(FILE *input)
{
    memset(buffer, 0, sizeof(buffer));
    fread(buffer, sizeof(buffer), 1, input);
    return buffer;
}

This simply reads from a file stream until it’s closed and returns what was read. I preemptively apologise for it not being thread safe.

We’ll create a module called test wrapping the C function message such that the following code (example.py) will output “Hello World”.

import os
import test
 
# Create and open pipe as file objects
r,w = os.pipe()
fr, fw = os.fdopen(r, 'r'), os.fdopen(w, 'w')
 
# Write and close
fw.write('Hello World')
fw.close()
 
# Read data from pipe using test module
print test.message(fr)

The first cut at a SWIG interface file yields

%module test
%{
#include "test.h"
%}
char *message(FILE *input);

Unfortunately when we build and run example.py we get a TypeError indicating the argument we passed couldn’t be converted to a FILE*.

$ python setup.py build_ext --inplace
$ python example.py 
Traceback (most recent call last):
  File "example.py", line 17, in <module>
    print test.message(fr)
TypeError: in method 'message', argument 1 of type 'FILE *'

How do we make SWIG correctly convert the Python file type to a FILE*? The solution lies in a SWIG feature called a typemap.

Typemaps allow you to write short stubs in C to convert objects from Python to C and vice-versa. Writing them requires you to have some knowledge of the Python API, for simple conversions however you can make good progress simply by digging through the Python include directory.

Adding the following typemap to our SWIG interface file tells squid how to convert incoming Python objects into FILE* pointers. Conveniently the Python C API provides the PyFile_Check function that checks if a PyObject* is of the PyFile type and PyFile_AsFile functions that returns a FILE* given a PyFile instance.

/* Converts a PyFile instance to a stdio FILE* */
%typemap(in) FILE* {
    if ( PyFile_Check($input) ){
        $1 = PyFile_AsFile($input);
    } else {
        PyErr_SetString(PyExc_TypeError, "$1_name must be a file type.");
        return NULL;
    }
}

Now when we build test and run example.py we get the desired result.

$ python example.py 
Hello World

Notice that if we pass a non-file type to message a TypeError will be raised.

>>> import test
>>> test.message('not a file')
Traceback (most recent call last):
  File "", line 1, in <module>
TypeError: input must be a file type.

This simple example demonstrates one powerful feature of SWIG that can be expanded to wrap complex C and even C++ APIs.

Forget functions: A class based approach to Python decorators

admin — Wed, 25 Nov 2009 12:07:28 +0000

A decorator is simple syntactic sugar for applying a transformation to a function or method. Both

class A(object):
    @decorator
    def name(self):
        return "foo"

and

class A(object):
    def name(self):
        return "foo"
    name = decorator(name)

are equivalent. The quintessential decorator example contains a function returning a closure that performs some transformation on the wrapped function’s return value. For example

def upper(wrapped):
    def shim(self):
        # convert returned value to upper case
        return wrapped(self).upper()
    return shim
 
class A(object):
    @upper
    def name(self):
        return "foo"
 
>>> a = A()
>>> print a.name()
FOO

When you wish to create a decorator that accepts additional arguments such as

class A(object):
    @expose("get_name")
    def name(self):
        return "foo"

the hierarchy of closures can become complicated, and in my opinion unpythonic.

Class based decorators with arguments

Since a decorator can be any callable and we can override a class’s __call__ method to emulate a function call, a “decorator class” can be constructed. This is nice because in the _shim we have access to both the decorator class instance (self) and the instance of the class containing the decorated method (instance).

class expose(object):
    def __init__(self, name):
        self._name = name
        self._wrapped = None
 
    def _shim(self, instance, *args, **kwargs):
        print 'Special RPC handling for %s, %r, %r' % \
            (self._name, args, kwargs)
        return self._wrapped(instance, *args, **kwargs)
 
    def __call__(self, wrapped):
        self._wrapped = wrapped
        def shim(instance, *args, **kwargs):
            return self._shim(instance, *args, **kwargs)
        return shim
 
class A(object):
    @expose("get_name")
    def name(self):
        return "foo"
 
>>> a = A()
>>> print a.name()
Special RPC handling for get_name, (), {}
foo

Class based decorators without arguments

Expose handles the case where the decorator accepts arguments. But what if we just wanted to write

class A(object):
    @expose
    def name(self):
        return "foo"

Since expose is now a class, its __init__ method will be called with the method name as an argument. We cannot simply implement a __call__ method on expose because we want to get hold of the decorated method’s containing class instance. The trick here is to change expose into a non-data descriptor by adding a __get__ special method. This __get__ is passed the decorator’s containing class instance, thus we create and return a closure that calls expose’s _shim method with the desired arguments.

class expose(object):
    def __init__(self, wrapped):
        self._wrapped = wrapped
 
    def _shim(self, instance, *args, **kwargs):
        print 'Special RPC handling for %s, %r, %r' % \
            (self._wrapped.__name__, args, kwargs)
        return self._wrapped(instance, *args, **kwargs)
 
    def __get__(self, instance, owner):
        def shim(*args, **kwargs):
            return self._shim(instance, *args, **kwargs)
        return shim
 
class A(object):
    @expose
    def name(self):
        return "foo"
 
>>> a = A()
>>> print a.name()
Special RPC handling for name, (), {}
foo

Generic class based decorator

We can combine the two methods above to create a generic decorator base class that handles both the argument and no argument case. The implementation of decorator_base can be found here. Below is an example where the example decorator class extends decorator_base.

class example(decorator_base):
    default_positional_args = ('Default Name',)
 
    def _arg_init(self,
                  name,
                  description="Default Description"):
        self._name = name
        self._description = description
 
    def _shim(self, instance, *args, **kwargs):
        print self._name, self._description, args, kwargs
        return self._wrapped(instance, *args, **kwargs)
 
class Test(object):
    @example
    def upper(self, text):
        return text.upper()
 
    @example('Verbose Name', description='Verbose Description')
    def lower(self, text):
        return text.lower()
 
>>> t = Test()
>>> print t.upper('hElLo')
Default Name Default Description ('hEllo',) {}
HELLO
>>> print t.lower('HeLlO')
Verbose Name Verbose Description ('HeLlO',) {}
hello

An equivalent class based decorator can be constructed using metaclasses, this will be the subject of further post. I hope this post gives sheds some light on using classes to implement decorators.

Simple Python metaclass example

admin — Mon, 23 Nov 2009 03:16:14 +0000

Metaclasses seem to be an obscure, misunderstood area of Python’s object model. Here is a simple example that I’ve used to both understand and explain what a metaclass is a how it can be used.

This code snippet was spawned from a colleague’s query.

“What would be the most pythonic way to override a class type’s string representation?”

Expressed in code he wanted a new-style class type A to behave like

>>> A
<class '__main__.A'>
>>> str(A)
'Hello World'

Unfotunately overriding A’s __str__ method

>>> class A(object):
...  def __str__(self):
...   return "Hello World"
... 
>>> A
<class '__main__.A'>
>>> str(A)
""
>>> str(A())
'Hello World'
>>>

only overrides the string representation of instances of A rather than the type A.

The solution to the problem leads us to metaclasses. My colleague’s conundrum can be solved by creating a new-style class with base type and a __str__ method. This acts to override the __str__ method of A’s type rather than instances of A.

>>> class meta_A(type):
...  def __str__(cls):
...   return "Hello World"
... 
>>> class A(object):
...  __metaclass__ = meta_A
... 
>>> A
<class '__main__.A'>
>>> str(A)
'Hello World'
>>> str(A())
'<__main__.A object at 0x10049e710>'

Thus we have a pythonic mechanism for overriding the string representation of a class type.

Note that there is new syntax in Python 3.0 for specifying a metaclass.

>>> class A(metaclass=meta_A):
...  pass
...