pyCurious

Many times while creating LaTeX documents I’ve thought “Wouldn’t it be great if I could write an inline python script to generate some LaTeX code here.” In this post I describe how to embed python scripts in LaTeX, how to get Vim to correctly syntax highlight the embedded python, and the modifications required to get rubber to compile LaTeX that uses python.sty. All of the code and scripts mentioned in this post are available on bitbucket.

I have already built a system where python generated .tex files are \input into my documents (script execution is managed by makefiles). I want to take this a step further—allowing python source to sit in a .tex file alongside LaTeX source.

In 2007 Martin R. Ehmsen released a LaTeX package python.sty that allows such python embedding. python.sty defines a new python environment in which arbitrary python code can be placed. Everything that is written to stdout by the embedded python code is interpreted as LaTeX in the final document.

For example, this sequence of identity matrices is generated with the mixture of LaTeX and python code below.

$N \times N$ element identity matrices for $N \in \{1,2,\ldots,5\}$:

\begin{python}
def identity(n):
    """
    Generates and returns the LaTeX code for
    a n x n identity matrix.

    """
    return '\n'.join([
        r'\left[',
        r'\begin{array}{%s}' % (n*'c',),
        '\\\\'.join('&'.join('1' if i==j else '0' for j in range(n))
                        for i in range(n)),
        r'\end{array}',
        r'\right]',
    ])

# Displays identity matrices for
# n \in {1, 2, ..., 5}.
print '\n'.join([
    r'\begin{equation*}',
    r'\begin{array}{%s}' % (5*'c',),
    '&'.join(identity(i) for i in range(1,6)),
    r'\end{array}.',
    r'\end{equation*}',
])
\end{python}

What improvements can we make to python.sty?

By default when python.sty encounters \begin{python} the following actions are executed:

1. All text up to the closing \end{python} is output to outfile.py; outfile is the basename of the .tex document being processed;
2. outfile.py is piped to the python executable with the resulting stdout and stderr stored to outfile.py.out and outfile.py.err;
3. outfile.py.out is included in the LaTeX document using \input{outfile.py.out}.

This is great when a document contains a single python script, however since every environment overwrites outfile.py{.out,.err} it is impossible to view the python output and errors of all script executions. An updated python.sty fixes this by writing each environment to a different outfileN.py where N is a counter that is incremented each environment.

The updated python.sty also stores python’s return code to outfileN.rc. If python fails with a return code other than 0 the traceback and any other stderr output is included with red text in the document.

For example, if we deliberately throw an exception the traceback is included in the document body.

\begin{python}
# Create a deliberate exception.
print 'This text will be displayed.'
print 1/0
print 'This text won\'t be displayed.'
\end{python}

Can Vim correctly syntax highlight LaTeX embedded python?

Python syntax-highlighted as LaTeX isn’t pretty. A stackoverflow question and answer describes how to get Vim to correctly highlight .tex as LaTeX while using the python syntax highlighter for \begin{python}\end{python} blocks. A condensed series of steps is

1. Create a new Vim syntax file ~/.vim/syntax/pytex.vim containing:

   " Syntax for LaTeX files with embedded Python
   " environments.
   let b:current_syntax = ''
   unlet b:current_syntax
   runtime! syntax/tex.vim
   
   let b:current_syntax = ''
   unlet b:current_syntax
   syntax include @TeX syntax/tex.vim
   
   let b:current_syntax = ''
   unlet b:current_syntax
   syntax include @Python syntax/python.vim
   syntax region pythonCode matchgroup=Snip start="\\begin{python}" end="\\end{python}" containedin=@TeX contains=@Python
   
   hi link Snip SpecialComment
   let b:current_syntax = 'pytex'
       

2. Add the following line to your .vimrc:

       au BufRead *.tex set syntax=pytex
       

The LaTeX code snippits in this post were generated using this syntax highlighting setup and the :TOhtml command.

Rubber can’t compile documents that use the python.sty package!

Rubber is a python command line tool that greatly simplifies compiling LaTeX documents. Unfortunately it provides no simple way to pass additional command line arguments to the underlying latex and pdflatex executables.

python.sty uses the \write18 command that executes an arbitrary shell command from within LaTeX. This is, by default, restricted for security reasons. Full \write18 functionality can be enabled by passing the –shell-escape argument to latex or pdflatex.

I’ve created a branch of Rubber that accepts and passes on the –shell-escape command line argument. You should download and install this latest version if you wish to use Rubber in concert with python.sty.

With this shell-escape Rubber branch the example.tex file can be compiled using:

mkdir .build
rubber -d --shell-escape --into=.build example.tex
cp .build/example.pdf .

Links of Interest

1. https://bitbucket.org/brotchie/python-sty/ – python.sty source, Vim syntax scripts, and an example .tex file.
2. http://stackoverflow.com/questions/5176972/trouble-using-vims-syn-include-and-syn-region-to-embed-syntax-highlighting – stackoverflow question detailing how to syntax python within LaTeX documents.
3. https://code.launchpad.net/~brotchie/rubber/shell-escape – shell-escape enabled branch of Rubber.
4. http://thewikiblog.appspot.com/blog/python-inside-latex – where I found a copy of the original python.sty.

I heavily use KeepNote for taking random notes, keeping a daily journal, and managing a research pipeline. I’ve recently discovered the power of mind mapping with FreeMind, an open source mind mapping application. Out of curiosity I wanted to see what my KeepNote notebook looked like as a mind map.

KeepNote has an extension system and an easily navigable object hierarchy. I used the existing export_html extension as a base, adjusting the output to the FreeMind file format. The content of KeepNote notes are added as richcontent notes on the FreeMind nodes. Both KeepNote and FreeMind save notes in XHTML making the translation trivial.

The code is available at https://bitbucket.org/brotchie/export_freemind/overview.

Export of my primary KeepNote notebook. KeepNote on left; resultant FreeMind mind map on right.

Six months ago I indulged in some C++ metaprogramming hackery. I’d read through Abraham’s and Gurtovoy’s excellent book and wanted to try out some of their concepts. I recently found some of the code I’d written during this phase on paste bin. Here I present the two example with a brief description.

The first example implements some contrived functionality that demonstrating the metaprogramming equivalents of vectors, maps, and iterators. The source is available at http://pastebin.com/UMPh6TuX.

The second example brute forces problem 1 from Project Euler (the problem statement is repeated in the heading comment). The source is available at http://pastebin.com/NK9BMyz6.

(Image source http://martinfowler.com/articles/rubyAtThoughtWorks.html)

libpurple is the instant messaging library underlying the Pidgin and Adium multi-protocol IM clients. Out of the box libpurple supports a wide variety of protocols accessible via a consistent programmatic interface. The Carman project provides a set of python bindings for libpurple. The source for the bindings can be checked out from SVN at http://vcs.maemo.org/svn/carman/trunk/python-purple/.

The python-purple bindings are provided as a set of .pyx files that must be compiled into a Python module using Cython. Instructions on compiling the module are available at http://briglia.net/wiki/tiki-index.php?page=Python-purple+Howto. Although the instructions mention Python 2.5, they work fine when Python 2.6 is used.

The trunk version of the python-purple bindings utilizes python-ecore (a python abstraction of the core libraries used by Enlightenment) to manage the main loop. If you wish to use the twisted mainloop instead simply apply the following patch (python-purple-twisted.patch) and recompile the module.

Index: python-purple/purple.pyx
===================================================================
--- python-purple/purple.pyx	(revision 2088)
+++ python-purple/purple.pyx	(working copy)
@@ -22,8 +22,8 @@
 cdef extern from "c_purple.h":
     glib.guint glib_input_add(glib.gint fd, eventloop.PurpleInputCondition condition, eventloop.PurpleInputFunction function, glib.gpointer data)
 
-import ecore
 import signal
+from twisted.internet import reactor
 
 cdef glib.GHashTable *c_ui_info
 
@@ -91,8 +91,8 @@
         if default_path:
             util.purple_util_set_user_dir(default_path)
 
-        # adds glib iteration inside ecore main loop
-        ecore.timer_add(0.001, self.__glib_iteration_when_idle)
+        # adds glib iteration inside twisted main loop
+        reactor.callLater(0.001, self.__glib_iteration_when_idle)
 
         # libpurple's built-in DNS resolution forks processes to perform
         # blocking lookups without blocking the main process.  It does not
@@ -171,7 +171,7 @@
 
     def __glib_iteration_when_idle(self):
         glib.g_main_context_iteration(NULL, False)
-        return True
+        reactor.callLater(0.001, self.__glib_iteration_when_idle)
 
     def purple_init(self):
         '''Initializes the purple.
Index: python-purple/conversation_cbs.pxd
===================================================================
--- python-purple/conversation_cbs.pxd	(revision 2088)
+++ python-purple/conversation_cbs.pxd	(working copy)
@@ -104,7 +104,7 @@
         message = None
 
     # FIXME: Maybe we need add more purple flags in the future
-    if flags & conversation.PURPLE_MESSAGE_SEND:
+    if int(flags) & conversation.PURPLE_MESSAGE_SEND:
         flag = "SEND"
     else:
         flag = "RECV"
Index: python-purple/nullclient.py
===================================================================
--- python-purple/nullclient.py	(revision 2088)
+++ python-purple/nullclient.py	(working copy)
@@ -17,7 +17,7 @@
 #  along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-import ecore
+from twisted.internet import reactor
 import getpass
 import purple
 import sys
@@ -60,5 +60,7 @@
     # Enable account (connects automatically)
     account.set_enabled(True)
 
-    # Initializes ecore mainloop
-    ecore.main_loop_begin()
+    # Start twisted mainloop
+    reactor.run()
+
+

The amazingly comprehensive open-source quantitative finance library QuantLib supplies a set of Python bindings generated with SWIG. Unfortunately much of QuantLib’s adaptability is made available via C++ templates. With the current SWIG wrapper it’s difficult to expose the entirety of QuantLib’s functionality without compiling every permutation of template parameter.

I discovered this quirk whilst trying to apply cubic spline interpolation to a zero curve. It turns out that by default the SWIG interface only exposes a linearly interpolated zero curve class. Fortunately there are some nice macros within the SWIG interface that ease the exposure of additional interpolation schemes, albeit with a recompile of the Python module.

Zero curves with additional interpolation methods can be added to the end of QuantLib-SWIG-0.9.7/SWIG/zerocurve.i using the export_zero_curve macro as follows:

export_zero_curve(ZeroCurve,Linear);
export_zero_curve(CubicZeroCurve,Cubic);

After recompiling the Python bindings you’ll now have a CubicZeroCurve class that performs cubic spline interpolation between data points.

This approach can be used throughout much of the SWIG interface files to expose template customized QuantLib classes.

SWIG works great out of the box when you’re dealing with basic types. Once you start to wrap more complicated APIs the type conversion magic can only go so far.

I have been writing a Python SWIG interface for a library that accepts stdio FILE* pointers for input/output. Here I’ll present a minimal example where a Python file object is passed as a FILE* to a simple C function. All source is available here if you wish to follow along.

char buffer[1024];
 
char*
message(FILE *input)
{
    memset(buffer, 0, sizeof(buffer));
    fread(buffer, sizeof(buffer), 1, input);
    return buffer;
}

This simply reads from a file stream until it’s closed and returns what was read. I preemptively apologise for it not being thread safe.

We’ll create a module called test wrapping the C function message such that the following code (example.py) will output “Hello World”.

import os
import test
 
# Create and open pipe as file objects
r,w = os.pipe()
fr, fw = os.fdopen(r, 'r'), os.fdopen(w, 'w')
 
# Write and close
fw.write('Hello World')
fw.close()
 
# Read data from pipe using test module
print test.message(fr)

The first cut at a SWIG interface file yields

%module test
%{
#include "test.h"
%}
char *message(FILE *input);

Unfortunately when we build and run example.py we get a TypeError indicating the argument we passed couldn’t be converted to a FILE*.

$ python setup.py build_ext --inplace
$ python example.py 
Traceback (most recent call last):
  File "example.py", line 17, in <module>
    print test.message(fr)
TypeError: in method 'message', argument 1 of type 'FILE *'

How do we make SWIG correctly convert the Python file type to a FILE*? The solution lies in a SWIG feature called a typemap.

Typemaps allow you to write short stubs in C to convert objects from Python to C and vice-versa. Writing them requires you to have some knowledge of the Python API, for simple conversions however you can make good progress simply by digging through the Python include directory.

Adding the following typemap to our SWIG interface file tells squid how to convert incoming Python objects into FILE* pointers. Conveniently the Python C API provides the PyFile_Check function that checks if a PyObject* is of the PyFile type and PyFile_AsFile functions that returns a FILE* given a PyFile instance.

/* Converts a PyFile instance to a stdio FILE* */
%typemap(in) FILE* {
    if ( PyFile_Check($input) ){
        $1 = PyFile_AsFile($input);
    } else {
        PyErr_SetString(PyExc_TypeError, "$1_name must be a file type.");
        return NULL;
    }
}

Now when we build test and run example.py we get the desired result.

$ python example.py 
Hello World

Notice that if we pass a non-file type to message a TypeError will be raised.

>>> import test
>>> test.message('not a file')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: input must be a file type.

This simple example demonstrates one powerful feature of SWIG that can be expanded to wrap complex C and even C++ APIs.

A decorator is simple syntactic sugar for applying a transformation to a function or method. Both

class A(object):
    @decorator
    def name(self):
        return "foo"

and

class A(object):
    def name(self):
        return "foo"
    name = decorator(name)

are equivalent. The quintessential decorator example contains a function returning a closure that performs some transformation on the wrapped function’s return value. For example

def upper(wrapped):
    def shim(self):
        # convert returned value to upper case
        return wrapped(self).upper()
    return shim
 
class A(object):
    @upper
    def name(self):
        return "foo"
 
>>> a = A()
>>> print a.name()
FOO

When you wish to create a decorator that accepts additional arguments such as

class A(object):
    @expose("get_name")
    def name(self):
        return "foo"

the hierarchy of closures can become complicated, and in my opinion unpythonic.

Class based decorators with arguments

Since a decorator can be any callable and we can override a class’s __call__ method to emulate a function call, a “decorator class” can be constructed. This is nice because in the _shim we have access to both the decorator class instance (self) and the instance of the class containing the decorated method (instance).

class expose(object):
    def __init__(self, name):
        self._name = name
        self._wrapped = None
 
    def _shim(self, instance, *args, **kwargs):
        print 'Special RPC handling for %s, %r, %r' % \
            (self._name, args, kwargs)
        return self._wrapped(instance, *args, **kwargs)
 
    def __call__(self, wrapped):
        self._wrapped = wrapped
        def shim(instance, *args, **kwargs):
            return self._shim(instance, *args, **kwargs)
        return shim
 
class A(object):
    @expose("get_name")
    def name(self):
        return "foo"
 
>>> a = A()
>>> print a.name()
Special RPC handling for get_name, (), {}
foo

Class based decorators without arguments

Expose handles the case where the decorator accepts arguments. But what if we just wanted to write

class A(object):
    @expose
    def name(self):
        return "foo"

Since expose is now a class, its __init__ method will be called with the method name as an argument. We cannot simply implement a __call__ method on expose because we want to get hold of the decorated method’s containing class instance. The trick here is to change expose into a non-data descriptor by adding a __get__ special method. This __get__ is passed the decorator’s containing class instance, thus we create and return a closure that calls expose’s _shim method with the desired arguments.

class expose(object):
    def __init__(self, wrapped):
        self._wrapped = wrapped
 
    def _shim(self, instance, *args, **kwargs):
        print 'Special RPC handling for %s, %r, %r' % \
            (self._wrapped.__name__, args, kwargs)
        return self._wrapped(instance, *args, **kwargs)
 
    def __get__(self, instance, owner):
        def shim(*args, **kwargs):
            return self._shim(instance, *args, **kwargs)
        return shim
 
class A(object):
    @expose
    def name(self):
        return "foo"
 
>>> a = A()
>>> print a.name()
Special RPC handling for name, (), {}
foo

Generic class based decorator

We can combine the two methods above to create a generic decorator base class that handles both the argument and no argument case. The implementation of decorator_base can be found here. Below is an example where the example decorator class extends decorator_base.

class example(decorator_base):
    default_positional_args = ('Default Name',)
 
    def _arg_init(self,
                  name,
                  description="Default Description"):
        self._name = name
        self._description = description
 
    def _shim(self, instance, *args, **kwargs):
        print self._name, self._description, args, kwargs
        return self._wrapped(instance, *args, **kwargs)
 
class Test(object):
    @example
    def upper(self, text):
        return text.upper()
 
    @example('Verbose Name', description='Verbose Description')
    def lower(self, text):
        return text.lower()
 
>>> t = Test()
>>> print t.upper('hElLo')
Default Name Default Description ('hEllo',) {}
HELLO
>>> print t.lower('HeLlO')
Verbose Name Verbose Description ('HeLlO',) {}
hello

An equivalent class based decorator can be constructed using metaclasses, this will be the subject of further post. I hope this post gives sheds some light on using classes to implement decorators.

Metaclasses seem to be an obscure, misunderstood area of Python’s object model. Here is a simple example that I’ve used to both understand and explain what a metaclass is a how it can be used.

This code snippet was spawned from a colleague’s query.

“What would be the most pythonic way to override a class type’s string representation?”

Expressed in code he wanted a new-style class type A to behave like

>>> A
<class '__main__.A'>
>>> str(A)
'Hello World'

Unfotunately overriding A’s __str__ method

>>> class A(object):
...  def __str__(self):
...   return "Hello World"
... 
>>> A
<class '__main__.A'>
>>> str(A)
"<class '__main__.A'>"
>>> str(A())
'Hello World'
>>>

only overrides the string representation of instances of A rather than the type A.

The solution to the problem leads us to metaclasses. My colleague’s conundrum can be solved by creating a new-style class with base type and a __str__ method. This acts to override the __str__ method of A’s type rather than instances of A.

>>> class meta_A(type):
...  def __str__(cls):
...   return "Hello World"
... 
>>> class A(object):
...  __metaclass__ = meta_A
... 
>>> A
<class '__main__.A'>
>>> str(A)
'Hello World'
>>> str(A())
'<__main__.A object at 0x10049e710>'

Thus we have a pythonic mechanism for overriding the string representation of a class type.

Note that there is new syntax in Python 3.0 for specifying a metaclass.

>>> class A(metaclass=meta_A):
...  pass
...