Interfacing multi-threaded C API's with Python using SWIG ...

Ran into an interesting problem yesterday. There is a third party C library that I need to interface to using Python. Normally, this isn't a problem. Just use SWIG to make a Python Wrapper on the desired API functions and call away. But this library was a little nastier because it spawned other threads and notified the caller via callbacks. Callbacks on their own aren't really a big deal for SWIG, you just have to handle the delegation from C to Python yourself, which is easy to do in the .i file. But when the callback is coming from another thread there are some twists.

CPython is notorious for the Global Interpreter Lock (GIL). You have to release the GIL whenever you do blocking I/O and re-obtain it when you are done. With a callback it's a little different, you want to acquire the lock, call Python and release it again when your back in C-land.

Starting off, my .i file looked something like this:

%module probe
%{
#include "myapi.h"
%}

%include "windows.i"

typedef void __stdcall (*HandleDataCallBack)(char*, int);

int Start(HandleDataCallBack);
int Stop();
int SendInquiry();

%{
PyObject* python_callback = 0;

void __stdcall api_start_callback(char* foo, int blah)
{
PyObject *arglist;
PyObject *result;
PyGILState_STATE gstate;

if (!PyCallable_Check(python_callback))
{
PyErr_SetString(PyExc_TypeError, "Python callback must be callable");
return;
}
arglist = Py_BuildValue("(si)", foo, blah);

gstate = PyGILState_Ensure();

result = PyEval_CallObject(python_callback, arglist);

Py_XDECREF(arglist);
Py_XDECREF(result);

PyGILState_Release(gstate);
}

void alt_start(PyObject *pyfunc)
{
python_callback = pyfunc;
Start(api_start_callback);
Py_INCREF(pyfunc);
}
%}

void alt_start(PyObject*);

Essentially what I'm doing here is defining a new function alt_start() to replace the API Start() function. A well known callback (api_start_callback) is then handed off the the real Start(). api_start_callback() does the heavy lifting of assembling the arguments, acquiring the GIL, calling the Python callback and then giving the GIL back to Python.

For other places where the threading is an issue, SWIG can take of this itself when you run Swig with the -threads option. In my case: swig -python -threads probe.i

The simplified Python code for calling this looks something like this:

import probe
import time

def ProbeCallback(foo, blah):
....print "Foo: %s, Blah: %d" % (foo, blah)

probe.alt_start(ProbeCallback)
probe.SendInquiry()
time.sleep(60)
probe.Stop()

Someday perhaps, I'll be able to call Start() directly and pass my Python callback handler without the need for this special handling code. Until then, the work around isn't too horrible.