Embedding Python in C Applications

Many applications, including most of those shipped with Windows, Linux, KDE, Gnome, and Apache are written in C. C is perhaps the most universal of all programming languages -- its expressive power, portability, minimalism, and speed make it a popular choice.

Many applications, including most of those shipped with Windows, Linux, KDE, Gnome, and Apache are written in C. C is perhaps the most universal of all programming languages — its expressive power, portability, minimalism, and speed make it a popular choice.

There are, of course, other programming languages, each with its own unique advantages. Indeed, programmers often choose a particular language based on the characteristics and challenges of the task at hand. For example, Perl is invaluable for text processing; PHP makes quick work of web site development; and Python provides for rapid, object-oriented development.

Ideally, all programming languages would “link” somehow, so you could combine and re-use libraries and source code from any one of them in a single executable. (That’s the premise of Microsoft’s .NET and the model for Perl’s new Parrot engine.)

But, alas. If you’re coding in C, you can’t mix and match. Or can you? As it turns out, many scripting languages, including PHP, Perl, and Python, can all be extended with C. And, in the cases of Perl and Python, you can reciprocally extend your C code by embedding interpreters that run scripts.

In fact, Python was explicitly crafted to be both embeddable and extensible: you can integrate the Python interpreter into your C application; you can share data and objects between your C and Python code; and you can even create user interfaces for your software written entirely in Python.

Embedded Python provides a powerful object system and “macro” language for C applications. C code can extend Python in ways its developers did not or could not foresee. This month, let’s see how Python scripts and C code can be mixed.

Embedding and Extending

Although embedding and extending Python differ in intent, both are realized with the same set of C functions. The same API is used to convert C data to Python objects, to run Python code, and to convert Python objects back to C data.

The Python C API is broken down into three layers: the Very High Level Layer (VHLL); the Abstract Object Layer (AOL); and the Concrete Object Layer (COL). These layers define functions that allow you to use Python from C — anything from executing a few lines of Python code to creating Python objects and manipulating them with your program.

The VHLL defines only a few functions, used to create interpreters, run Python code, and check for errors.

The AOL functions are used to manipulate any type of object that defines a standard Python protocol. For example, many Python objects can implement Python’s mapping or sequence protocols. These protocols allow the objects to be used with Python mapping, sequence, and slicing syntax. The AOL defines functions that work with any kind of objects that define these protocols.

The COL functions are used to create new, specific Python types and to manipulate them directly. COL functions don’t work with protocols, but with concrete object types.

While the AOL and COL functions sound similar, COL is much simpler, but also more limited. Where the AOL could use a list, tuple, or any other kind of sequence, the COL is designed to be used with only specific types.

In this article, you’ll see how to use the COL because it’s simpler to understand. (A later article will show how the abstractions of the AOL can be applied to embedded Python objects.)

The Very High Level Layer

The simplest way to embed Python in a C program is to initialize an interpreter, feed it some Python code, and then retrieve any results. Consider the example shown in Listing One. It’s the most minimal example of embedding Python.

Listing One: demo.c embeds Python in a C program

#include <Python.h>

char* cmd = “print ‘this text brought to you by Python’”;

int main()
if( PyRun_SimpleString(cmd) == -1) {
printf(“There was an error executing Python code”);
return 0;

First, the Python.h file is #included: it includes the C declarations for all of the various Python API functions and data structures. The next line defines a C string of Python code that, when executed, simply prints a friendly message. Finally, main() initializes the interpreter, runs the command (and checks for an error), and finalizes the interpreter. That’s all there is to it.

Compiling this example requires that the Python source code be installed. If you don’t have that source code already, download it from http://www.python.org, unpack it, and look in the Demo/embed directory. That directory contains a Makefile file that can be used to compile Listing One.

To compile the example:

1. Save the C code into a file called demo.c.

2. Copy the Makefile from the source package to your working directory.

3. Edit the Makefile and change the lines marked with XXX. Typically you need only change the srcdir and builddir variables to point to where you unpacked the Python source code.

4. Next, run make demo. This should build a demo program in your working directory that can be executed from a prompt.

If all goes well, you should be greeted with a friendly message when you run the demo. PyRun_SimpleString() takes a string of Python code as its argument, and executes it in the interpreter.

If you get an error, check your Makefile again and make sure everything is set correctly. (The steps required to build a C program can vary greatly from one platform to the next. You can find additional documentation on compilation at http://www.python.org.)

There are several other functions in the VHLL API that can be used to execute code:

* PyRun_SimpleFile() is similar to PyRun_Simple String(), but the Python source code is read from a file instead of an in-memory string.

* PyRun_InteractiveOne() reads and executes a single statement from an interactive device such as your terminal.

* PyRun_InteractiveLoop() reads and executes statements from a file associated with an interactive device until EOF is reached.

* PyRun_String() runs a string in the context provided by arguments. The context is a Python dictionary of local and global variables.

* PyRun_File() is similar to PyRun_String(), but the Python source code is read from a file instead of an in-memory string.

* Py_CompileString() parses and compiles the Python source code in the string argument, returning the resulting code object. Once a code object has been compiled, it can be executed directly by the Python virtual machine.

Using any of these functions, you can embed a Python interpreter directly into your C programs.

The Concrete Object Layer

While the functions listed above are certainly convenient, they don’t offer you a very sophisticated way to communicate with the Python interpreter. Passing Python code in a string is slow and unwieldy.

Since you’re writing a C program and Python itself is written in C, it’s much easier work with the Concrete Object Layer to bypass the actual Python language and work with Python objects directly in the C language. New Python objects can be created in C by calling a COL constructor method that creates a new object and returns it.

Constructors are very specific, like PyDict_New() or PyList_New(), which create a new Python dictionary and list, respectively. Python also comes with a function that lets you build Python objects from C data named Py_BuildValue().

For example, you can create a Python dictionary, populate it with values, and pass it into the interpreter for further processing by a Python script. Listing Two shows how.

Listing Two: demo2.c builds and passes a dictionary

FILE* fp;
PyObject *my_dict, *my_module;
my_dict = PyDict_New();
PyDict_SetItemString(my_dict, “key1″,
Py_BuildValue(“s”, “this is a string”));
PyDict_SetItemString(my_dict, “key2″,
Py_BuildValue(“i”, 13));
PyDict_SetItemString(my_dict, “key3″,
Py_BuildValue(“(is)”, 13, “this is a string”));
my_module = PyImport_AddModule(“my_module”);
PyModule_AddObject(my_module, “my_dict”, my_dict);
fp = fopen(filename, “r”);
PyRun_SimpleFile(fp, “demo2.py”);
return 0;

The listing shows off several new features. First, you can see that two pointers to the type PyObject are declared. These two variables will point to a new module and dictionary. The next few lines create the dictionary and populate it with three (key, value) pairs. The PyDict_SetItemString() assigns a mapping with a string key (which is the most common case), although you can also use PyDict_SetItem() to use any kind of Python object for a key.

Notice how the Py_BuildValue() function is used to create Python objects from C data. The first argument of Py_BuildValue() is a format string that tells the function what kind of object to build, and the final arguments are the C data that represent the values of the new objects. In the above example, a string, an integer, and a tuple containing an integer and a string are all easily constructed. (Py_Build Value() is documented on the Python web page in the Extending and Embedding API Reference.)

Next, the code creates a new module with PyImport_ AddModule(). This function looks for an existing module named my_module or creates a new one if it’s not found. Here, a new module is created because there are no standard modules with that name. The next line adds the my_dict dictionary to the my_module module, creating a new Python module with only one object in it. Finally, the call to PyRun_ SimpleFile() loads and runs a file named demo2.py. That file contains the following code:

from my_module import my_dict

for key, value in my_dict.items():
print “The key is”, key, “The value is”, value

This Python script is really simple: it imports the dictionary you just created in C and prints the keys and values out.

When you compile and run Listing Two, you get the following output:

The key is key3 The value is (13,
‘this is a string’)
The key is key2 The value is 13
The key is key1 The value is this is a string

For brevity, Listing Two omits error checking. Your code should check return codes and values. For example, COL functions that return new objects (like PyDict_New() or Py Import_AddModule()) return NULL if an error occurred. Other COL functions return an integer value to indicate success or failure. If a function returns an error value, the Python interpreter has set an exception that you should handle.

For example, consider the possibility that the PyImport_ AddModule() returns NULL. In this case, you should abort your program so that later calls that use the module object don’t cause segmentation faults:

my_module = PyImport_AddModule(“my_module”);
if (my_module == NULL) {
printf(“module object couldn’t be added”);

PyModule_AddObject(my_module, “my_dict”, my_dict);

If Python raises an exception, there’s usually little you can do about it other than abort or try again.

By the way, Python API functions that expect objects almost never expect those objects to be NULL — that would add many, many unnecessary checks to the Python code and generally slow down the language and your programs. Instead, you must check for those conditions yourself and handle them appropriately. Only you can prevent core files.

Every COL function is documented in the Extending and Embedding C API Reference on the Python.org home page. COL functions are divided into categories, depending on the type of object involved. Listing Two demonstrates some of the PyDict() functions, and there are many others for all the various types that come with Python. If you’re doing embedding or extending programming, this guide will be invaluable to you.

No Snake Oil Here

Embedding a Python interpreter and using Python functions within a C program is easy and very functional. This article provided a brief introduction, as there are more issues to deal with when it comes to Python and C. In the future, we’ll see how to create new Python extension modules in C, how to create new types of Python objects in C, and more.

Michel Pelletier is a Python, Java, and C programmer, author of “The Zope Book.” You can reach Michel via e-mail at michel@ dialnetwork.com.

Comments are closed.