Strategies for Scripting

Scripting has been a recurring theme in this column. It wasn't planned that way -- but the fact that every article to date has featured scripting in some way shouldn't be overlooked.

Scripting has been a recurring theme in this column. It wasn’t planned that way — but the fact that every article to date has featured scripting in some way shouldn’t be overlooked.

When a program is scriptable, you can apply the program in novel and inventive ways. Better yet, if a program is scriptable, the work to extend the program can be distributed and delegated.

Can’t highlight C# in vim? Just copy the Java configuration file, add and subtract a few keywords, and you’re done. Need to convert ASCII text to an image? Check the Web for a suitable GIMP script (http://lemur.ecs.soton.ac.uk/~cjg/script-fu/).

Simply put, adding scripting to an application is powerful stuff. Provide some basics and all the hooks and leave the rest to your user’s imaginations.

While the concept of scripting isn’t new, it has only recently come into vogue. As a result, programmers often rush in with the best intentions to enable scripting, but don’t pause to consider or research prior art. Developers tend to reinvent the wheel or impose their own biases towards this language or that. For example, if a programmer’s favorite language is Python, he may make his application scriptable only in Python. This may not seem that harmful, but consider what could be achieved if biases could be transcended.

This month, let’s look at several approaches to scripting to see what works and to learn what to avoid.

Elisp-centric Emacs

emacs (http://www.gnu.org/software/emacs/emacs-paper.html) has been around for so long, there’s a good chance that it’s older than you. emacs‘s long life is a tribute to its forward-looking design — a design that deliberately included extensibility through scripting in a dialect of Lisp called Elisp (http://www.emacswiki.org/cgi-bin/wiki.pl).

Over the years, quite a few applications have been built on top of emacs and Elisp, and an active community of users and programmers has developed around the popular text editor. Why Lisp? Richard M. Stallman, the original author of emacs, explained his rationale in an old interview by Linuxcare (http://www.linuxcare.com/viewpoints/os-interviews/latest.epl):

LINUXCARE: How did you come to choose Lisp as the emacs engine?

STALLMAN: Lisp is the most powerful programming language, and if you want an interpreter, Lisp is the best. None of the other languages come anywhere near Lisp in their power. The most exciting things about Lisp are read, eval, and print. If you look at other languages, there’s no equivalent for any of those.

Relatively speaking, Lisp doesn’t seem as powerful as it used to be because abstractions like garbage collection and lexical closures are readily available to programmers in the mainstream. However, it’s sobering to think that we’re just now able to appreciate what Lisp already had decades ago. Considering the period when emacs was implemented, Lisp was an excellent choice.

Also, in an essay titled “Why you should not use Tcl”, Stallman retells a very important lesson learned while implementing emacs.

The principal lesson of emacs is that a language for extensions should not be a mere “extension language”. It should be a real programming language, designed for writing and maintaining substantial programs. Because people will want to do that!

By “real”, Stallman meant that a programming language should support a mature set of abstractions that allow the programmer to implement programs beautifully rather than in a contrived way. Lisp satisfies this criteria.

On the other hand, Lisp is not without its faults. To people who have their first programming experience using a language like Basic or JavaScript, Lisp just looks really weird. Right off the bat, there’s a social barrier that Lisp must overcome. To partially solve this, John Tobey tried to bring Perl to emacs. His Perl module, called Emacs::EPL (http://john-edwin-tobey.org/perlmacs/), creates a bridge between emacs‘s Elisp interpeter and Perl. However, Perl is a second-class citizen here. The Perl interpreter is executed as a subprocess, and interprocess communications between emacs and the interpreter is accomplished via pipes. You also need to have a good understanding of Lisp to use Emacs::EPL — Perl alone is not enough.

The bottom line is that emacs was never meant to be programmed in anything other than Lisp; ultimately, the decision to use Lisp has limited emacs‘s potential.

Vim and Diversity in Scripting

vi is the other text editor in Unix land, and the philosophy embodied by vi and its derivatives is diametrically opposed to much of emacs.

Where emacs is big, vi is small. Where emacs tries to do everything by itself, vi focuses on the core task of text editing. Where emacs prefers a modeless style of editing, vi makes very explicit use of modes (such as insert mode and command mode). In fact, the use of modes is one of vi‘s defining characterstics, something that sets it apart from almost every other editor in existence.

Suffice it to say, emacs and vi are very different from each other. However, scripting has added an interesting twist to this story.

The most popular editor in the vi family is vim (http://www.vim.org/), and vim became scriptable in the late 1990s. Scripting vim is odd, because vi users used to scoff at emacs users for implementing all kinds of functionality not directly related to the task of text editing. For example, some vim users might see emacs‘s built-in psychotherapist (accessible via M-x doctor) and feel a strange combination of amusement and disgust. vi aficionados think the multiple mail and news clients that come bundled with emacs are blasphemous. “Emacs isn’t a text editor; it’s an operating system!”

What would vi people do if they found out that Tetris had been implemented on top of vim? What if they discovered that a project called VINE (http://www.mossbayeng.com/ron/ vim/vine.html) aims to extend vim to handle all one’s mail, news, and calendaring needs? In light of this, people who use vim need to ask themselves whether vim and emacs are as philosophically different as they once believed.

Of course, significant differences in the scripting models exist. For one thing, vim‘s APIs are minimal compared to emacs‘s. However, vim can support many scripting languages simultaneously. In addition to its own scripting language, Vim Script, vim can also be extended using Tcl, Perl, Python, and Ruby.

Multiple scripting languages sounds great in theory, but is less than ideal in practice. Although the problems are partly technical, the social issues are much more significant. Specifically, it’s very rare for someone to compile vim with support for more than one scripting language at a time. You can do it — the build scripts support it — but people tend to demonstrate an amazing loyalty to their scripting language of choice.

Unfortunately, this means that if a useful extension is created using vim‘s Ruby interface, but you only compiled in support for Perl, you’re out of luck. This is such a serious problem that people who write extensions for vim opt to use Vim Script instead of the more powerful alternatives. Vim Script captures the largest population of users. To improve this situation, extension writers need to be assured that any scripting language that vim supports will be enabled in a majority of vim installations.

(You’d be surprised how negligible the impact of enabling Tcl, Python, Perl, and Ruby in a single vim binary actually is. Start-up time is not noticably different because the interpreters are started on-demand. File size increases by only 60KB, and memory usage can be counted in two ways. If you count shared libraries, memory consumption increases 720KB; if you don’t, then it’s only increased by 180KB. Neither increase is that much by today’s standards. Remember, this is with four additional interpreters. See the sidebar The Efficiency of Vim for details.)

The Efficiency of Vim

Enabling Tcl, Python, Perl, and Ruby in a single I<vim> binary is simple and costs virtually nothing. Compared to a stock binary with no interpreters enabled, the differences in start-up time and memory usage are negligible.

In the comparisons that follow, vim-6.1-rppt has Ruby, Perl, Python, and Tcl enabled, and vim-6.1 is a stock binary with no interpreters enabled.

Time To Start and Stop Vim

Write a script to repeatedly execute the following two commands such that your sample set is sufficiently large. Calculate the average time for each vim binary. Then, calculate the diference between these two values. In our tests, on a humble 566MHz Celeron, the difference was only 0.053 seconds.

/usr/bin/time -f “%E” ./vim-6.1-rppt -c :q
/usr/bin/time -f “%E” ./vim-6.1 -c :q

File Size

$ ls –sort=size -o vim-*
-rwxrwxr-x 1 beppu 1106708 May 30 12:51 vim-6.1-rppt
-rwxrwxr-x 1 beppu 1046356 May 30 12:56 vim-6.1

Memory Consumption

$ ps -aux | grep vim
16892 beppu 9 0 4044 4032 2468 S 0.0 3.1 0:00 vim-6.1-rppt
16891 beppu 9 0 3312 3312 1928 S 0.0 2.6 0:00 vim-6.1

The RSS column is the total amount of memory used in kilobytes. The SHARE column is the amount of shared memory used in kilobytes.

Not counting shared memory, the additional memory requirement of vim-6.1-rppt compared to vim-6.1 is (4032KB – 2468KB) – (3312KB – 1928KB) = 180KB. If you count shared memory, the difference is 4032KB – 3312KB = 720KB

Indeed, if the benefits are judged to outweigh the costs, all of the interpreters could be enabled by default in the configure script. Then, in a few years, extension authors would be free to use the language of their choice. (Perhaps this should be a TODO item for Vim 7. After all, what good is supporting multiple scripting languages if hardly anyone uses them?)

Scripting in vim also suffers from a variation on the same problem emacs had when Emacs::EPL attempted to add Perl support. In vim‘s case, Vim Script is privileged: it has access to certain capabilities of vim that neither Tcl, Perl, Python, nor Ruby have direct access to. In effect, those powerful languages are second class citizens. Anyone who wants to create a non-trivial extension to vim has to know some Vim Script. That’s not a terribly onerous requirement, but it’s worth mentioning because the GIMP has solved this problem in a very convincing fashion.

The GIMP and the Genius of the PDB

GIMP (GNU Image Manipulation Program, http://www.gimp.org) is a free Photoshop clone that’s challenged many of the stereotypes of what Unix programs can and cannot do. In addition to proving that a Unix program can be heavily graphical, one of GIMP’s most touted features was its scriptability. At first, the GIMP could only be scripted in Scheme, but mature interfaces for Perl (http://www.goof.com/pcg/marc/gimp.html) and Python (http://www.daa.com.au/~james/pygimp/) have since been developed. What’s more impressive is that all the extension languages stand on a level playing field. Therefore, although Scheme was first, it doesn’t have any special benefits over Perl or Python.

The heart of the GIMP’s flexible scripting system is the PDB (Procedural Database). As its name implies, PDB is a database of procedures (or functions). While a large portion of PDB is devoted to keeping track of C functions from libgimp, any scripting language that the GIMP supports can register its own functions in the PDB. Even better, any of the supported languages can execute any function in the database — the ultimate in interoperability.

For example, a script written in Python can execute a C, Scheme, Python, and even a Perl function registered in the PDB, and it’s not at all contrived. Due to the good taste of the people who wrote the Scheme, Perl, and Python interfaces, calling a function from the PDB looks just like you’re calling a native function! (See Listing One.)

Listing One: Scheme, Perl, Python, and C Working Together

# Try this in the Gimp-Python Console.

def peace():
# Scheme
“100″, # radius
“45.0″, # lighting angle
FALSE, # shadow?
(255, 255, 255), # background color
(192, 0, 0)) # sphere color

# Perl
img = pdb.perl_fu_yinyang(
256, # width
256, # height
TRUE, # insert eyes?
FALSE, # eyes are images?
“”, # top eye filename
“”, # bottom eye filename
TRUE) # anti-aliasing?

# Python
100, # radius
45.0, # lighting angle
FALSE, # shadow?
(255, 255, 255), # background color
( 0, 0, 192)) # sphere color

# C function from libgimp


GIMP’s PDB is probably one of the most important innovations in interoperability to come around in a while, and it’s unfortunate that not many people are aware of it.

The Future of Scripting

Hopefully, you can now fully appreciate scripting and how scripting in more than one language is even better (if done right).

Some of you may think that implementing something like the PDB may require too much work, but remember that we live and work in a world of Free Software. What if the PDB could be separated from the GIMP, and then generalized and documented to the point where it could be distributed independently? Then every programmer who wanted to add scripting support to his or her application could just reuse PDB from the start. If implemented, there is no question that the long-term benefits would be positive. The only question is how positive.

“The best way to predict the future is to invent it.” — Alan Kay, the creator of Smalltalk.

John Beppu <beppu@cpan.org> wants people to use their computers creatively and skillfully.

Comments are closed.