Linux and Executables

For the past few months, we've been learning how the compiler and linker work together to take the programs you write and convert them into executables that the operating system can run. We've followed the process from source code to object module to executable, with static and shared libraries thrown in as well.

For the past few months, we’ve been learning how the compiler and linker work together to take the programs you write and convert them into executables that the operating system can run. We’ve followed the process from source code to object module to executable, with static and shared libraries thrown in as well.

But when it’s time to actually run your executable, what does Linux do for you?

What The Shell Does

Let’s say you’ve just finished compiling a program to run. Use the code found in Listing One and Listing Two as our examples. It’s a simple executable that loads a shared library and calls a function in that shared library. It then calls the sleep() system call to suspend itself for a while (so we can find out information about it while it’s running) before exiting. Compile the shared library by running:

machine:~/> gcc -fPIC -c b.c
machine:~/> ld -shared -o b.so b.o

Then compile the program with:

machine:~/> gcc main.c -ldl

Listing One: main.c — A Program That Uses a Shared Library

#include <stdio.h>
#include <dlfcn.h>

int main ()
void* handle;
void (*printer)(void);
char* error;

handle = dlopen (“/home/chelf/linuxmag/0202/b.so”, RTLD_LAZY);

error = dlerror ();
if (error)
printf (“%s\n”, error);
exit (1);

printer = dlsym (handle, “b_printer”);

error = dlerror ();
if (error)
printf (“%s\n”, error);
exit (1);

printer ();

sleep (200);

dlclose (handle);

Listing Two: b.c — A Shared Library Source Module

#include <stdio.h>

void b_printer ()
printf (“Printing something from a shared library.\n”);

This generates an executable file named a.out. So what exactly happens when you run a.out at your prompt?

machine:~/> ./a.out

After checking for shell functions or aliases (and finding none for this command), the shell calls fork() so that now two copies of the shell are running (for more on the fork() system call, see the Spring 1999 Compile Time column online at http://www.linux-mag.com/1999-05/compile_01.html). After the fork() returns, the resulting child shell process calls the exec() system call, which causes the operating system to load and run the a.out program in place of the child shell, effectively killing the shell (but not the process). The parent shell process waits for a.out to finish, then issues another prompt.

The exec() system call (short for execute), is one of many different system calls that control program execution. Each one works a little bit differently, so the calling program needs to carefully choose which one to use. The prototypes for the different versions of exec() are listed in Figure One.

Figure One: Prototypes for the ‘exec’ system call found in <unistd.h>

int execl (const char *path, const char *arg, …);
int execlp (const char *file, const char *arg, …);
int execle (const char *path, const char *arg , …, char* const envp[]);
int execv (const char *path, char *const argv[]);
int execvp (const char *file, char *const argv[]);
int execve (const char *filename, char *const argv [], char *const envp[]);

The functions that end in a “p” (execlp() and execvp()) will search your path for the given file. For example, if the program calls execlp (“gcc”, …), the system will look for the file “gcc”. For the functions that do not end in a “p”, you must specify the entire path to the program, including the filename, as the path argument.

Functions that have an “l” in their name (execl(), execlp(), and execle()) have the program’s arguments passed as arguments to the specific exec function. There can be as many arguments as you like.

Functions that have a “v” in their name (execv(), execvp(), and execve()) have the program’s arguments passed via a single pointer to an argv-like structure (a pointer to an array of char *‘s).

Functions that end in an “e” (execle() and execve()) allow you to pass an environment to the executable in addition to the program’s arguments. This lets you adjust environment variables like HOME or PATH, or add or remove any environment variables you want. The environment is always passed via a pointer to an array of char *‘s (just as the arguments are passed in the “v” functions).

What the Kernel Does

Once the child shell calls exec() to start running a.out, the kernel has to do some work before before the program can begin execution. Let’s take a closer look at the a.out program.

Figure Two shows an edited sample of the output generated by running objdump -s a.out. After printing the type of executable a.out is, it lists its various sections (that’s what the -s flag controls).

Figure Two: Output of the command objdump -s a.out

a.out: file format elf32-i386

Contents of section .text:
80484e0 31ed5e89 e183e4f8 50545268 dc860408 1.^…..PTRh….
80484f0 68048404 08515668 dc850408 e87bffff h….QVh…..{..
8048500 fff49090 5589e553 50e80000 00005b81 ….U..SP…..[.
8048510 c3661200 008b8338 00000085 c07402ff .f.....8.....t..
8048520 d08b5dfc c9c39090 90909090 90909090 ..]………….
8048530 558b155c 97040889 e583ec08 85d27549 U..\……….uI
8048540 8b155897 04088b02 85c0741a 8d742600 ..X…….t..t&.
8048550 8d4204a3 58970408 ff128b15 58970408 .B..X…….X…
8048560 8b0a85c9 75eab84c 84040885 c0741083 ….u..L…..t..
8048570 ec0c6860 970408e8 d0feffff 83c410b8 ..h`…………
8048580 01000000 a35c9704 0889ec5d c38d7600 …..\…..]..v.
8048590 5589e583 ec0889ec 5dc38db6 00000000 U…….]…….
80485a0 5589e5b8 2c840408 83ec0885 c0741583 U…,……..t..
80485b0 ec086858 98040868 60970408 e86bfeff ..hX…h`….k..
80485c0 ff83c410 89ec5dc3 908db426 00000000 ……]….&….
80485d0 5589e583 ec0889ec 5dc39090 5589e583 U…….]…U…
80485e0 ec1883ec 086a0168 20870408 e8dbfeff …..j.h …….
80485f0 ff83c410 89c08945 fce86efe ffff89c0 …….E..n…..
8048600 8945f483 7df40074 1f83ec08 ff75f468 .E..}..t…..u.h
8048610 3f870408 e873feff ff83c410 83ec0c6a ?….s………j
8048620 01e896fe ffff89f6 83ec0868 43870408 ………..hC…
8048630 ff75fce8 04feffff 83c41089 45f8e829 .u……….E..)
8048640 feffff89 c08945f4 837df400 741e83ec ……E..}..t…
8048650 08ff75f4 683f8704 08e82efe ffff83c4 ..u.h?……….
8048660 1083ec0c 6a01e851 feffff90 8b45f8ff ….j..Q…..E..
8048670 d083ec0c 68c80000 00e8defd ffff83c4 ….h………..
8048680 1083ec0c ff75fce8 20feffff 83c410c9 …..u.. …….
8048690 c3909090 90909090 90909090 90909090 …………….
80486a0 55a16497 040889e5 5383ec04 83f8ffbb U.d…..S…….
80486b0 64970408 74168d76 008dbc27 00000000 d…t..v…’….
80486c0 83eb04ff d08b0383 f8ff75f4 585b5dc3 ……….u.X[].
80486d0 5589e583 ec0889ec 5dc39090 U…….]…
Contents of section .data:
8049750 00000000 00000000 70970408 00000000 ……..p…….
Contents of section .sbss:

The exact contents within each section aren’t that important; we’re interested in the section names right now. The sections that the kernel cares most about are .text, .data, and .bss. In FigureTwo, objdump shows the .bss section as .sbss which can be considered the same as the .bss section. More on what these sections contain in a bit.

Figure Three
Figure Three: A process’s virtual address space

In the March issue (http://www.linux-mag.com/2002-03/compile_01.html), we showed how a process virtual address space is divided into various pieces, more properly called “segments.” Figure Three shows a more detailed map of the virtual address space.

Below the heap, there are three separate segments (text, data, and bss) that match the sections in a.out that objdump reported, which, as you might have guessed, isn’t a coincidence.

Copying Sections to Segments

If you were to follow the Linux kernel as it ran through one of the exec() functions, it would eventually lead to a function in the kernel called load_elf_binary() (located in the kernel source file fs/binfmt_elf.c) that copies the data from the .bss section of the executable into the bss segment in the new process’s virtual address space. It also copies the .data section into the new data segment, and the .text section into the new text segment.

You may be asking what’s in these sections and/or segments? The text segment contains the instructions to be executed, the data segment contains the static and global data that is initialized, and the bss segment contains global data that is uninitialized. “Global data” corresponds to variables or constants declared outside the scope of any function, such as the main() function. “Static data” corresponds to variables or constants that have been declared as static, even those inside functions. The compiler will automatically initialize all variables, so the bss segment is often empty.

After the segments are set up and the arguments to the program and its environment are copied into the address space, load_elf_binary() calls the start_thread() function, and the program begins executing the code in its text segment.

What About Shared Libraries?

Last month, we explained how you can dynamically load object files into your programs. But how does this work once the operating system has already set up the processes address space as described above? Each new library that is dynamically loaded gets its own text, bss, and data segments in the process address space. The operating system stacks them up as they are loaded. You can see this best by looking at a process while it’s running.

You can get a map of a running process’s virtual address space by looking in the file /proc/pid/maps, where pid is the process ID of the one we’re interested in. To get the process ID, use the ps command:

machine:~/> ps ax | grep a.out
23098 pts/5 S 0:00 ./a.out

To see the process’s virtual address map, run:

machine:~/> more /proc/23098/maps

Figure Four shows the (edited) map of the process space of the executable (this is why we added the call to sleep() in Listing One). The numbers in the first column show the virtual address ranges for each of the segments. The first two are for the text and data segments of a.out. There’s no bss segment in the process map since there was no data in the bss section of a.out.

Figure Four: A map of the address space of a.out

08048000-08049000 r-xp 00000000 /home/chelf/linuxmag/0202/a.out
08049000-0804a000 rw-p 00000000 /home/chelf/linuxmag/0202/a.out
40000000-40015000 r-xp 00000000 /lib/ld-2.2.4.so
40015000-40016000 rw-p 00014000 /lib/ld-2.2.4.so
40016000-40017000 rw-p 00000000
40017000-4001b000 r-xp 00000000 /lib/libsafe.so.1.3
4001b000-4001c000 rw-p 00003000 /lib/libsafe.so.1.3
4001c000-4001d000 r-xp 00000000 /home/chelf/linuxmag/0202/b.so
4001d000-4001e000 rw-p 00000000 /home/chelf/linuxmag/0202/b.so
4001e000-4001f000 rw-p 00000000
40025000-40027000 r-xp 00000000 /lib/libdl-2.2.4.so
40027000-40029000 rw-p 00001000 /lib/libdl-2.2.4.so
40029000-40155000 r-xp 00000000 /lib/libc-2.2.4.so
40155000-4015a000 rw-p 0012b000 /lib/libc-2.2.4.so
4015a000-4015f000 rw-p 00000000
bfffe000-c0000000 rwxp fffff000

You can identify the text segment by the “x” in the second column (it stands for “executable,” just like in the output from ls -l). The “w” means the segment is writable and identifies the data segment. Executable segments cannot be written on because Linux doesn’t allow self-modifying code. All segments must be readable (the “r” in the second column).

Below the segments for our executable are the segments for the “ld” library. In this case, “ld” doesn’t stand for the ld linker, but rather what’s called the program interpreter (yes, that’s a very poor name for it). The program interpreter analyzes the executable, figures out which shared libraries are necessary for it to run, and locates and loads them into the virtual address space for the executable. In this case, it brought in libc, libdl, and libsafe.

The libsafe library (along with libc) is always linked into executables by the ld linker. It ensures that you don’t overflow your stack or try to write past what you are currently allowed. (For more on libsafe, see http://www.gnu.org/directory/libsafe.html).

Also in the map, we can see the shared library we built, b.so. However, this library wasn’t loaded into the virtual address space by the program interpreter. That’s because b.so wasn’t specified on the command line that built a.out. Instead, it was the dl library (and the dlopen() function) that loaded b.so.

In a sense, the dl library duplicates some of what the ld program interpreter does. The difference is when these two libraries load other shared libraries into the process’s virtual address space: the program interpreter does it when the process starts up, the dl library does it any time after the process has begun executing.

But That’s Not All!

Believe it or not, there’s still a lot that hasn’t been covered. The Linux kernel (in those exec() functions) does a lot of other things to make your executables run. You can consult the kernel source file fs/exec.c or the last chapter of the O’Reilly book Understanding the Linux Kernel if you’d like to learn more.

An interesting thing you’ll find if you look in the source is that the Linux kernel is not limited to running ELF executables. You can also run executables of the old a.out format (not to be confused with the program in our example) on your system.

So hopefully you can now see that although it might not seem like it, Linux is hard at work “under the hood” to make your programs run.

In the meantime, happy hacking!

Benjamin Chelf is an author and engineer at CodeSourcery, LLC. He can be reached at chelf@codesourcery.com.

Comments are closed.