How To Make Sure Your Driver Will Work On The Power Macintosh

There is an increasing number of people running Linux on their Power Macintoshes. One of the advantages of running Linux (as opposed to other variants of UNIX) on a Power Mac is that the Linux source includes drivers for a large number of PCI cards. All of the current Power Macs use the PCI bus as their internal I/O bus and many have PCI expansion card slots, allowing users to install any of the wide range of PCI expansion cards which are available today. The drivers in the Linux source will usually work on a Power Mac, but often only after some tweaking. The problem is usually that the driver author has made some assumptions that are true on an Intel PC but not on other machines. In this column, I will discusses some of those assumptions and how driver authors can make sure that their drivers will work not only on an Intel PC, but also on the Power Mac and other machines.

There is an increasing number of people running
Linux on their Power Macintoshes. One of the advantages of running Linux
(as opposed to other variants of UNIX) on a Power Mac is that the Linux
source includes drivers for a large number of PCI cards. All of the
current Power Macs use the PCI bus as their internal I/O bus and many
have PCI expansion card slots, allowing users to install any of the wide
range of PCI expansion cards which are available today. The drivers in
the Linux source will usually work on a Power Mac, but often only after
some tweaking. The problem is usually that the driver author has made
some assumptions that are true on an Intel PC but not on other machines.
In this column, I will discusses some of those assumptions and how
driver authors can make sure that their drivers will work not only on an
Intel PC, but also on the Power Mac and other machines.








Gearheads 1
Figure 1: The block diagram of a typical
Power Macintosh.

Architecture

Figure 1 shows the block diagram of a typical Power Macintosh. There
are some similarities with Intel PCs (such as the fact that they both
have a PCI bus) and some differences, the most obvious of which is that
they use different CPUs — Power Macs use a PowerPC processor while
Intel PCs use x86-based processors. Another difference is that the Power
Mac does not have an ISA bus.

What difference does it make to the driver author that the CPU is a
PowerPC? First, the PowerPC runs natively big-endian rather than
little-endian. (The PowerPC can run in little-endian mode, but with
possibly reduced functionality or performance. The Linux kernel on the
PowerPC runs in big-endian mode.)

Secondly, all I/O on the Power Mac is memory-mapped. That is, I/O
devices are accessed using normal load and store instructions rather
than special I/O instructions. The PowerPC architecture, like all other
modern architectures, allows memory accesses to be reordered for
increased performance. For example, if a program stores one value to
memory and then loads another, a PowerPC processor is allowed to perform
the load before the store, because getting the value from the load is
more critical for making forward progress with the execution of the
program. (Of course, if the store and load were to the same memory
location, the load is not permitted to return the old value in the
memory location. It must appear to the program as if the loads and
stores in the program are done in the order they appear.)

With normal memory, it doesn’t matter which order loads and stores
are performed in, but it certainly does matter for most I/O devices. For
this reason, the PowerPC architecture defines the “EIEIO” instruction –
Enforce In-order Execution of I/O (with apologies to Old MacDonald –
ed.) The EIEIO instruction effectively inserts a barrier into the stream
of memory accesses and prevents memory accesses from being reordered
across that barrier.

Usually a device driver will use an I/O macro such as
inb,outb, readl, or writel to access
a device’s registers. On the PowerPC, these macros are defined to do the
access with a load or store, followed by the EIEIO instruction.
Therefore, the programmer normally doesn’t need to consider the
possibility that the CPU may reorder memory accesses. If your device has
memory or registers which are accessed directly (e.g. by dereferencing a
pointer), you will need to be aware of this possibility. You may need to
insert memory barriers at the appropriate points using the mb()
or wmb() macro.

Along with not having an ISA bus, the Power Mac architecture doesn’t
include the Intel-style DMA controller that Intel PCs have. The I/O ASIC
in Figure 1 includes a DBDMA (Descriptor-Based DMA) engine for the
devices in the ASIC. PCI devices which are capable of bus mastering will
have their own DMA engine which is programmed using the device’s
registers.

Accessing PCI Address Spaces

The PCI standard defines three address spaces for the PCI bus:
memory, I/O, and configuration. PCI memory space is accessed using the
read/write* family of macros. To get the address to pass to
read/write*, you will need to look at the appropriate base
address register in your device’s configuration space to get the
physical address of your device and then pass that to ioremap()
to convert it to a virtual address that you can use with
read/write*, as shown in the following code example:


struct pci_dev *pdev;
unsigned long base_phys;
char *base;
int reg_value;

pdev = pci_find_class(MY_DEVICE_CLASS, NULL);
base_phys = pdev->base_address[0];
base = (char *) ioremap(base_phys,
MY_DEVICE_SIZE);
reg_value = readl(base + MY_REGISTER_OFFSET);

The in/out* macros take a port number which is an address
in PCI I/O space. Most systems only have one PCI host bridge and one PCI
bus (or set of buses interconnected with PCI-PCI bridges). On these
systems there is no ambiguity. However, a few systems have two or more
PCI host bridges, and thus two or more completely separate PCI buses,
each with their own distinct I/O space. On these systems, there is,
unfortunately, no easy way to access the I/O space of the second or
subsequent buses. The in/out*macros will access the I/O space
of the primary PCI bus only.

PCI configuration space is accessed using the pci_
read/write_config_*
(or pcibios_read/write_ config_*)
functions, just the same as on other platforms. (There is no problem
with multiple PCI host bridges with these functions.)

An important point to note is that it is an error to access a
location in I/O or memory space where there is no device to respond. If
you do, it causes a ‘machine check’ exception. The machine check
exception is an imprecise nonrecoverable exception, which means that the
processor doesn’t guarantee to give you a coherent set of register
values after the exception. (Most PowerPC exceptions are precise,
recoverable exceptions, which means that the register state is
consistent with all the instructions before a certain point having been
completed, and none of the instructions after that point having been
started. Being able to do this when you discover an exception condition
in the middle of executing several instructions atonce, possiblyout of
order, is actually a rather clever trick. Because a memory access can take an
arbitrary length of time before it returns the error ack signal
which causes the machine check, requiring the machine check to be a
precise recoverable exception would reduce performance.)

Therefore, on the Power Mac, it is necessary to disable any code
which probes for the presence of a device by accessing random I/O ports,
where there may or may not be a device present. Usually it is only necessary to probe like this for
ISA devices, since PCI devices have the configuration space registers
which tell you where the device’s registers are. So, it should not be
necessary to probe for devices on a Power Mac. Simply enclosing the
probing code in an #ifndef__powerpc__#endif block
would be a reasonable way to disable the probing code on the Power Mac,
for now at least. I would like to see a CONFIG_ISA symbol
defined on those systems which have an ISA bus, which can be used to
enable probing code.

Similarly, it shouldn’t be necessary to probe for IRQs on the Power
Mac. The best thing is to use the irq field of the pci_dev
struct
for the device, rather than reading the
PCI_INTERRUPT_LINE register in configuration space, as the
latter is sometimes wrong. Another point is that your driver shouldn’t
assume the system only has 16 IRQs –most Power Macs have 32 or 64, and
some have more. (Note that the 16 IRQ line assumption is wrong on modern
Intel PCs also.)

Endianness

Probably the most obvious issue in porting a driver is endianness.
The PowerPC CPU runs natively in big-endian mode, whereas the PCI bus is
little-endian. That is, when an integer is stored in memory, the PowerPC
stores the most significant byte at the lowest address, while integers
are transferred across the PCI bus with the least significant byte at
the lowest address. Thus the processor and the PCI bus agree on which
byte is at the lowest address, but disagree on whether it is the most
significant or least significant.

Fortunately, the PowerPC instruction set includes instructions to
load and store 2 and 4 byte integers in byte-reversed order
(little-endian, when the CPU is running in big-endian mode). Since the
I/O bus is a PCI bus for almost all PowerPC machines, the PPC
machine-dependent include files define the in*, out*,
read* and write* families of macros to use these
instructions.

Note that byte-swapping is only appropriate when reading or writing
a value which is considered as a 16-bit or 32-bit quantity. If the data
is a buffer, which is really an array of bytes, then you should not do
byte-swapping, because you want the byte at the lowest address to stay
at the lowest address. Sometimes, drivers use the ins* or
outs* macros to transfer a whole bufferful of data. On the
PowerPC, this ends up with the data in the wrong order. The PPC include
files define a family of ins*_ns and outs*_ns macros
which don’t do the byte-swapping, which you should use for transferring
data buffers. Unfortunately, the i386 include files don’t define these
macros yet (hopefully they will soon, as just the same as ins*
and outs*).

So, the endianness difference is catered to by the
in/out/read/write* macros when the CPU is transferring data to
or from a PCI device. What about data that is transferred using DMA? In
that case, the CPU is not involved and the data is not byte-swapped.
This is correct for data which is just an array of bytes. However, if
the data is interpreted by the device as 2 or 4-byte integers (or
addresses), then the CPU will need to byte-swap the data explicitly.

One example of this is in the de4x5.c driver for the DC21x4x family
of Ethernet chips. The chip uses two linked lists of descriptors (one
each for transmit and receive), each of which is a 16-byte structure
containing a 32-bit status word, two buffer pointers (physical
addresses), and a pointer to the next descriptor (again a physical
address). Each of these is a 4-byte quantity and needs to be
byte-swapped in memory from the CPU’s perspective so that the device
sees the intended values.




Figure 2


dmabuf->pointer = cpu_to_le32(virt_to_bus (buffer_address));
dmabuf->count = cpu_to_le32(buffer_count);
start_and_wait_for_dma(dmabuf);
status = le32_to_cpu(dmabuf->status);



This is where the cpu_to_le*() and le*_to_cpu()
macros are handy. They should be used when accessing values in
memory which are going to be accessed by a PCI device using DMA, as in
Figure 2.

These macros are defined on all platforms, and on little-endian
platforms they do nothing. It doesn’t take too much effort to include
them at the appropriate places, and it helps make your driver more
widely useful, without flooding the code with a lot of #ifdefs or
slowing it down unnecessarily.

DMA and Cache Coherence

On Power Macs, the hardware maintains coherence between the CPU data
cache and the PCI bus, just as it does on Intel PCs. The hardware
doesn’t necessarily maintain coherence between the instruction and data
caches in PowerPC systems, but that shouldn’t normally be of concern in
a device driver.

Open Firmware

Open Firmware is the ROM-based software in Power Macs which is
responsible for initializing the hardware, building the device tree, and
booting the operating system. The device tree is a data structure that
describes all of the buses and devices in the system. Each device has a
node which has a list of properties attached. Each property has a name
and a value, which is a string of bytes.

Normally, drivers for PCI devices don’t need to look at the device
tree nod, because they can usually get all the necessary information
from the device’s PCI configuration space registers. Sometimes, though,
the manufacturer of the device will include code in ROM on the device
which Open Firmware will use in constructing the de-vice’s node in the
device tree. This code can define additional properties which can be
useful to the de-vice driver. An example is that a video card might have
a property indicating the frequency of the main clock input to the video
chip.

One point to note is that Open Firmware probes each PCI device but
doesn’t necessarily turn on the bits which enable it to respond to
memory and I/O space accesses. Your driver may need to check whether
these bits are set (the PCI_COMMAND_IO and
PCI_COMMAND_MEMORY bits in the PCI_COMMAND register)
and set them if they are not. (Possibly the PCI framework code will
enable these bits automatically in future.)

Things to Remember

So there you have it. By keeping these main points in mind, it is
possible to write a PCI device driver that is useful not just on Intel
PCs, but on Power Macs and other machines.




Paul Mackerras is the leader of a research group at the Australian
National University. He has been hacking the Linux kernel since
mid-1996, when he started porting Linux to the Power Mac. He can be
reached at paulus@cs.anu.edu.au.

Fatal error: Call to undefined function aa_author_bios() in /opt/apache/dms/b2b/linux-mag.com/site/www/htdocs/wp-content/themes/linuxmag/single.php on line 62