So we get into Andrew Tanenbaum's notion of "intelligent design," as applied to software. The core rules are:
Isolate components from each other so that they cannot interfere with each other - or even communicate unless there is a reason to do so.
Stick to the "principle of least authority"; no component should have more privilege than it needs to get its job done.
The failure of one component should not cause others to fail.
The health of components should be monitored; if one stops operating properly, the system should know about it.
One must be prepared to replace components in a running system.
There is a series of steps to take to apply these principles. The first is to move all loadable modules out of the kernel; these include drivers, filesystems, and more. Each should run as a separate process with limited authority. He pointed out that this is beginning to happen with Linux with the interest in user-space drivers - though it is not clear how far Linux will go in that direction.
Then it's time to isolate I/O devices. One key to reliability is to do away with memory-mapped I/O; it just brings too many race conditions and opportunities for trouble. Access to devices is through I/O ports, and that is strictly limited; device drivers can only work with the ports they have been specifically authorized to use. Finally, DMA operations should be constrained to memory areas which the driver has been authorized to access; this requires a higher level of support from the hardware, however.
The third step is minimizing privileges to the greatest extent possible. Kernel calls should be limited to those which are needed to get a job done; device drivers, for example, should not be able to create new processes. Communication between processes should be limited to those which truly need to talk to each other. And, when dealing with communications, a faulty receiver should never be able to block the sender.