Cover of The Operating System: Supervisor, Kernel, Ghost, a book by Ketineni RamaraoFrom the book The Operating System: Supervisor, Kernel, Ghost
Chapters 33 to 36. Read the full book free as a PDF.

Two worlds, one gate

The machine is divided into two kinds of code. Kernel code may do anything: touch any memory, command any device. Application code may act only by asking. The border is enforced by the processor itself, which carries privilege levels - descendants of the rings Multics designed for in the 1960s - that flip only at guarded crossing points.

Those crossings are system calls. When a program calls open("notes.txt"), the library packages the request and executes a trap instruction. The processor switches to privileged mode and jumps to a fixed, kernel-owned entry point. The kernel checks the request - is the pointer legal, does this process hold rights to this directory - performs it, and returns a file descriptor. The whole crossing takes microseconds and happens millions of times per second. The entire security model of the world's computing reduces to this: billions of lines of untrusted application code asking for services through a doorway a few hundred instructions wide.

Diagram of processor privilege rings: applications in the outer ring, the kernel in the inner ring, and the system-call gate as the only crossing between them.
Two worlds, one gate. Original IOI diagram.

The staff around the core

Around the scheduler, memory manager and file systems sits the bulk of the code by far: device drivers, the per-hardware translators. Of the Linux kernel's roughly 40 million lines, the great majority is drivers. That pattern holds in Windows, macOS and Android, and it explains a paradox: a student can write a kernel in a semester, but almost no company can make an operating system usable, because making one usable means supporting ten thousand kinds of device. It also explains the crash statistics: drivers are historically the buggiest layer, which is why systems increasingly push drivers out of the kernel so a crashed driver dies alone.

A DEC PDP-11/40 minicomputer from 1973: a rack of dark panels with rows of small coloured switches and lights, photographed against a white background.
A DEC PDP-11/40 of 1973. The ideas every modern operating system rests on - privilege rings, virtual memory, the file interface - were settled on machines of this generation. Photo: Jeff Keyzer, CC BY-SA 2.0, via Wikimedia Commons.

Whose turn it is

The scheduler answers one question - who runs next - thousands of times a second. Batch systems answered crudely: whoever is next in the queue. Time-sharing brought the round-robin and the time slice, buying the illusion of simultaneity by switching faster than a human notices. Linux's Completely Fair Scheduler of 2007 tracked each task's received service to nanoseconds; in 2023-24 the kernel moved to EEVDF, folding latency sensitivity into the same accounting.

The pathology that teaches the most is priority inversion. Give tasks strict priorities and one shared resource, and a nightmare ordering appears: a low-priority task holds the resource, a high-priority task waits on it, and unrelated medium-priority work runs instead. This is exactly what froze a bus task aboard NASA's Mars Pathfinder in July 1997; the watchdog fired and the spacecraft reset itself, again and again, until engineers on Earth read the logs and switched on the ready-made cure - priority inheritance, which lends the blocked task's priority to the blocker. One scheduling-policy change, uploaded across 190 million kilometres, saved the mission.

Mobile and the datacenter each added an axis. A phone's scheduler must cooperate with the power manager - a frame every 16.7 milliseconds or the user sees a stutter - and with battery life. A hyperscaler wants every expensive core busy, which is why Google built Borg and then Kubernetes: a scheduler above the scheduler, allocating whole machines to workloads the way the kernel allocates milliseconds to threads. At every scale, someone is running the same 1961 loop: who is waiting, who is urgent, whose turn is next.

The memory illusion

Virtual memory is the operating system's best con: every program gets a private, vast, contiguous address space that does not exist. Memory is cut into pages; each program's addresses are translated through tables; pages not in RAM trigger a page fault, and the kernel quietly fetches them from disk. The idea belongs to the Atlas team in Manchester, who built the first paged store in 1962 because fast memory was small and drums were slow; Peter Denning's 1968 working-set model explained why the con pays - programs touch only a small, shifting set of pages at any moment. When the working set outgrows the machine, the system thrashes.

Diagram of virtual memory: each process has a private address space mapped through a page table to scattered pages in RAM, with cold pages parked on disk.
Virtual memory: every program gets a private machine that does not exist. Original IOI diagram.

Side effects reshaped computing: private address spaces mean memory protection arrives as a by-product, and kernels can overcommit, promising more memory than exists on the actuarial observation that not every program draws at once. Phones keep the illusion but change the policy - Android compresses pages in RAM instead of swapping, and both mobile platforms kill background apps under pressure. That is why an app you left an hour ago sometimes returns to your exact place and sometimes starts over.

One interface for everything

Unix's famous slogan - everything is a file - is an engineering economy: disks, terminals, processes and network connections all answer to the same handful of operations, so one interface, learned once, covers the machine, and small programs chain together through pipes. Underneath, file systems are among the hardest software humans build, because they must keep promises across power cuts. The industry's answer is the journal and the copy-on-write design, and modern systems such as ZFS and Apple's APFS add checksums over everything, because disks occasionally lie. The file system's success is measured in losses that do not happen.

Quick answers

What is a system call?

A system call is the guarded doorway between an application and the kernel. The program executes a special instruction, the processor switches to privileged mode, the kernel validates and performs the request - open a file, send a packet - and returns.

What is virtual memory, in simple terms?

Each program is given a private, huge address space that does not physically exist. The kernel maps pieces of it to real memory as needed and quietly fetches missing pages from disk, so programs can be bigger than the machine's RAM.

Why do device drivers cause so many crashes?

Drivers are the largest and least reviewed part of any kernel - written by hardware vendors to ship on schedule. Modern systems increasingly run drivers outside the kernel so a crashed driver restarts without taking the machine down.

Sources

The Linux kernel size (roughly 40 million lines, mostly drivers) is from Phoronix's Git analysis of 2024; totals vary slightly by counting method. Scheduler and memory mechanisms follow the OSTEP textbook; the Pathfinder account follows Mike Jones's RISKS Digest post recounting David Wilner's account.

More from this book

Definition · From the book The Operating System: Supervisor, Kernel, Ghost

What is an operating system?

Every tap on a screen ends the same way: a program asks the operating system to do something it is not allowed to do itself.

Systems · From the book The Operating System: Supervisor, Kernel, Ghost

Real-time operating systems: the software that cannot be late

An anti-lock brake controller that is a little late is not slow; it is wrong.

History · From the book The Operating System: Supervisor, Kernel, Ghost

The history of Unix: from a spare PDP-7 to everywhere

The most consequential operating system ever written started as a way to play a game cheaply.