Mechanism · Information Technology · Software
How operating systems work: kernels, schedulers and virtual memory
Under seventy years of marketing, an operating system rests on four ideas settled early and never replaced: two worlds of privilege, a referee called thousands of times a second, a memory illusion, and one interface for everything. This is what each of them does.
From the book The Operating System: Supervisor, Kernel, GhostChapters 33 to 36. Read the full book free as a PDF.
Two worlds, one gate
The machine is divided into two kinds of code. Kernel code may do anything: touch any memory, command any device. Application code may act only by asking. The border is enforced by the processor itself, which carries privilege levels - descendants of the rings Multics designed for in the 1960s - that flip only at guarded crossing points.
Those crossings are system calls. When a program calls open("notes.txt"), the library packages the request and executes a trap instruction. The processor switches to privileged mode and jumps to a fixed, kernel-owned entry point. The kernel checks the request - is the pointer legal, does this process hold rights to this directory - performs it, and returns a file descriptor. The whole crossing takes microseconds and happens millions of times per second. The entire security model of the world's computing reduces to this: billions of lines of untrusted application code asking for services through a doorway a few hundred instructions wide.
The staff around the core
Around the scheduler, memory manager and file systems sits the bulk of the code by far: device drivers, the per-hardware translators. Of the Linux kernel's roughly 40 million lines, the great majority is drivers. That pattern holds in Windows, macOS and Android, and it explains a paradox: a student can write a kernel in a semester, but almost no company can make an operating system usable, because making one usable means supporting ten thousand kinds of device. It also explains the crash statistics: drivers are historically the buggiest layer, which is why systems increasingly push drivers out of the kernel so a crashed driver dies alone.

Whose turn it is
The scheduler answers one question - who runs next - thousands of times a second. Batch systems answered crudely: whoever is next in the queue. Time-sharing brought the round-robin and the time slice, buying the illusion of simultaneity by switching faster than a human notices. Linux's Completely Fair Scheduler of 2007 tracked each task's received service to nanoseconds; in 2023-24 the kernel moved to EEVDF, folding latency sensitivity into the same accounting.
The pathology that teaches the most is priority inversion. Give tasks strict priorities and one shared resource, and a nightmare ordering appears: a low-priority task holds the resource, a high-priority task waits on it, and unrelated medium-priority work runs instead. This is exactly what froze a bus task aboard NASA's Mars Pathfinder in July 1997; the watchdog fired and the spacecraft reset itself, again and again, until engineers on Earth read the logs and switched on the ready-made cure - priority inheritance, which lends the blocked task's priority to the blocker. One scheduling-policy change, uploaded across 190 million kilometres, saved the mission.
Mobile and the datacenter each added an axis. A phone's scheduler must cooperate with the power manager - a frame every 16.7 milliseconds or the user sees a stutter - and with battery life. A hyperscaler wants every expensive core busy, which is why Google built Borg and then Kubernetes: a scheduler above the scheduler, allocating whole machines to workloads the way the kernel allocates milliseconds to threads. At every scale, someone is running the same 1961 loop: who is waiting, who is urgent, whose turn is next.
The memory illusion
Virtual memory is the operating system's best con: every program gets a private, vast, contiguous address space that does not exist. Memory is cut into pages; each program's addresses are translated through tables; pages not in RAM trigger a page fault, and the kernel quietly fetches them from disk. The idea belongs to the Atlas team in Manchester, who built the first paged store in 1962 because fast memory was small and drums were slow; Peter Denning's 1968 working-set model explained why the con pays - programs touch only a small, shifting set of pages at any moment. When the working set outgrows the machine, the system thrashes.
Side effects reshaped computing: private address spaces mean memory protection arrives as a by-product, and kernels can overcommit, promising more memory than exists on the actuarial observation that not every program draws at once. Phones keep the illusion but change the policy - Android compresses pages in RAM instead of swapping, and both mobile platforms kill background apps under pressure. That is why an app you left an hour ago sometimes returns to your exact place and sometimes starts over.
One interface for everything
Unix's famous slogan - everything is a file - is an engineering economy: disks, terminals, processes and network connections all answer to the same handful of operations, so one interface, learned once, covers the machine, and small programs chain together through pipes. Underneath, file systems are among the hardest software humans build, because they must keep promises across power cuts. The industry's answer is the journal and the copy-on-write design, and modern systems such as ZFS and Apple's APFS add checksums over everything, because disks occasionally lie. The file system's success is measured in losses that do not happen.
Quick answers
What is a system call?
A system call is the guarded doorway between an application and the kernel. The program executes a special instruction, the processor switches to privileged mode, the kernel validates and performs the request - open a file, send a packet - and returns.
What is virtual memory, in simple terms?
Each program is given a private, huge address space that does not physically exist. The kernel maps pieces of it to real memory as needed and quietly fetches missing pages from disk, so programs can be bigger than the machine's RAM.
Why do device drivers cause so many crashes?
Drivers are the largest and least reviewed part of any kernel - written by hardware vendors to ship on schedule. Modern systems increasingly run drivers outside the kernel so a crashed driver restarts without taking the machine down.
Sources
- Arpaci-Dusseau and Arpaci-Dusseau, "Operating Systems: Three Easy Pieces", free online edition
- D. M. Ritchie and K. Thompson, "The UNIX Time-Sharing System", Communications of the ACM 17(7), July 1974
- Mike Jones (recounting David Wilner's account), "What really happened on Mars Rover Pathfinder", RISKS Digest 19(49), December 1997
- Phoronix, Michael Larabel, "Linux Git Stats for 2024"
