Hot Buttered Kernels!

Hot Butter – Popcorn (Version Originale) 1972 full 7” Single

In every life, and every generation, there are events and moments which represent something that cannot be fully explained or communicated to people who did not experience them.

IYKYK, right?

A good example. While I understand the reference, and can appreciate and use it, the usage is not really part of my generation’s lexicon, so I can’t really understand it in the way that people who lived it do.

Show someone an Etch A Sketch, and they’ll likely experience bafflement, recognition, or a flood of memories about these devices and how amazing they seemed in the 1960’s and 1970’s.

It applies to sound as well. For those old enough to remember, the first experience many people had of hearing “electronic music” was the song “Popcorn”, performed by Hot Butter (Yes. Really), in 1972. Though the first commercial synthesizer, the Moog, was invented in 1964, and there was a lot of work being done by pioneers in electronic music – notably the 1968 album Switched-On Bach, by Wendy Carlos - Popcorn was one of the first major hits that would be familiar to most people - the kernel of their experience with electronic music.

Which leads, of course, to the core of computer operating systems, the kernel.

The startup process for a computer is quite complex and can be confusing, but I think much of this is due to overlapping names and definitions, depending on the type of computer, the manufacturer, the operating system used, and various other factors.

For (relative) simplicity, I find it useful to break the process into three parts- hardware, firmware, and software.

Hardware includes the processor, memory (RAM), and peripherals (such as disk, keyboard, monitor and mouse). Software consists of the instructions which are loaded into memory and executed by the processor.

Firmware sits in a grey area between hardware and software. It originally referred to the “microcode” involved in implementing machine instructions on the CPU, but now generally refers to any programs or code which are tightly linked to hardware.

Under normal operation, the computer “runs” because the processor (hardware) loads and executes instructions from memory (hardware), and then performs whatever actions are desired.

On startup, though, the processor doesn’t “know” anything about memory. All it has to go on is a single, hard-coded instruction: “Load instructions from Boot ROM”.

The boot ROM is part of the firmware of the system, generally located on the same chip as the processor, or on a separate chip at a specific location. In either case, the processor loads and executes these instructions, which are usually to load the bootloader from firmware.

The bootloader is usually referred to as “BIOS” (Basic Input Output System), even though BIOS was replaced by “UEFI” (Unified Extensible Firmware Interface) years ago. Whatever it’s called, the bootloader begins by running a power-on self-test (aka “POST”), which detects and tests attached hardware. Once the POST is complete, the processor is fully configured, memory is configured and tested, and connected hardware is identified and configured.

So far, the instructions have come from the system’s firmware, but now the system’s disk and memory are available. The “second stage bootloader” is loaded from the master boot record (not to be confused with the musician), and proceeds to locate the operating system kernel code, load the kernel into memory, and start it running.

The kernel checks the hardware, sets up device drivers and other kernel modules, and loads the operating system. It is the part of the operating system that is always loaded into memory, and includes those programs which directly interact with the processor, memory, disk, and other hardware. It provides the ability for the computer to support multiple users and processes securely and without interference by separating “kernel mode” from “user mode”.

Applications in user mode cannot directly access hardware. Instead, they submit requests to the kernel, which checks the requests and then executes them. If there is something wrong with the requests, or the user is not authorized to perform the actions requested, the kernel stops them. If the user mode application fails, it will not generally affect the kernel itself.

So, when a computer “freezes” or “crashes”, that means that something happened at the kernel level, which is bad. It means that a flaw or vulnerability is present deep in the “guts” of the operating system or, more frequently, an application has been given some degree of access to kernel mode and then failed, taking the kernel down with it.

The simplistic solution is to not allow any application to use kernel mode.

In practice, though, the ability to directly access the hardware makes it possible for an application to do things that are impossible (or much harder) in user mode. Optimizing performance for games is one example, which is why many games used to crash the entire system (particularly in earlier versions of Windows).

Operating system developers need to balance the stability of the system with the demand for access from application developers. Windows has traditionally tended toward allowing kernel mode more often than MacOS or Linux, but the balance has been tending more and more toward forbidding kernel access whenever possible.

Widespread IT outages might have an impact on this trend. As you may recall, there was a widespread outage of Microsoft Windows systems in July of 2024, affecting an estimated 8.5 million devices. While this represents “only” about 1% of all Windows devices, the impact was immense, as the affected devices used CrowdStrike’s commercial security software – meaning that the impact was mainly on larger commercial interests, including almost 60% of the Fortune 500 and more than half of the Fortune 1000. The estimated cost to the UK economy was estimated at between £1.7 and £2.3 billion ($2.18 and $2.96 billion USD), while financial losses for the top 500 US companies (excluding Microsoft) was nearly $5.4 billion (USD)

This was bad.

One reason it was bad was that CrowdStrike’s “Falcon” sensor was running in kernel mode, so the failing application ended up taking Windows down with it, and systems got stuck in a “reboot loop”, or booted into “recovery mode”.

To be fair, there were a number of problems, including the fact that the CrowdStrike update was not tested sufficiently before wide-scale release, but one of the major issues raised was the use of kernel mode.

On the one hand, the impact of the issue would have been dramatically less if the application did not have access to the kernel, but not having that access would also have significantly reduced the effectiveness of the security tool’s scanning ability.

The excuse reason Microsoft gave for not blocking kernel access for security tool vendors was a 2009 antitrust agreement with the European Union. Interestingly, though, the agreement does not state that Microsoft had to provide kernel access to security tool vendors. Instead, the agreement stated that Microsoft needed to provide security tool vendors with access to the same APIs (Application Programming Interface) used by its own security products.

From this, it’s easy to infer that Microsoft’s security products have access to the kernel, and Microsoft was not previously interested in doing the work which would be needed to “move” their own product out of the kernel, along with other vendors.

This appears to have changed, though. After the CrowdStrike debacle incident, Microsoft opened discussions with vendors to discuss the issue, and recently (a year after the original incident) announced the upcoming release of a private preview of Windows changes which move antivirus (AV) and endpoint detection and response (EDR) apps out of the Windows kernel.

I actually find the slowness of this process encouraging, as it seems clear that Microsoft is serious about finding the best way to manage kernel access, rather than simply blocking it. I expect it will be a long process, and will have some challenges, but this seems like a good strategy.

Or at least the kernel of one.

Cheers!