In his book Security Engineering, Anderson really focuses on how in the 90s and early 2000s programs would need to access memory that wasn't their own, and programmers programmed with the assumption the program would be run with administrative privileges:


  The latest Microsoft system, Vista, is trying to move away from
  running all code with administrator privilege, and these three modern
  techniques are each, independently, trying to achieve the same thing —
  to get us back where we'd be if all applications had to run with user
  privileges rather than as the administrator. That is more or less
  where computing was in the 1970s, when people ran their code as
  unprivileged processes on time-shared minicomputers and mainframes.
  Only time will tell whether we can recapture the lost Eden of order
  and control ...
  , and to escape the messy reality of today... but certainly the attempt is worth making.


In particular this quote


  PCs carry an unfortunate legacy, in that the old single-user operating
   systems such as DOS and Win95/98 let any process modify any data; as a result many applications won’t run unless they
  are run with administrator privileges and thus given access to the
  whole machine


Does this mean that ideas such as these, where you can assign any address to a pointer and have access to the memory would work on Windows 95/98? Why would they design it this way as what would be the benefit of reading some other programs memory (or writing to it!)?

int * firefoxmemory = (char*) 0x11111111 //this is just an example of address.
*firefoxmemory = 200;//screw firefox


Memory isolation

Your example wouldn't work on Windows 95, but it did work on DOS and Windows up to 3.11 (not Windows NT).

The PC architecture, and the Microsoft series of operating systems, started with the Intel 8086 processor and an operating system (DOS) designed to run a single program at a time. You would run a program, and when you were finished with it, you'd exit it, so the overwriting data of another process wasn't really a concern. (You could have programs that remained in memory after exiting; they were essentially part of the operating system.)

The Intel 80386 processor changed the deal by being the first of its lineage to have a memory management unit (MMU). An MMU is a hardware component that provides virtual memory by translating virtual addresses (pointers in a program) to physical addresses (actual locations in RAM¹). Windows 95 was the first in its lineage to take advantage of it: each Windows 95 application ran with its own MMU configuration, so that *firefoxmemory would either point into the program's own address space (in which case whatever bad thing it did could only affect that program) or not point anywhere at all (in which case the program — and not the whole system — would crash).

Windows 3.0–3.11 took advantage of some of the 386's novelties, but they ran all applications in the same address space. This was due to several factors:


a requirement to keep old Windows 1.x/2.x applications running (they were designed to run on single-address-space hardware, and many took advantage of that to poke into OS internals to achieve things that weren't possible through documented interfaces);
lack of development time to redesign the whole operating system based on a completely different architecture;
a requirement to run on computers that had little RAM: keeping applications isolated does cost more memory on a scale of 2 MB or 4 MB (because data has to be copied between programs, and because the operating system has to keep track of all program interactions to allow applications to communicate and to regulate the communication).


It wasn't so much that the operating system was explicitly designed to allow programs to access each other's memory, but that there was no way to prevent it. When ways to prevent it became affordable, it took a few OS versions to take advantage of them.

For example, let's consider a feature such as inter-program copy-paste (clipboard). If you don't have memory isolation, this can be implemented by having the source program keep a copy of the data in its own memory; when the data is pasted in another program, that other program copies the data directly from the source program. The only thing the operating system needs to track is who currently owns the clipboard. If applications' memory is isolated, the operating system needs to arrange for the data copying, and possibly to store a copy of the data independently of the source application. This requires more development work to write this code, and more resources to store this code in memory and run it.

I've simplified a number of things here; for more about how an operating system uses an MMU to isolate applications, you can read How the kernel can prevent a malicious program to operate?, How can two identical virtual addresses point to different physical addresses?, Is it possible to support multiple processes without support for Virtual memory? (they're Unix-oriented but the principles apply to Windows or any OS you're likely to encounter on a PC).

Privilege isolation

In this passage, Anderson isn't actually discussion memory isolation, but privilege isolation, where a running program is prevented from affecting certain parts of the system. Memory isolation is necessary, but not sufficient. The operating system must also control the ways processes interact, control what files they open, etc. Even with each application running in its own memory space, applications can interact. Memory isolation merely forces applications to use operating system services (via system calls to a kernel which can access all memory) to interact.

The Windows 1.x/3.x/9x/ME series of operating system was designed as a single-user operating system and did not isolate applications. Memory isolation was added in Windows 95, but only to improve stability, not to implement security restrictions. These operating systems have no security restrictions: if you ran a program on your machine, it is allowed to do everything. (This is of course helpful to virus writers.) The memory space is isolated, but not the file space.

The Windows NT/2000/XP/… series of operating systems, like operating systems in the Unix family, was designed as a multi-user operating system, which introduces a basic security goal: what a user does should not adversely affect what other users can do. So the operating system has to enforce, for example, that programs executed by Alice cannot interact with programs executed by Bob, cannot modify Bob's files, etc. Programs and data that are part of the operating system concern every user and so should be protected from all users. (Of course, some interactions are allowed by explicit permission, e.g. a network server does process requests that it receives from the network, Bob can change a file's permissions to be writable by others, some users are granted administrative privileges and so can modify the operating system, etc.)

A great many applications were written to target Windows 9x, which had no security restrictions. So they took liberties such as writing to the operating system directory (under Windows 95, copying the file to the Windows directory was the normal way to install a shared library). Much of what these applications do was considered somewhat messy even by the standards of that time, but it worked so people did it. Versions of Windows from the NT series (which includes all versions since XP) necessarily broke some of these applications, though Microsoft did add a number of workarounds (such as pretending that writing a file to a shared directory succeeded, but actually writing it to a user-specific directory).

Windows XP enforced user isolation, but most installations wouldn't take advantage of it: many installations just had the user be an administrator, so all the programs they ran had the privileges to do pretty much anything. There were two main reasons for that:


It allows badly-behaved legacy applications to run. While most such applications are only used by a small proportion of users, many users do run such applications and it takes a long time and a huge effort to renew them all.
Introducing privileges means that sometimes the user will be told that something can't be done because of a lack of privileges. Witness the complaints about Vista, that it would keep showing those privilege escalation prompts.


Application isolation

If you're young enough to have discovered computers through portable devices, you may be used to a model of isolation centered on applications, rather than users. In the Unix and Windows model, Alice's data is protected from Bob. Applications are neutral in the security model: they just execute under the identity of the logged-in user. This means that the application code has to be trusted not to misbehave.

Windows has traditionally coped with misbehaving applications by trying to detect and contain them through antivirus software. That doesn't work too well. Unix systems, especially Linux, tend to cope by distributing more software through traceable channels from which malware is eliminated.

Operating systems designed for mobile devices, in particular iOS and Android, don't trust application developers, so applications are isolated; for example each application has its own file space. This has a security benefit, but also a significant cost in that it reduces what applications can do. Applications require privileges to modify operating system behavior (e.g. to create shortcuts and more generally automation), you can't easily manipulate the same file in different applications, you can't easily allow multiple applications to display information at the same time, etc. Or, to take a more dramatic example, you can't write a debugger with restricted privileges, because the whole point of a debugger is to snoop on and perturb the execution of the application that's being debugged. This cost is why the mobile application isolation model can't just be grafted onto a PC environment.
