Building The Web: A History of Web Architecture

tl;dr I’m writing a book about the history of web architecture. The audience will be engineers and anyone who wants to deep dive on how architectures have evolved over time. It will focus on the practitioners who made significant contributions to foundational ideas and technologies in web architecture, whether they intended to or not. Updates will be published at buildingtheweb.dev.

The years spanning from 1993 to 2015 were extraordinary. Linux and FreeBSD became mainstream server operating systems, the NCSA httpd project was released, it begat the Apache HTTP server, which in turn inspired projects like nginx. The world saw the birth of a global network for information exchange and its evolution into the substrate of our everyday lives. We went from editing files and placing them in directories on a server to shipping fully containerized applications through complex pipelines. Client-server architecture became three-tiered architecture, eventually morphing into the microservice death star diagrams shared in conference talks as humble brags about the amount of infrastructure required to run large sites. Most of this innovation was done in the open, often by people collaborating across organizational and geographic boundaries. I owe my career to this time period and to the people who shaped it. The amount of knowledge sharing and learning, between practitioners figuring things out the hard way, that has happened during this time period is staggering.

I’m writing a book detailing the major shifts in web architecture that happened during this time period. Things continued to happen, of course, and 2015 is admittedly a bit arbitrary - but I chose this timeline because we had containerization and orchestration cemented. Much work that has happened since has been focused on ML / AI, which is out of scope for this project. I’m writing the book because it’s important to illustrate the connective tissue between these developments. It’s also important to appreciate how these contributions were made by practitioners working in specific contexts under specific constraints, and how their innovations compounded - sometimes unexpectedly. I want to dive into how the industry evolved from stateless web servers serving static documents through the C10K problem, through developments in caching reverse proxies, improvements to OS kernels, experiments with new programming models, infrastructure like CDNs, and beyond. I’ll treat security as a common thread that runs through each chapter, with each innovation addressing some challenges and introducing new attack surface area for others.

I’m going to publish incrementally, online at buildingtheweb.dev. I’ll make each chapter available as a PDF as well as on the website. I want this to be an open process, so I may look into ways to make early versions available (probably a public git repo).

I am grateful for the support of a few trusted people in my network for reviews and connecting me to people to talk to for this project. If you or anyone you know was involved in any part of this story and would be open to speaking with me, I’d love to chat and get your perspective - or if you want to support this project in any way, don’t hesitate to reach out!

What’s Covered

The exact table of contents is a work in progress, but the progression will be roughly chronological, with each chapter covering a major architectural shift. It’s important to note that work was going on in parallel in many of these areas. I will try to draw connections where possible. Here’s the current layout:

Prologue - The Network: Then and Now - Networking was very different in 1993. This short prologue will go into some detail about the physical constraints that would have shaped architectural decisions. The evolution from dial-up to broadband to always-on mobile devices had a huge impact on the industry. The evolution of networking infrastructure, from the routing protocols that held the internet together to the physical links that determined what was possible paved the way for many of the innovations discussed in later chapters.

The Early Web - Starting with the design of HTTP and the initial RFCs. The development of the first web server and what the early web looked like as a network of servers hosting static HTML documents. The story of the early web is well told, but I want to go deeper into the specific architectural trade-offs that were baked in at this stage.

The Dynamic Web - At some point, static web documents were joined by dynamic applications with the development of protocols like CGI. Languages like Perl and PHP became ubiquitous on the internet, sometimes by accident. This was when the web stopped being a library and became a platform for application development. I want to dive into how HTTP servers handled this, from preforking and worker models, to protocols like FastCGI, to evented frameworks like Twisted, EventMachine, and eventually node.js. In parallel, browsers became a platform in their own right. JavaScript support shipped in Netscape Navigator in 1995. Microsoft shipped an at-the-time obscure ActiveX object called XMLHttpRequest in 1999. It took a while to catch on, but by 2005 the term AJAX (Asynchronous JavaScript and XML) entered the lexicon and applications like Google Maps and GMail pushed the envelope in terms of what web applications could do in the browser.

Load Balancers and Proxies - Eventually it all became too much for one server to handle. The earliest scaling challenges required changes in thinking that would have been radical at the time. The first applications (that sometimes ran on specialized hardware) to intercept the requests and either reroute them or serve them from a cache. I also want to dive into early scaling stories - for example, the first time teams had to deal with melting servers or the need to serve large amounts of content quickly to a global audience.

State and Sessions - Clients and servers needed to establish how to use the primitives provided by HTTP to manage more than just individual requests and responses. We needed to build caching, session data, and other forms of state into the protocol itself. Browsers started being much more active participants in the platform. Support for cookies was shipped in Netscape Navigator in 1994. They were formalized in various RFCs authored between 1997 and 2011.

SSL/TLS and HTTPS - Netscape recognized that the web needed a mechanism for secure transport. After some revisions of SSL, the IETF took over the initiative, creating TLS 1.0. Systems administrators now had to figure out ways to obtain, and maintain certificates. Pushing adoption became a huge priority for the broader web community, driven by initiatives like Let’s Encrypt. Google also helped tremendously by giving ranking boosts to sites that used https with a valid certificate.

Scaling Databases - I remember the first time I installed MySQL on my laptop and wrote a web application that persisted data to a database. It felt magical. Relational databases predate the web by decades, but by the late nineties and early aughts it was common to use MySQL and PostgreSQL as the persistence layer for web applications. Scaling became a concern and read-replicas were often added to clusters with libraries that could pin reads to the primary after writes to provide read-after-write consistency. Contributions from various early practitioners as well as companies like Percona turned databases like MySQL and PostgreSQL into battle tested web software.

Caching - Caching will be discussed in the load balancer and proxies chapter, but this chapter will go into more detail about how tools like squid and varnish were used as “accelerators”. It will also discuss the origins of memcached, a distributed key value store that commonly sits in between web applications and databases, saving databases from having to serve unnecessary reads. Redis served a similar role, but also acted as a broader data structure server used for caching of chunks of data, sets, counters, etc.

CDNs and The Edge - There are a few techniques that have been published over the years that gave birth to whole industries. Consistent hashing was one of those. Akamai was first, but the category grew to include cloud providers and other companies like Fastly and Cloudflare. Serving content quickly from edge servers geographically close to a consumer became a significant differentiator for sites with a global audience. Interest quickly grew and companies started experimenting with how much compute could reasonably be pushed to the edge.

The API Era - SaaS hit a certain critical mass and APIs started becoming an expectation. If you ran a web business, chances were you had some kind of API offering. After the dust settled from attempts like SOAP and WS-* specifications, REST APIs accepting and serving XML or JSON became ubiquitous. Specifications such as OAuth and OpenID attempted to ease interoperability. Large companies jumped in and authentication and authorization became huge economic moats with enormous privacy implications. This era also saw the proliferation of microservice based architectures and aggregation patterns like BFF and GraphQL.

NoSQL (including Search) - Early web applications almost exclusively used relational databases for durable persistence. In 2006, Google published the BigTable paper. In 2007, Amazon realized that a large percentage of database reads were primary key lookups, and published the Dynamo paper. This period saw a huge influx in activity in specialized NoSQL databases. Search also started to become common in web applications. Apache Lucene had its origins as far back as 1999, but it wasn’t until Solr and later ElasticSearch were released that it became an almost drop-in piece of architecture.

Queues and Event Streams - Queues and job systems go back to the early nineties, but few blog posts have had as big an impact in how I think about software as Jay Kreps’ 2013 manifesto, “The Log”. Kafka was open sourced in 2010 and has become ubiquitous in mid-sized and large distributed systems. A parallel work stream on job queues saw the development of ActiveMQ, RabbitMQ and lightweight solutions like sidekiq. Asynchronous processing became a foundational way to scale out backend systems, solving a whole bunch of problems and introducing quite a few as well.

Virtual Machines and The Cloud - It was common in the late nineties to use tools like VMWare Workstation to run virtual machines on development workstations, often to test portability of software. In 2003, the Xen hypervisor was born, which modified the operating system so that a guest OS and a host OS could cooperate using a technique called paravirtualization. Xen was chosen by Amazon as the virtualization solution that formed the basis of AWS. KVM built virtualization directly into the Linux kernel. This simplified virtualization significantly which along with advances in hardware virtualization, made it the eventual de facto choice. Later work on microVMs made starting up virtual machines even faster, making concepts like serverless architecture feasible.

Configuration Management and IaaC - Scaling would have had very strict limits if we didn’t have some kind of automated, version controlled way to provision and configure new compute and storage resources. Configuration management solutions such as CFEngine, Puppet and Chef provided a way to describe in code how to set up a machine based on its role as a web server, database server, or something else. Infrastructure as Code went a step above and provided ways to describe, in code, what virtual machines were required. IaaC made it possible to spin up not just a database server, but an actual hosted database run by a cloud provider, or a load balancer and cluster of application servers.

Containers and Orchestration - Virtualization allowed a single physical computer to run multiple operating systems, but with a certain amount of overhead. It wasn’t feasible to ship around a configured virtual machine containing your application, so it was necessary to provision and configure the virtual machine before deploying your application to it. Containers changed this, making it possible to ship around a fully contained environment for your application. Orchestration inverted the relationship that cloud computing started - instead of treating a single machine as a bunch of different computers, you could treat an entire data center as one big virtual machine. Containers have their origin as far back as the late seventies with chroot in UNIX environments as well as FreeBSD jails in the early aughts, but it was when namespaces and cgroups hit the Linux kernel and projects like Docker started building on them that containers became a real thing.

Continuous Delivery - A key goal in web architectures has been to minimize feedback loops. The faster we can provision and deploy an application and its environment, the faster we can deliver and iterate on value. It has also been observed that shipping smaller increments can be a lot safer than shipping big batches of changes at once. Eric Ries at IMVU started talking about how they shipped to production multiple times per day. John Allspaw and Paul Hammond gave a talk called “10+ Deploys Per Day” about how Flickr practiced continuous delivery. The practice of continuously shipping to production, once a highly polarizing idea, has now become table stakes for anyone running a 24/7 web product.

Monitoring and Observability - “Is it working?” is a fundamental question when running a web application. Monitoring goes back to the nineties, starting with tools that checked to see if a server was up and running, and eventually what metrics could be recorded, such as CPU, memory, and disk space utilization. Tools like MRTG, RRDTool, and Nagios popularized time-series data and graphs for web application developers and operators. When Chris Davis wrote Graphite in 2006 and then Erik Kastner wrote statsd in 2010, things really exploded. Engineers started emitting custom metrics from their applications and building dashboards to visualize an innumerable amount of operationally relevant information. As systems became more complicated, Observability, an idea with its roots in Control Theory and popularized by Charity Majors as well as the engineering team at Twitter, started to gain popularity. It should now be possible to ask a huge range of dynamic questions about a system running in any environment and get answers quickly.

SRE and DevOps - In order to be able to provision, configure and deploy frequently, while monitoring and being able to observe your production environments adequately, it became a huge advantage to reduce or eliminate cultural and organizational barriers between the people building applications and those tasked with operating them. The DevOps movement and Google’s popularization of the Site Reliability Engineering role changed the way many organizations architected and built web applications. Primarily a cultural movement, these ideas nevertheless had profound implications for how we architect web applications.

Timelines and Updates

I have no idea how long this is going to take - it’s a big undertaking, and I’m a parent with only weekends and evenings to work on it. At the moment, Chapter 1 is under way and I’m shooting for momentum, not strict timelines. If you’re interested in the content, you can get updates here or you can subscribe via email or RSS at buildingtheweb.dev. I appreciate your interest!

Thank you to John Allspaw, Niall Murphy, and Meg Osman for reviewing drafts of this blog post.