Paul Osman

Building The Web: A History of Web Architecture

Thu, 05 Mar 2026 00:00:00 +0000

tl;dr I’m writing a book about the history of web architecture. The audience will be engineers and anyone who wants to deep dive on how architectures have evolved over time. It will focus on the practitioners who made significant contributions to foundational ideas and technologies in web architecture, whether they intended to or not. Updates will be published at buildingtheweb.dev.

The years spanning from 1993 to 2015 were extraordinary. Linux and FreeBSD became mainstream server operating systems, the NCSA httpd project was released, it begat the Apache HTTP server, which in turn inspired projects like nginx. The world saw the birth of a global network for information exchange and its evolution into the substrate of our everyday lives. We went from editing files and placing them in directories on a server to shipping fully containerized applications through complex pipelines. Client-server architecture became three-tiered architecture, eventually morphing into the microservice death star diagrams shared in conference talks as humble brags about the amount of infrastructure required to run large sites. Most of this innovation was done in the open, often by people collaborating across organizational and geographic boundaries. I owe my career to this time period and to the people who shaped it. The amount of knowledge sharing and learning, between practitioners figuring things out the hard way, that has happened during this time period is staggering.

I’m writing a book detailing the major shifts in web architecture that happened during this time period. Things continued to happen, of course, and 2015 is admittedly a bit arbitrary - but I chose this timeline because we had containerization and orchestration cemented. Much work that has happened since has been focused on ML / AI, which is out of scope for this project. I’m writing the book because it’s important to illustrate the connective tissue between these developments. It’s also important to appreciate how these contributions were made by practitioners working in specific contexts under specific constraints, and how their innovations compounded - sometimes unexpectedly. I want to dive into how the industry evolved from stateless web servers serving static documents through the C10K problem, through developments in caching reverse proxies, improvements to OS kernels, experiments with new programming models, infrastructure like CDNs, and beyond. I’ll treat security as a common thread that runs through each chapter, with each innovation addressing some challenges and introducing new attack surface area for others.

I’m going to publish incrementally, online at buildingtheweb.dev. I’ll make each chapter available as a PDF as well as on the website. I want this to be an open process, so I may look into ways to make early versions available (probably a public git repo).

I am grateful for the support of a few trusted people in my network for reviews and connecting me to people to talk to for this project. If you or anyone you know was involved in any part of this story and would be open to speaking with me, I’d love to chat and get your perspective - or if you want to support this project in any way, don’t hesitate to reach out!

What’s Covered

The exact table of contents is a work in progress, but the progression will be roughly chronological, with each chapter covering a major architectural shift. It’s important to note that work was going on in parallel in many of these areas. I will try to draw connections where possible. Here’s the current layout:

Prologue - The Network: Then and Now - Networking was very different in 1993. This short prologue will go into some detail about the physical constraints that would have shaped architectural decisions. The evolution from dial-up to broadband to always-on mobile devices had a huge impact on the industry. The evolution of networking infrastructure, from the routing protocols that held the internet together, to the physical links that determined what was possible paved the way for many of the innovations discussed in later chapters.

The Early Web - Starting with the design of HTTP and the initial RFCs. The development of the first web server and what the early web looked like as a network of servers hosting static HTML documents. The story of the early web is well told, but I want to go deeper into the specific architectural trade-offs that were baked in at this stage.

The Dynamic Web - At some point, static web documents were joined by dynamic applications with the development of protocols like CGI. Languages like Perl and PHP became ubiquitous on the internet, sometimes by accident. This was when the web stopped being a library and became a platform for application development. I want to dive into how HTTP servers handled this, from preforking and worker models, to protocols like FastCGI, to evented frameworks like Twisted, EventMachine, and eventually node.js. In parallel, browsers became a platform in their own right. JavaScript support shipped in Netscape Navigator in 1995. Microsoft shipped an at-the-time obscure ActiveX object called XMLHttpRequest in 1999. It took a while to catch on, but by 2005 the term AJAX (Asynchronous JavaScript and XML) entered the lexicon and applications like Google Maps and GMail pushed the envelope in terms of what web applications could do in the browser.

Load Balancers and Proxies - Eventually it all became too much for one server to handle. The earliest scaling challenges required changes in thinking that would have been radical at the time. The first applications (that sometimes ran on specialized hardware) to intercept the requests and either reroute them or serve them from a cache. I also want to dive into early scaling stories - for example, the first time teams had to deal with melting servers or the need to serve large amounts of content quickly to a global audience.

State and Sessions - Clients and servers needed to establish how to use the primitives provided by HTTP to manage more than just individual requests and responses. We needed to build caching, session data, and other forms of state into the protocol itself. Browsers started being much more active participants in the platform. Support for cookies was shipped in Netscape Navigator in 1994. They were formalized in various RFCs authored between 1997 and 2011.

SSL/TLS and HTTPS - Netscape recognized that the web needed a mechanism for secure transport. After some revisions of SSL, the IETF took over the initiative, creating TLS 1.0. Systems administrators now had to figure out ways to obtain, and maintain certificates. Pushing adoption became a huge priority for the broader web community, driven by initiatives like Let’s Encrypt. Google also helped tremendously by giving ranking boosts to sites that used https with a valid certificate.

Scaling Databases - I remember the first time I installed MySQL on my laptop and wrote a web application that persisted data to a database. It felt magical. Relational databases predate the web by decades, but by the late nineties and early aughts it was common to use MySQL and PostgreSQL as the persistence layer for web applications. Scaling became a concern and read-replicas were often added to clusters with libraries that could pin reads to the primary after writes to provide read-after-write consistency. Contributions from various early practitioners as well as companies like Percona turned databases like MySQL and PostgreSQL into battle tested web software.

Caching - Caching will be discussed in the load balancer and proxies chapter, but this chapter will go into more detail about how tools like squid and varnish were used as “accelerators”. It will also discuss the origins of memcached, a distributed key value store that commonly sits in between web applications and databases, saving databases from having to serve unnecessary reads. Redis served a similar role, but also acted as a broader data structure server used for caching of chunks of data, sets, counters, etc.

CDNs and The Edge - There are a few techniques that have been published over the years that gave birth to whole industries. Consistent hashing was one of those. Akamai was first, but the category grew to include cloud providers and other companies like Fastly and Cloudflare. Serving content quickly from edge servers geographically close to a consumer became a significant differentiator for sites with a global audience. Interest quickly grew and companies started experimenting with how much compute could reasonably be pushed to the edge.

The API Era - SaaS hit a certain critical mass and APIs started becoming an expectation. If you ran a web business, chances were you had some kind of API offering. After the dust settled from attempts like SOAP and WS-* specifications, REST APIs accepting and serving XML or JSON became ubiquitous. Specifications such as OAuth and OpenID attempted to ease interoperability. Large companies jumped in and authentication and authorization became huge economic moats with enormous privacy implications. This era also saw the proliferation of microservice based architectures and aggregation patterns like BFF and GraphQL.

NoSQL (including Search) - Early web applications almost exclusively used relational databases for durable persistence. In 2006, Google published the BigTable paper. In 2007, Amazon realized that a large percentage of database reads were primary key lookups, and published the Dynamo paper. This period saw a huge influx in activity in specialized NoSQL databases. Search also started to become common in web applications. Apache Lucene had its origins as far back as 1999, but it wasn’t until Solr and later ElasticSearch were released that it became an almost drop-in piece of architecture.

Queues and Event Streams - Queues and job systems go back to the early nineties, but few blog posts have had as big an impact in how I think about software as Jay Kreps’ 2013 manifesto, “The Log”. Kafka was open sourced in 2010 and has become ubiquitous in mid-sized and large distributed systems. A parallel work stream on job queues saw the development of ActiveMQ, RabbitMQ and lightweight solutions like sidekiq. Asynchronous processing became a foundational way to scale out backend systems, solving a whole bunch of problems and introducing quite a few as well.

Virtual Machines and The Cloud - It was common in the late nineties to use tools like VMWare Workstation to run virtual machines on development workstations, often to test portability of software. In 2003, the Xen hypervisor was born, which modified the operating system so that a guest OS and a host OS could cooperate using a technique called paravirtualization. Xen was chosen by Amazon as the virtualization solution that formed the basis of AWS. KVM built virtualization directly into the Linux kernel. This simplified virtualization significantly which along with advances in hardware virtualization, made it the eventual de facto choice. Later work on microVMs made starting up virtual machines even faster, making concepts like serverless architecture feasible.

Configuration Management and IaaC - Scaling would have had very strict limits if we didn’t have some kind of automated, version controlled way to provision and configure new compute and storage resources. Configuration management solutions such as CFEngine, Puppet and Chef provided a way to describe in code how to set up a machine based on its role as a web server, database server, or something else. Infrastructure as Code went a step above and provided ways to describe, in code, what virtual machines were required. IaaC made it possible to spin up not just a database server, but an actual hosted database run by a cloud provider, or a load balancer and cluster of application servers.

Containers and Orchestration - Virtualization allowed a single physical computer to run multiple operating systems, but with a certain amount of overhead. It wasn’t feasible to ship around a configured virtual machine containing your application, so it was necessary to provision and configure the virtual machine before deploying your application to it. Containers changed this, making it possible to ship around a fully contained environment for your application. Orchestration inverted the relationship that cloud computing started - instead of treating a single machine as a bunch of different computers, you could treat an entire data center as one big virtual machine. Containers have their origin as far back as the late seventies with chroot in UNIX environments as well as FreeBSD jails in the early aughts, but it was when namespaces and cgroups hit the Linux kernel and projects like Docker started building on them that containers became a real thing.

Continuous Delivery - A key goal in web architectures has been to minimize feedback loops. The faster we can provision and deploy an application and its environment, the faster we can deliver and iterate on value. It has also been observed that shipping smaller increments can be a lot safer than shipping big batches of changes at once. Eric Ries at IMVU started talking about how they shipped to production multiple times per day. John Allspaw and Paul Hammond gave a talk called “10+ Deploys Per Day” about how Flickr practiced continuous delivery. The practice of continuously shipping to production, once a highly polarizing idea, has now become table stakes for anyone running a 24/7 web product.

Monitoring and Observability - “Is it working?” is a fundamental question when running a web application. Monitoring goes back to the nineties, starting with tools that checked to see if a server was up and running, and eventually what metrics could be recorded, such as CPU, memory, and disk space utilization. Tools like MRTG, RRDTool, and Nagios popularized time-series data and graphs for web application developers and operators. When Chris Davis wrote Graphite in 2006 and then Erik Kastner wrote statsd in 2010, things really exploded. Engineers started emitting custom metrics from their applications and building dashboards to visualize an innumerable amount of operationally relevant information. As systems became more complicated, Observability, an idea with its roots in Control Theory and popularized by Charity Majors as well as the engineering team at Twitter, started to gain popularity. It should now be possible to ask a huge range of dynamic questions about a system running in any environment and get answers quickly.

SRE and DevOps - In order to be able to provision, configure and deploy frequently, while monitoring and being able to observe your production environments adequately, it became a huge advantage to reduce or eliminate cultural and organizational barriers between the people building applications and those tasked with operating them. The DevOps movement and Google’s popularization of the Site Reliability Engineering role changed the way many organizations architected and built web applications. Primarily a cultural movement, these ideas nevertheless had profound implications for how we architect web applications.

Timelines and Updates

I have no idea how long this is going to take - it’s a big undertaking, and I’m a parent with only weekends and evenings to work on it. At the moment, Chapter 1 is under way and I’m shooting for momentum, not strict timelines. If you’re interested in the content, you can get updates here or you can subscribe via email or RSS at buildingtheweb.dev. I appreciate your interest!

Thank you to John Allspaw, Niall Murphy, and Meg Osman for reviewing drafts of this blog post.

SFP+ Support on the Honeycomb LX2

Sat, 07 Feb 2026 08:14:23 -0500

A few years ago, I bought a SolidRun Honeycomb LX2. The LX2 is a 16-core ARM board (NXP LX2160A) with four SFP+ ports and a 1GbE RJ45. Paired with the extremely nice looking Lone Industries L5 mini itx case, it’s a very cool small form factor machine that requires very little power. When I first got it, I followed Liz Fong-Jones’ tips for getting Ubuntu 20.04 running and since then it’s been happily existing as a node in a heterogeneous kubernetes cluster in my homelab.

Recently, I wanted to repurpose it as a router and take advantage of the SFP+ ports to build a 10Gbit backbone. I also was using a USB ethernet adapter because I had problems getting the onboard 1GbE NIC to work back when I first set it up, so I wanted to see if I could fix that. I wanted to start with a fresh install of Ubuntu 24.04 LTS. The onboard NIC uses NXP’s ENETC driver, which was merged in kernel 5.14. I figured by running 6.8 which ships with Ubuntu 24.04, I should be good to go there.

The Problem

NXP’s LX2160A uses a subsystem called DPAA2 (Data Path Acceleration Architecture) for its high-speed network ports. DPAA2 uses a hardware component called the Management Complex (or MC) for managing the hardware resources. Unlike normal NICs that show up as eth0, the MC is used to create and wire up network objects before Linux can see them. The tool used to do this is called restool and it has a convenience wrapper called ls-addni:

sudo ls-addni dpmac.7   # should create an eth interface for SFP+ port 7

On a fresh Ubuntu 24.04 install with the prebuilt UEFI firmware from SolidRun’s image server, this fails:

MC error: Unsupported operation (status 0xb)
Error: dpmcp object was not created!

The MC firmware bundled in SolidRun’s prebuilt UEFI images was version 10.28.1, which does not support dynamic object creation. This has been a known issue and has been fixed, but I don’t think the prebuilt images have been updated yet. Time to build a new version of the firmware!

I performed all of the following steps using minicom over the micro-USB serial console port from my laptop. Follow Liz’s instructions linked above if you have trouble getting that working. I downloaded Ubuntu 24.04 and put it on a USB thumb drive, then selected the device from the BIOS when booting the Honeycomb LX2.

Step 0: Install Ubuntu 24.04 with IOMMU Workaround

Ubuntu 24.04 (kernel 6.8) won’t install cleanly on the LX2. The installer crashes with ARM SMMU context (arm-smmu arm-smmu.0.auto: Unhandled context fault). At the GRUB menu during install, press e to edit the boot entry and add to the kernel command line:

iommu.passthrough=1 arm-smmu.disable_bypass=0

After install, make it permanent:

sudo sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT=".*"/GRUB_CMDLINE_LINUX_DEFAULT="console=ttyAMA0,115200 iommu.passthrough=1 arm-smmu.disable_bypass=0"/' /etc/default/grub
sudo update-grub

Step 1: Verify the 1GbE Works

The onboard RJ45 uses the ENETC driver, which works out of the box on kernel 6.8. Hooray, first success!

sudo ip link set eth0 up
sudo ethtool eth0   # Look for "Link detected: yes"

Now I could get rid of the annoying USB ethernet adapter.

Step 2: Build and install restool

You need restool to interact with the DPAA2 subsystem. It’s not packaged for Ubuntu, so clone the repo and download it from source. Building the man pages required pandoc as a dependency, and I didn’t want to screw around with that, so I just disabled it:

git clone https://github.com/nxp-qoriq/restool.git
cd restool
make MANPAGE= install

Now time to verify the MC firmware version and confirm the SFP+ MAC objects exist:

sudo restool --mc-version          # shows 10.28.1 on prebuilt firmware
ls /sys/bus/fsl-mc/devices/        # should show dpmac.7, dpmac.8, dpmac.9, dpmac.10

The MAC objects are there. The MC firmware just refuses to do anything with them:

sudo ls-addni dpmac.7   # should create an eth interface for SFP+ port 7
MC error: Unsupported operation (status 0xb)
Error: dpmcp object was not created!

Step 3: Fix the UEFI Build Environment

SolidRun’s UEFI build repo uses Docker. Unfortunately, it seems to be broken as of 2025. Here’s what I did to fix it:

Problem 1 - Debian Buster is Dead

The Dockerfile uses debian:buster-slim. Buster reached end-of-life and its repos return 404. Change to Bookworm:

cd lx2160a_uefi/docker
sed -i 's/debian:buster-slim/debian:bookworm-slim/' Dockerfile
sed -i 's|/usr/bin/python3.7|/usr/bin/python3|' Dockerfile
cd ..
docker build -t lx2160a_uefi docker/

Problem 2 - The Entrypoint Fails Silently

Docker entrypoint (entry.sh) uses #!/bin/sh -e and runs git rev-parse early. Because Docker volume mounts trigger git’s safe directory check, this command fails with exit code 128, and set -e kills the script before it produces any output. You get no error message, no log, nothing, just an instant return.

Bypass the entrypoint entirely:

docker run ... --entrypoint /bin/bash lx2160a_uefi -c '
 git config --global --add safe.directory /work
 cd /work
 INITIALIZE=1 ./runme.sh
 ./runme.sh
'

Problem 3 - GCC 12 Breaks edk2 BaseTools

BookWorm ships GCC 12, which enables new warnings under -Wall that edk2’s -Werror promotes to build failures. Four suppresions were needed:

Flag	File	Issue
`-Wno-vla-parameter`	`BrotliCompress/brotli/c/dec/decode.c`	VLA parameter declaration mismatch
`-Wno-use-after-free`	`GenFfs/GenFfs.c`	Upstream bug: closed FILE* passed to printf
`-Wno-dangling-pointer`	`LzmaCompress/Sdk/C/LzmaEnc.c`	False positive on local struct address
`-Wno-stringop-overflow`	`DevicePath/DevicePathUtilities.c`	False positive on memcpy into struct

The fix uses edk2’s existing EXTRA_OPTFLAGS hook in header.makefile. In runme.sh, change the BaseTools make invocation to:

EXTRA_OPTFLAGS="-Wno-vla-parameter -Wno-use-after-free -Wno-dangling-pointer -Wno-stringop-overflow" \
 PYTHON_COMMAND=/usr/bin/python3 make -C BaseTools

This survives INITIALIZE=1 because it lives in runme.sh (parent repo), not in the edk2 submodule which gets reset.

Step 4 - Get a Modern MC Firmware

Here’s the actual fix. Everything above is just getting the build to compile.

The Management Complex is a separate microcontroller embedded in the LX2160A SoC that runs its own firmware, independent of Linux. It manages all DPAA2 network objects: the MACs, buffer pools, channels, and network interfaces that make the SFP+ ports work. When you run ls-addni dpmac.7, the restool utility sends commands to the MC over an ioctl, asking it to dynamically allocate these objects and wire them together into a usable network interface.

MC firmware 10.28.1 (bundled in SolidRun’s UEFI images) has all these objects pre-allocated at boot in a static layout, but it does not support creating new ones at runtime. When ls-addni asks it to create a dpmcp (management command portal), the MC responds with “Unsupported operation.” The objects you need are technically there - you can see them in /sys/bus/fsl-mc/devices/ - but without the ability to create new dpni (network interface) objects and connect them to the existing dpmac objects, they’re useless from Linux.

Newer MC firmware versions support dynamic object creation. NXP publishes these binaries on GitHub:

git clone --depth 1 https://github.com/NXP/qoriq-mc-binary.git
ls qoriq-mc-binary/lx216xa/
# mc_lx2160a_10.40.0.itb

The MC firmware is embedded in the UEFI image at a specific offset on the SD card - it’s loaded by the SoC before Linux boots. There’s no way to update it independently; you have to rebuild the entire UEFI firmware image with the new binary included.

I added an MC_FW environment variable hook to runme.sh that copies a user-supplied .itb into the edk2-non-osi tree and patches the FDF filename before building. This is opt-in - without MC_FW, you get the old bundled version.

Step 5: Build and Flash

Putting it all together:

git clone https://github.com/SolidRun/lx2160a_uefi.git
cd lx2160a_uefi
git submodule update --init --recursive

# Apply the Dockerfile and runme.sh fixes described above

docker build -t lx2160a_uefi docker/

docker run \
  -e DDR_SPEED=2600 \
  -e MC_FW=/work/mc_fw.itb \
  -v "$PWD":/work:Z \
  -v /path/to/mc_lx2160a_10.40.0.itb:/work/mc_fw.itb:ro \
  --rm -t --entrypoint /bin/bash lx2160a_uefi -c '
    git config --global --add safe.directory /work
    cd /work
    INITIALIZE=1 ./runme.sh
    ./runme.sh
  '

Output lands in images/lx2160acex7_2000_700_2600_8_5_2_sd_<hash>.img.

DDR speed matters. My RAM is DDR4-2666, so I use DDR_SPEED=2600. My first build attempt used 3200, which produced a firmware that failed at boot with “DDR Clk is faster than DIMM can support.” The prebuilt images report 3200 in dmidecode because the firmware attempted that speed, not because the RAM supports it.

Flash to microSD on another machine (early HoneyComb revisions can’t write to the microSD from Linux):

sudo dd if=images/lx2160acex7_2000_700_2600_8_5_2_sd_*.img of=/dev/sdX bs=4M status=progress

Step 6: Activate SFP+ Ports

Boot with the new firmware and verify:

$ restool --mc-version
MC firmware version: 10.40.0

$ sudo ls-addni dpmac.7
created eth1

Hooray! We’ve got functioning SFP+ Ports now.

Wrapup

The HoneyComb LX2 is a genuinely interesting piece of hardware, but the toolchain can be a little tough to work with sometimes.

If you’re running a HoneyComb LX2 with UEFI and your SFP+ ports don’t work, the answer is almost certainly that your MC firmware is too old. Build your own UEFI image with a newer one from NXP’s qoriq-mc-binary repo.

What’s Next

I still need to actually configure this as a router - nftables NAT, DHCP, the whole deal. That’s for a future post once my switch and SFP modules arrive. But at least now, hard part is done!

Shipping When You're Not Sure

Mon, 26 Jan 2026 08:14:23 -0500

I recently had to write a query planner for a graph store. The query planner wasn’t terribly complicated - it used some simple heuristics and estimates to decide whether it would be better to execute a query term against the underlying data store or perform in-memory filtering. The hope was that this would improve the vast majority of queries without introducing any regressions.

Normally in a situation like this, I would ship the query planner behind a feature flag and instrument the code with some metric or trace that I could use to compare query performance from before and after the query planner was enabled. The trouble is, our product was not yet generally available, so query traffic was sporadic and highly variable. Some queries would take 200 ms without the planner, and some would timeout at 30 seconds. This was not an ideal dataset to test performance improvements.

What I wanted was to compare the query planners performance against a baseline using the same queries under similar conditions. This required a different approach. Luckily, I had a couple of advantages: load wasn’t a concern, and all of the queries were read-only.

The pattern

I figured it would be pretty easy to execute both code paths in parallel: the new query planner and the baseline. Go makes this especially easy - I wrapped the actual query logic in a method that took a flag as a parameter indicating whether to use the new planner or not:

func execQuery(ctx context.Context, query string, usePlanner bool) *queryResult {
    result := u.execQueryVariant(ctx, query, usePlanner)
    go func(primary *queryResult, primaryIsPlanner bool) {
        ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
        defer cancel()
        baseline := u.execQueryVariant(ctx, query, !usePlanner)
        // log results, compare primary vs baseline
    }(result, usePlanner)
    return result
}

The primary path returns immediately. The comparison runs in a goroutine, logs both results, and records the delta. I also set up an email alert so that I would be notified in the case that results diverged - latency differences could be inspected manually, but I wanted to know immediately if there were correctness issues.

Load wasn’t a concern in this case. If it were, this approach wouldn’t work. Also, all of the queries are read-only and therefore free from side effects. When these constraints hold, this approach is very helpful for comparing apples to apples.

The payoff

I deployed this with the feature flag turned off at first. User traffic was processed using the old query execution path and the query planner was executed asynchronously. This paid off immediately - I spotted several cases where the planner was making suboptimal decisions. Because I was comparing against baseline in real time, I could tune the planner incrementally, deploy, and watch the metrics improve.

This isn’t shadow testing in the traditional sense. Shadow testing typically routes traffic to a secondary system to validate it can handle load. This is closer to a continuous correctness check: same input, both paths, compared outputs.

Cleaning up

After a couple of weeks running the system with the query planner enabled and seeing good results, I removed the dual execution path so nobody tripped over it. These experiments have a tendency to become permanent complexity so it’s important to remember to remove the experiment code.

A Complementary Technique

As I was doing this work, I remembered a similar situation. I’ve written before about shipping on a spent error budget – using traffic isolation to keep iterating when you can’t afford to break things. Swimlanes and parallel runs solve similar problems differently:

Swimlanes work when you can route traffic separately and tolerate the new path being broken (no real users yet)
Parallel runs work when you need to validate against real production queries but can afford the extra compute and ensure no side effects.

Both are tools for building confidence, hopefully others find them helpful!

Where I'm at with AI

Sun, 18 Jan 2026 08:14:23 -0500

We’re in new paradigm territory with generative AI. A lot of commentary falls into the skeptic, evangelist, or doomsdayer categories, or are practical but narrow takes. In this article, I’ll discuss my current use of generative AI tools and outline areas that concern me - areas that raise questions about how our industry and others will evolve. I am certain that generative AI is a productivity amplifier, but its economic, environmental, and cultural externalities are not being discussed enough.

A quick note on terminology: in this article, I’ll use the term “generative AI” to describe the current wave of generative AI systems - large language models like Claude and ChatGPT and similar tools. I want to be clear that I’m talking about LLMs and not other areas of AI such as autonomous vehicles or medical diagnostic algorithms.

We’re Moving Quickly

If you asked me six months ago what I thought of generative AI, I would have said that we’re seeing a lot of interesting movement, but the jury is out on whether it will be useful in my day to day work. It’s remarkable how quickly my position has changed - fast forward just a few months and I am using Claude daily at work and at home. I use it for routine coding tasks like generating scaffolding or writing tests, and for ideation on new projects. I treat Claude like another software engineer and give it specific instructions. I spend a lot of time reading code it generates and making corrections before submitting a PR for my coworkers to review. I often have a generative AI tool do a pass on the PR before I ask a human to have a look, which has saved me many iterations. Coworkers of mine have used generative AI tools to build some truly mind blowing things in a short period of time. I’ve become a convert.

Coding with generative AI absolutely increases velocity. Absent the concerns I outline in this article, the role of software engineers has changed. I am probably most closely aligned with the view Mark Brooker puts forward - that most software will be built very quickly, and more complicated software should be developed by writing the specification, and then generating the code. We may still need to drop down to a programming language from time to time, but I believe that almost all development will be done with generative AI tools.

On the surface, my position here shouldn’t be surprising or controversial. I’ve long held the belief that our job as software engineers is not to write code, but rather to solve problems. If code is the most efficient way to solve a problem, then great. Generative AI makes code much cheaper to generate. That comes with some huge wins, and some very real concerns that I’ll outline here. My purpose is not to express skepticism, or cast doubt, but rather to shine a light on questions that to my knowledge, are still open.

Ironies of Automation

Lisanne Bainbridge’s 1983 paper “Ironies of Automation” posits that automation can relegate humans to exception handling tasks (think of this whenever you hear someone say that it’s important to always “have a human in the loop”) and that when humans are relegated to such tasks, they become less effective than if they had a more active role.

The best example of this that I can think of relates to roadway design and safety. “Stroads” (hybrid throughfares that combine qualities of high traffic streets and roads), for example, are dangerous because they encourage driving at high speeds and reduce the amount of friction encountered on a route. This causes inattentive driving which leads to more crashes and fatalities. Replacing stroads with more obstacles results in fewer crashes. One of the more striking examples is redesign of the La Jolla Boulevard in San Diego, where crashes were reduced by 90% after going from 5 lanes and 70ft pedestrian crossings to 2 lanes and 12-14 foot crossings with islands. Traffic volume stayed the same, and crashes plummeted. This video documents similar phenomenon contrasting urban design in Toronto and cities in the Netherlands.

I’m certainly not the first to draw similarities between Bainbridge’s paper and the current use of generative AI tools. Mica R Endsley, former Chief Scientist of the U.S. Air Force published a paper called “Ironies of artificial intelligence” in 2023 which directly builds on Bainbridge in this context.

The principle applies beyond roads. In software and operations, we have long accepted that a certain amount of friction is necessary or beneficial. Very few companies would release software without doing some kind of security evaluation, and software teams frequently debate how many gates need to be included to operate safely. Anyone who has found themselves in a vibe coding loop with their role reduced to periodically saying “yes” or “no” to a coding tool knows how this principle could easily apply to generative AI and coding.

Certain kinds of friction, such as code review, also have secondary benefits – they are a tool for vicarious learning and reducing the bus factor for parts of a system. By reviewing each others code, we are more able to safely modify and operate that code. Just as drivers lose attentiveness on over-automated roads, software teams risk losing deep system understanding if they offload too much judgment to AI.

Open Source is Behind

People sometimes compare the current wave of generative AI coding tools with other shifts in how we build software. It is true that in my career, I’ve seen us move further and further away from bare metal thanks to the introduction of new tools. Higher level languages, frameworks, and tools that allow us to develop at a higher level of abstraction, such as virtualization, containers, and orchestrators. These comparisons are fair, in my opinion.

Most of the previous waves were dominated by Open Source technology. Docker, Kubernetes, Linux, Xen, GCC, Ruby, Python, Rails, Numpy, JQuery, React, and countless other technologies have made software engineers more productive. They were also, however, open source and available to anyone with an internet connection. I am deeply concerned that the current wave of generative AI is highly dependent on a small set of vendors (OpenAI, Anthropic, Google, etc). I do not know what implications that will have, but I can say that before the great mainstreaming of open source in the late nineties and early aughts, we were far worse off as an industry – accessibility was a real concern (it was harder for newcomers to the field to get started) and innovation was nowhere near as plentiful.

Vendor dependency concentrates control over core development infrastructure. That centralization risks slower innovation, higher pricing, and reduced accessibility - the opposite of what open source has historically delivered.

The reliance on vendors brings me to my next concern.

We Aren’t Paying the Real Cost

The current landscape is a battle between loss-leaders. OpenAI is burning through billions of dollars per year and is expected to hit tens of billions in losses per year soon. Your $20 per month subscription to ChatGPT is nowhere near keeping them afloat. Anthropic’s figures are more moderate, but it is still currently lighting money on fire in order to compete and gain or protect market share.

We’ve seen loss-leader strategies before, most notably with Uber, which I’ll return to later. The danger in these strategies is that dependencies are being created while the product is cheap, meaning alternatives – even open source tools, can’t compete. This has the potential of creating lock-in where companies integrate these tools into their products, developers become dependent on them in their workflows, and even students learn using them.

What will this mean for generative AI? I have no idea - obviously there is a chance that some breakthrough will make LLMs far cheaper to build and operate, or companies will have to start forking out a much higher price to use these tools, which could make our industry even harder to break into for people who can’t afford the premium. For now, the question isn’t whether this is sustainable - the companies themselves admit it isn’t. The question is what happens when the subsidy phase ends.

Environmental Impact

Even if the markets appetite for losing billions of dollars annually continues, the use of generative AI is not at all free. LLMs require enormous compute. That compute generates heat. Cooling that heat consumes water - massive amounts of it. A recently published study in Nature Sustainability concluded that AI could have a footprint of 731-1,125 million m³ of water and 24-44 Mt CO2-equivalent emissions annually from 2024 to 2030. That’s equivalent to 200 - 500 bottles of water per person on Earth annually. A similar study published in Patterns concludes that AI systems could produce the same amount of CO2 as the entire city of New York in 2025.

I agree that generative AI is exciting, but the speed with which a lot of industry leaders went from being apparently concerned with the environment to being happy emitting a NYC worth of CO2 in a year is dizzying.

Marketing is Off - Not Really AI

This might be comparatively trivial, but it’s always bothered me how loose we are with the term “AI”. Artificial Intelligence has the humorous distinction of being a field that is named for what it hopes to achieve, not what it actually does. We have not yet created anything that can be called intelligent, instead we have created impressive technology that looks like thinking without being it.

Noam Chomsky, Ian Roberts, and Jeffrey Watumull made this point sharply in their New York Times piece “The False Promise of ChatGPT,” characterizing these systems as statistical pattern-matchers rather than thinking machines. They note these tools are useful for tasks like computer programming, but we shouldn’t mistake that utility for understanding.

This framing matters because it affects how we talk about these tools and what we can expect from them. Calling them “AI” rather than “LLMs” or “generative systems” inflates expectations and enables hype bubbles. Setting realistic expectations helps us figure out where these tools actually fit into our workflows and what their genuine limitations are.

Beyond terminology, there’s a more fundamental question about who benefits from this shift.

Where’s the Wealth?

Generative AI will certainly be economically disruptive. Jobs will change and in some cases, certain types of work may be eliminated. Many similar technological shifts have resulted in increased demands for certain kinds of work – meaning that overall, economic opportunity grows, but the disruption is still there.

I’ve heard more optimistic people suggest that generative AI may lead to more leisure time for more people, without negatively impacting them economically. I disagree with this. You don’t have to look very far into the history of technological shifts to see that increases in worker productivity at best increase demand for labor, and at worst result in massive disruption - they never result in the same pay for less manual work.

Because these technologies are vendor specific, I actually predict that generative AI will make a relatively small number of people massively wealthy, and a large number of people moderately wealthier, but I fear that an even larger number of people could be left out in the cold as certain types of work become possible without human labor.

When a technological shift is centralized to one or two vendors, there is a real possibility of a massive wealth transfer. Uber famously benefited from massive investor subsidies while following a loss-leader strategy. This enabled them to drive out competition, both from established players such as taxis and from future projects such as public transportation that would have belonged to the public. Once competitors were eliminated or reduced in size and Uber’s user base was established, prices were raised and consumers were left with few alternatives. During the subsidy phase of Uber’s growth, passengers paid only 41% of costs, leading to a loss of $20 Billion dollars from 2015-2019. Once the subsidy phase ended, fares increased by 65% and the rate that Uber took from drivers was increased.

The end result was that competitors were eliminated, cities became dependent on ride sharing, and users were locked into higher prices. I fear we could end up in a similar situation with generative AI where productivity is gained through use of generative AI tools, companies become dependent on those tools, prices increase once the land-grab has been established, and workers are left either paying higher premiums for AI tools, or seeing reduced compensation because of the productivity gains seen by using generative AI.

Where AI Doesn’t Belong

I’ve focused so far on technical and economic concerns, but there’s something that troubles me more fundamentally.

Generative AI is being used for a variety of non-technical tasks such as writing and creating art, including visual art and music. This disturbs me greatly. Art does not have a single purpose, and debating the purpose of art is another subject entirely, but for me, art is a way to communicate a variety of ideas and emotions across time and geography. Experiencing art as a viewer or listener provides a connection with the artist in a very real and human way. Replacing the artist with a computer, in my strong opinion, strips away something essential. Even if the consumer can’t tell that the art was created by generative AI, we’ve lost something real and unquantifiable in the exchange and I fear for what that means for our culture and civilization.

So, What Now?

The productivity gains experienced by using generative AI tools are not small, but neither are these concerns. As a professional, I am obligated to use the most effective tools that help me do my job well. I am also obligated to consider the trade-offs between doing something fast and doing something safely. Generative AI applies a thumb on this scale in a very real way, but does not completely eradicate the need for some friction. This will be something that we in the software industry figure out over the next few years. I predict a bit of a roller coaster.

As a human, I am concerned about my impact on the environment and our collective experiences on this planet. As I’ve mentioned, there’s no putting the genie back in the bottle, so it’s imperative that we continue to focus research on making LLMs more environmentally sustainable and I implore those subsidizing the unrestrained growth to take a very real look at the economic realities staring at us all. I do not know what the future holds, and I would never venture to guess what impact this will all have on the environment, our collective sense of purpose, and our careers, but it is up to us all to try and make it as positive as possible.

The challenge isn’t choosing “AI or not AI” - that ship has sailed. It’s navigating the shift thoughtfully and considering the trade offs of how and when we use it in our day to day lives.

Solving a Murder Mystery

Tue, 29 Nov 2022 08:30:00 +0000

A coworker asked me if I ever write about difficult bugs I’ve encountered. I hadn’t thought about doing that before, but I like the idea. The conversation made me think about a bug I ran into at Honeycomb that I want to document before I forget the details. Solving this one was a team effort, requiring different insights from different people. It was also the kind of bug I could only imagine solving with good observability and teamwork.

I will detail how some of Honeycomb’s systems worked when I worked there. This, and the details of the bug, are from memory so something is bound to be incorrect. Also, a lot has changed since then, so this isn’t intended to be current. Finally, because I no longer work at Honeycomb, I don’t have access to the queries we ran while debugging this issue, so you’ll have to settle for my hand-drawn approximations! (did you know that queries in Honeycomb last forever so you can always refer back to them? Seriously underrated feature if you ask me).

Context

Honeycomb is an observability tool. It is a service that accepts ultra-wide events (as JSON or OpenTelemetry Protocol) and stores them in datasets. Users run queries against their datasets, slicing and dicing them by multiple dimensions.

Honeycomb can serve most queries in a second or two, and queries can be filtered on or grouped by high cardinality fields. Example queries that a user may run include “Show me the P95 and a heatmap of latency of requests to a particular endpoint in one of two provided availability zones, grouped by customer id over the last two days.”. Or “Show me counts of requests grouped by HTTP status codes over the last two weeks.”.

The Symptom

A customer complained that a specific query was consistently generating an error. The particular error message they were seeing usually indicated a timeout, which generally indicates an availability problem with our columnar datastore. However, this case was consistently reproducible, pointing to a problem with this specific query, not database availability. There was a problem with the data causing errors in the query processor—running the same query with different time windows produced no error, providing further evidence for the bad data theory.

Technical Details

Honeycomb stores events in a custom columnar datastore called retriever. Retriever stores all customer events and processes queries for customers.

All customer events flow through a distributed event store, Kafka. Honeycomb maintains a mapping of customer datasets to Kafka partitions (higher volume datasets get more partitions). When an event is received by Honeycomb’s ingest service, it is published to one of the Kafka partitions that the customer dataset is mapped to. At the other end of the Kafka partitions, two redundant retriever nodes consume events and persist them to their local disks. Event logs are append-only, and retriever nodes have fast nvme disks.

When a retriever node consumes an event, it writes the event to separate files per column. Retriever groups events into segments. Segments are routinely rolled out (when I worked at Honeycomb, they were marked read-only, and a new segment was created when the segment hit 1 million events, 1GB of data, or when 12 hours had passed).

An out-of-band process manages the lifecycle of segments. Segments go from being live and having data written to them, to being read-only but still stored on disk, to being read-only and stored in S3, and finally to being tombstoned and eventually deleted when the data ages out (it is no longer queryable).

When a query is processed, each retriever node that could have data for the query processes the data it has on disk. Data from segments in S3 is retrieved and processed with lambda functions, which merge with the results from the retriever nodes. AWS Lambda functions are a core part of how retriever processes queries.

Custom Instrumentation to the Rescue

Okay, back to the problem. At this point, I’m assuming that there’s a piece of bad data causing the queries to fail. I narrowed it down to a one-minute range of data, so I was confident in this hypothesis. However, I assumed it was a problem with invalid data (e.g., a division by zero error somehow being thrown by Honeycomb’s derived column DSL).

One of my coworkers started looking at traces and noticed that the lambda functions had an error message that read NoSuchKey in the error field of the span. The custom error field alerted us to the possibility that the segment was somehow missing from S3 – we independently confirmed this by trying to fetch one of the bad segments using the AWS CLI tool and seeing it return a 404.

This discovery was a scary realization - data we thought we had stored was missing!

NOTE: While losing data is always scary, especially when the scope is unknown, a single segment represents a relatively tiny amount of data. If the query wasn’t failing, none of the aggregate values (SUM, AVG, P95, etc) in an average sized dataset would be noticeably skewed by a single missing segment when queried over a typical time window. While observability data is fundamentally lossy and data loss isn’t great, it doesn’t have the same criticality as say, a transactional database.

S3 Prefixes

Before continuing, it’s necessary to understand how Honeycomb stores segments in S3.

S3 is a great storage system. It allows Honeycomb to store a massive volume of events, enabling customers to query their data over long time windows (The default is 60 days as of this writing).

To optimize performance and avoid hotspots, it’s a good practice to prefix objects in a bucket with high variability. An engineer at Honeycomb had previously led a project to design a prefixing scheme for our segments. When uploading to S3, objects are given a fixed-length hash code as a prefix. We calculate the hash code from the dataset id, the partition id, and the segment id. For example, given dataset id 432, partition id 32, and segment id 1023:

hash(432, 32, 1023) = 9d4

We then prepend this prefix to the S3 object name when uploading the segment, so the object name for files in segment 10253 on partition 32 and dataset 432 would have a prefix like this:

9d4/432/32/10253

Without the hashing, all objects in segments for dataset id 432 would have the same prefix. If 432 is a frequently queried, huge dataset, this could result in hotspots and performance degradations.

Finding Needles in Haystacks

So we knew that specific segments did not exist in S3. Now what? Failing to store data is pretty bad. We started combing through our S3 logs dataset in Honeycomb to find out what happened. We saw that requests for one of the problem segments went from returning 200 (OK) to suddenly returning 404 (Not Found). Okay, so the segment existed at some point, and then it didn’t.

Because of the overall volume, the S3 logs dataset we stored in Honeycomb is heavily sampled. If an errant process deleted the problem segment, it’s unlikely we would have the specific log line in our dataset.

As mentioned previously, retriever has an out-of-band process that manages the lifecycle of segments, including deleting the segments from S3. Still, it should only delete segments that have aged out. These segments that suddenly went missing were still supposed to be serving data.

Another engineer downloaded a huge trove of S3 logs and started running parallel grep tasks to find entries related to the problem segment. This was challenging - Honeycomb has many terabytes of S3 logs. Nevertheless, we hit the jackpot. The log files showed that when the out-of-band process deleted an aged-out segment with a name like 9d4/432/32/102, it also deleted a newer segment with a name like 9d4/432/32/10253. This accidental deletion only happened to a few segments out of billions, but that’s enough to fail a query if the time window spans the data contained in the missing segment.

Another engineer confirmed this by running a Honeycomb query showing that on sequential runs, our out-of-band lifecycle manager showed an unusually high count of objects deleted for a really old segment and then no objects deleted on the next run.

Looking closer at the code, I confirmed we were not appending a trailing forward slash to S3 objects when deleting aged-out segments from S3. We were using the S3 API in such a way to emulate deleting all “files” in a “directory”, so the provided key was not an object name, but a prefix. When we deleted 9d4/432/32/102, a very old segment, we were also deleting 9d4/432/32/10253 because the two segments were part of the same dataset, came in through the same partition, both had segment ids that began with 102, and happened to have a hash collision. We really should have been deleting 9d4/432/32/102/.

Hash collisions are common at scale, but in this case, the hash was never supposed to guarantee the uniqueness of keys. It just needed to spread out the prefix scheme so segments from the same dataset wouldn’t get grouped together. Some collisions were acceptable and even expected because the other components that made up the S3 object name would be different (even if the dataset and partition are the same, the segment id wouldn’t be).

The real issue here was a bug in our segment lifecycle management process. A simple missing trailing slash was inadvertently deleting more data than intended (Honeycomb employees love puns, so someone dubbed this murder mystery a “real slasher”). Funny enough, because hash collisions don’t happen all of the time, our prefixing scheme ended up hiding this bug from us for a long time. This bug would have been immediately apparent if we weren’t prepending segments with a fixed-width hash code.

The fix ended up being about twenty lines of code, including comments. I think I had it ready and submitted a PR within 15 minutes of the team discovering the source of the bug.

A Series of Coincidences

Someone pointed out that there are some fun statistical properties to this bug. For instance, while hash collisions are pretty common at scale, they only matter here if the two segments are on the same dataset, on the same partition, and have ids with the same first digit or digits. A hash collision wouldn’t matter if the paths end up being completely different anyway (e.g. 9d4/432/32/10253 and 9d4/321/23/43).

This bug skews heavily towards newer customers who generated a lot of events – these customers create new datasets with a lot of segments (because segments get rolled out every 1M events or once they hit 1GB). Segment ids start at 1 for every new dataset. The distance between 10⁰, 10¹, 10², …, 10ⁿ grows very quickly as n increases. Furthermore, the distance between 10ⁿ and 10ⁿ⁺¹ is much smaller than the distance between m · 10ⁿ and m · 10ⁿ⁺¹ for larger values of m where 0 < m < 10. This is a fun application of Benford’s law which maintains that real-world data sets of numbers will include many more leading 1s and 2s than 3s and so on. So we can see that this bug becomes increasingly unlikely as segment ids increase. If we accept that hash collisions will have a uniform distribution, we can see that they’re less likely to result in an accidental deletion with segments that begin with a leading 8 than with segments that begin with a leading 1 or 2.

So the hash collisions wouldn’t matter, except that all the rest of the data up until the segment prefix was the same. The rest of the data being the same wouldn’t matter if we had used a trailing slash in the object name when deleting objects from S3. This bug was noticed because we were growing fast and adding more and more teams, who were creating more and more high volume datasets, hence increasing the probability of one of these consequential collisions biting us.

Team Work and Observability

Solving this bug was a lot of fun and took four engineers looking at things from slightly different angles. The solution was simple but figuring out what was happening involved stitching together a series of clues. I don’t know how we would have caught it without custom instrumentation. No auto-instrumentation tool will know that the s3 hash prefix is a very important part of a segment, for example.

Thankfully this system had pretty good instrumentation that gave us observability from a few different angles - all of which ended up being necessary to come to the solution we did in the time we did. We used a fair amount of Honeycomb querying and also had to look at raw access logs — that’s rare in my experience, but further proof that while sampling and aggregation can save on disk space and bandwidth, it’s always nice to be able to get all of the source data you need, even if you have to pull it from cheaper storage.

(Thank you to Fred Hebert and other Honeycombers for reviewing this post and feedback. I hope it was a fun trip down memory lane!)

Getting up to Speed

Sat, 05 Nov 2022 00:00:00 +0000

I recently started a new role working on data infrastructure at LinkedIn. Working with data at a vast scale at an organization with a lot of experience in the space is exciting.

The area I work in is described internally as “Big Data.”. That term has been used in many marketing materials and can induce a lot of cringes. For this post, “Big Data” refers to working with large, semi-structured datasets. These can be hosted on distributed filesystems or streamed through a distributed event platform like Kafka. The datasets are typically not pre-loaded into an OTLP or OLAP database.

I’ve worked for 22 years in this industry, but most of my experience applies to online or real-time systems. Working with offline data is new to me, so I’ve had to get up to speed quickly with the history, trends, and vernacular common to the domain. Before starting in this role, I had some ambient awareness of terms like data warehouse, data lake, MPP, DAG, and others. Still, I wasn’t comfortable discussing them in-depth, and I wasn’t up to date on developments in the data ecosystem.

I focused my research on two categories: learning about the history of the space and learning about the customer profiles (data scientists and data engineers). In this post, I will focus on how I researched the former.

Reading Papers

Reading papers is something I started doing back when I worked in Bioinformatics. Most of the concepts I explored then came from academic research, and reading research papers was the only way to stay up to date. I picked the habit up again when I worked at PagerDuty.

Reading papers can be daunting for someone like me without an academic background. I’ve found that the approach Caitie McCaffrey describes in her blog works well for me (see “A Note on Reading Papers” in the linked article). Marc Brooker’s article Reading Research: A Guide for Software Engineers is also an excellent resource. Watching PapersWeLove talks is also a great way to read some computer science papers. I like reading a paper, watching a talk on it, then returning to it to re-read sections that confused me the first time.

Creating a Reading List

In Ten simple rules for developing good reading habits for grad school and beyond, the author recommends that if you have to learn about a new topic, consider reading in chronological order. I decided to do this and compiled a list of papers, starting with the Google File System paper published in 2003, continuing with the MapReduce paper in 2004, and so on.

I used a straightforward process to build my reading list. I knew that the GFS and MapReduce papers were among the earlier publications in the domain, so I anchored my search with them. I searched for them on Google Scholar and then looked at lists of papers that cited them. Many familiar pieces turned up, including the original Spark paper, but also many unfamiliar ones to me, such as the TensorFlow paper, the Flink paper, and eventually, the Photon paper and others that represent more recent contributions to the domain.

Here’s my list so far. I’m working through it, but please comment if you have anything to add!

Connecting to Experience

One exciting thing about this exercise was that it allowed me to compare what I was reading to what I was doing in the industry when the paper was published.

As mentioned, I’ve never worked directly in the “Big Data” domain, but I have worked with data in some form or another in my career. When the GFS and MapReduce papers were published, I worked in Bioinformatics. Bioinformaticians often need to analyze large datasets and feed the output of their analysis into some other process. The team I worked with built a workflow automation language and runtime engine designed to run on OpenMosix clusters with datasets stored on large NFS volumes. NFS was a pain to maintain and susceptible to common failure modes, so reading about how GFS was designed to run on commodity hardware and treated component failures as the norm made a lot of sense to me. Our lives would have been much easier if we had GFS and MapReduce (or the Hadoop ecosystem, the open-source project primarily inspired by these projects). (In retrospect, what we built was similar to Azkaban or Airflow).

Some of what I read doesn’t relate to my experience, but I still try to imagine how I’d solve the problem stated in the paper. It’s a valuable exercise that helps me get into the right headspace to read the rest of the paper.

Being Hands On

Reading papers is a great way to generate a broad understanding of work done in a particular domain. I also like to combine it with hands-on operational work. Thankfully, most of the topics covered relate to some open-source work that I can use to gain hands-on experience, even if just with toy examples. For example, after reading the GFS and MapReduce papers, I set up an HDFS cluster on a small set of raspberry PIs I had lying around (with my Honeycomb LX2 acting as a ResourceManager and NameNode). Maybe it’s my ops background, but I like seeing what’s happening at an OS level when running MapReduce jobs or Spark tasks. Some people get the same level of understanding while skipping this step, as always, YMMV.

Making Time

Researching and deep diving into a particular topic or domain takes a lot of time. I try to get through two papers per week. I’ve recently started working from the NYC office, about a 35-minute commute from my house in Brooklyn. I use this time to read through a paper on my Remarkable 2 tablet and make notes for later. I’ll also use time in the evenings after my daughter has gone to bed. Whatever works for you, if you can etch out an extra hour or so per day, that’ll likely be enough!

Being on a Board

Mon, 17 Oct 2022 00:00:00 +0000

I recently left my job at Honeycomb. They’re a fantastic company, and I think they will be wildly successful. Still, life circumstances are such that I wanted to explore an opportunity at a much larger technology organization. More on that in another post, perhaps. In this post, I want to reflect on a unique aspect of my employment at Honeycomb: I was the first employee to sit on their Board of Directors as a full voting member. While this is a common practice in some countries, as far as I know, I am the first non-founder/executive person at a venture-backed software company in North America to have had this opportunity (since first publishing, people have pointed out some other companies that have also done this, which is great to see!). Because I left my term early, I only got to participate in two meetings, so these reflections come from that partial sample.

Nominations and Voting

When Honeycomb decided to add an employee board member, our founders, Charity and Christine, outlined a process in a wiki document and circulated it to everyone in the company. Employees who had been employed full-time at Honeycomb for two years would be eligible for board membership and could be nominated. Anyone who had been working for six months could vote. After the nomination period, each candidate wrote a 500-word pitch explaining why they wanted to be on the Board and how they would approach their responsibilities. Employees then voted using ranked-choice ballots. There were four nominees, three from engineering and one from the sales team. Voting continued for a fixed period, after which Charity announced that I would be Honeycomb’s first employee board member!

What is a board

A part of this role involved demystifying what a board is and isn’t, so I want to cover that before going into more detail.

Every corporation has a board, and many other types of organizations do too. A board of directors is a group of people who jointly supervise the activities of an organization. For a venture-backed startup, the Board usually consists of the founders and representatives from venture capital firms that have led or participated in significant funding rounds.

Board members have a fiduciary responsibility, which means they are required to serve the interests of shareholders in the organizations. That is open to interpretation, of course. Still, usually, it means they’re out to maintain the financial health of the company and maximize the value of the company in the long term. In other words, boards want measured bets and continued growth.

Boards often have independent members who are there to either provide some outside perspective or to bring some specific experience to the group. These people have often worked in similar markets or have knowledge or skills that the company needs.

Meeting Format

The board meetings I attended had a pretty standard format. At Honeycomb, the executive team spends a few weeks in advance preparing the board deck. The deck outlines the meeting agenda, which usually includes a recap of various company health indicators for the last quarter and topics that might be of interest — strategy outlines or a deep dive into a particular area of the company.

We go through the deck in the first part of each meeting. The Honeycomb executive team is present, and each member reviews updates relevant to their specific areas (e.g., Chad, the VP of Sales, will give an overview of the sales pipeline). The Board, legal counsel, board observers, additional representatives of investor firms, and people who have participated in angel and other investment rounds are welcome to join this part of the meeting. The deck is sent out ahead of time as pre-read for the meeting. People tend to ask clarifying questions and might ask to dive deeper into a specific area on one or two slides.

The next part of the meeting is administrative. The board members discuss and vote on administrative issues such as certifying the minutes of the previous meeting and financial matters like compensation changes, stock grants, etc. Non-voting participants, except for the legal counsel, are asked to leave before this part of the meeting.

Summarizing Board Meetings to the Company

A big part of my responsibility as an employee board member was to summarize board meetings to the rest of the company during an all-hands meeting. I had a 15 - 20 minute slot, during which I would recap what the Board’s role is and who are the other board members. Especially at a growing company, repetition never hurts the message, as many people would be hearing this for the first time. I’d then go through a recap of what I thought the highlights were. Everyone in the company has access to the board deck, so I don’t have to recap everything, and I usually kept it to what I thought were the big topics of discussion. I’d also try to give everyone a sense of the mood and how receptive investors were to our strategy. As an employee, I might pick up on different things in the meeting, and execs who were also present would often tell me that they were delightfully surprised at what I would choose to highlight.

Funnels, Funnels, and more Funnels

One personal takeaway from my participation on the Board is that, at a certain level, every organization has to get good at looking at and interpreting marketing and sales pipelines. A pipeline is just a value delivery chain, so looking at the shape and properties of the funnel can hopefully project roughly how successful an organization will be in the short and medium term. You can break a funnel into different pieces, commonly referred to as Top of the Funnel, or Tofu, Middle of the Funnel, or Mofu, and Bottom of the Funnel, or Bofu. So if you hear people talking about Tofu, Mofu, and Bofu, they’re talking about funnels.

A marketing funnel feeds into the sales funnel. By looking at the percentage of movement from one part of a funnel to the other and the amount of time it takes to travel through a funnel, you can make educated guesses about how things will go for the next little while. Investors especially spend a lot of time looking at the health of various funnels; if these look healthy, everybody will be much more relaxed.

Trust

Honeycomb is doing well as a company. Everything an investor or executive would measure is going in the right direction for them. As a result, I didn’t witness any tense discussions. The Board acted as an experienced audience to guide Honeycomb’s strategy. I understood that the effort to prepare and present the board deck was worthwhile for everyone, even if there were no significant disagreements. As the quote goes, “plans are useless, but planning is indispensable.”

Of course, this level of trust isn’t going to be present on every Board or at all times. If a company is having significant difficulties, I’m sure the discussions in these meetings can be much more consequential.

Leaving Early

My original term was for a year, but I chose to leave Honeycomb and resign my board membership after six months (My family and I moved to NYC and I needed to bring home a higher salary than can be reasonably obtained at a startup). The process outlined specified that I would pick a successor if I resigned before my term was complete. I chose one of the original nominees who I thought would approach the responsibility similarly to me, providing as much consistency as possible to the Board.

Admittedly, it was a bummer to leave early. I didn’t want it to harm the experiment. I thought about delaying my family’s move to NYC, but with a school-aged child, I didn’t want them to have to change schools mid-year. On the bright side, the Board gets to experience another employee perspective!

Being a Representative

As an employee on the Board, people would sometimes approach me with concerns. These fell into two categories: 1) things that could be relevant to the Board and informed how I interpreted parts of the Board deck, and 2) things that the person should bring up with their manager. In the latter case, I tried to give that feedback directly to the person.

I tried to solicit feedback frequently, especially after my presentations to the company during our All-Hands meetings. It’s hard to know what people want to hear or what information is worth conveying since you’re mostly demystifying something for many people. I received positive feedback, both from execs who were at the meeting and employees who weren’t.

Reflections

I appreciated the opportunity to serve on Honeycomb’s Board of Directors. My term was short, but the experience was valuable and changed how I see startups — I now have a behind-the-scenes perspective typically reserved for executives and founders. I’ll probably understand the motivations of executive teams a lot better from now on.

I am aware that I served on the Board when Honeycomb was doing well as a company. Consequently, I didn’t participate in any discussions or decisions that had the potential for a profound impact on Honeycomb employees. I am also aware that this is precisely the time when a company has the opportunity to experiment with having an employee sit on the Board – when trust is high. When things are going well, the Board trusts the founders to steer the ship.

When Honeycomb first announced that I was joining the Board, some people outside the company compared having an employee serve on the Board with having a union. I want to be clear that these two things are nothing alike. Unions exist to provide workers leverage in negotiations on issues that concern them. The structure accepts divergent interests between groups and provides countering influence to even the playing field. Having an employee on the Board of Directors provides that employee an opportunity to represent a point of view that could be helpful to the rest of the Board. Still, if interests are not aligned, they will not be very impactful.

Having an employee Board member provides an extremely high level of transparency to the rest of the company’s employees. There’s an implicit trust that an employee can build that an executive would struggle to obtain. The Board benefits from an additional perspective- particularly one close to the knives- that can help it make more informed decisions.

I hope to see more companies perform experiments like this, and I hope that Honeycomb, both the company and the Board, benefited as much as I did.

On Programming Languages

Sat, 30 Apr 2022 00:00:00 +0000

I was having a conversation with another software engineer the other day and they asked for my thoughts on various programming languages and paradigms. The topic came up because I mentioned that Python3 supported Optional types which are a popular feature in many functional programming languages like Haskell, OCaml, and F#.

Conversations like these often turn into statements about the general merits of technology x or y. It can be very tempting to draw specific conclusions. I’ve heard many of these, and have held them myself in the past. Conclusions like “Functional programming makes parallelism easier” or “Static typing reduces bugs” are seductive in their simplicity, but they smooth over the massive surface area that Software Engineers find themselves navigating these days.

Personally, I really enjoy functional programming. I enjoy that functional programming often allows you to express very complicated processes as simple chains of operations on immutable data. I learned Haskell many years ago by watching Erik Meijer’s Channel 9 Lecture Series and reading along with Graham Hutton’s Programming in Haskell. I worked at a few companies where I was paid to write Scala, and we used the Finagle RPC System which uses a lot of functional programming concepts.

These days, most of the code I write at work is in Go which is about as imperative as you can get. Functions are first class objects, and Go 1.18 has introduced generics, but that’s about it as far as functional programming concepts go (sure, generics aren’t specifically a functional programming concept, but I’d argue that they’re kind of necessary in a statically typed language if you want to do any kind of functional programming). Go does, however, have a lot of power in its simplicity, in its treatment of errors as types, and the ability to model concurrency through channels and goroutines.

I’m also really enjoying learning Rust which is a multi-paradigm language particularly well suited for systems programming. For that matter, I’ve also been brushing up on the last 10 years of developments in C++ as part of doing the assignments in a CMU databases course that I’m following.

So, what languages and paradigms do I prefer, and why? Here’s the kicker: I don’t care. I really don’t. There are so many factors that go into whether a particular programming language is a good choice for a team or project. Factors such as, what does the team have experience with? What problems are you solving? What support is there for various types of tooling and infrastructure at your organization? What conventions already exist and is it worth the friction of going against those conventions? Sometimes it is worth it, sometimes not. Only you and your team know enough context to make those sorts of decisions. What’s the ecosystem for a language like? Some languages have found niche status in certain domains, for instance with Python and ML.

Changes have to be considered in context. There are merits, for instance, to using Optional types in Python3, that’s why they were added. But if a Python code base that you’re working with has a convention of throwing exceptions when a function cannot return a specific concrete type, which is going to cause more friction to use? A team can decide to adapt to changes in a language and start moving towards new conventions, but that’s a highly contextual decision that shouldn’t be made in isolation. Similarly, if an organization has really great infrastructure and tooling for developing services and applications in JVM based languages, there’s a huge benefit to sticking to those.

Learning new programming languages can open you up to new areas for development. For instance, there’d be quite a lot of friction to doing systems programming in a language that doesn’t support low level memory management, whereas a language like Rust or C or C++ will make this easier. Most enjoyably, learning a new programming language can broaden how you think about programming. This is wonderful. I’m also so happy that we live in a time with such rich diversity of problems that need solving. We’re not all working on transaction processing systems at banks and business intelligence applications like we seemed to be when I started my career in the early 2000s (just after the dot com bubble burst). Just as in databases, one size does not fit all.

I also think that cross-polination between language communities is a great thing. Features like list comprehensions, Optional types, Futures and Promises have found their way from one language to another. There’s an interview where Simon Peyton Jones talks about Haskell as a vessel for this kind of cross-polination that I find really refreshing, especially in a world where we can get caught up in language bigotry and holy wars. Have a watch:

Sociotechnical Lenses into Software Systems

Sat, 02 Oct 2021 08:30:00 +0000

Software systems are sociotechnical. I don’t think software professionals spend enough time discussing the challenges that span software and personal aspects. When we look at software through a sociotechnical lens, we begin to appreciate the complexity inherent in software development and operations. The systems we are building and operating are constantly being modified by different people, with different contexts, at different times, who may or may not speak to each other directly. This emergent collaboration can present unique challenges that are fun to navigate.

Early in my career, I thought about software systems as static, unchanging entities. If you’ve ever thought of service maps or architectural diagrams as a panacea (instead of yet another lens), or if you’ve ever talked about software being “done,” you’ve probably done the same.

When we think of software this way, though, we miss a fundamental property - people are constantly changing software, and those people almost certainly have different short and long-term goals, contexts, and pressures. Even software in “maintenance mode” has bug fixes, changes in its environment, and changes in the kind of usage patterns it supports.

One way to visualize a software system’s lifecycle is with a simple timeline and markers that represent deploys or changes to the software code and configuration. Let’s use red markers to represent deploys:

This visualization makes it clear that the software is changing over time. The new detail is good, as it prepares us for a mental model that embraces that software is a dynamic, ever-changing system. It still leaves a lot out, though.

The above visualization assumes that a software application lives in a perfect world without dependencies or changes in its environment. That’s not true. It’s impossible to separate operations from software development. So let’s add markers (in blue) to represent some of the changes we see in a software application’s environment:

Same system, but now we’ve added information about changes to the operational environment, like upgrading the version of MySQL that this fictional system uses, increasing the instance size of the AWS EC2 instances that this system runs on, or upgrading the OS kernel. These are all routine tasks when operating a web-based software system running in a public cloud environment, and they should impact the way we think about our software systems. The changes might have been made by the team responsible for deploying changes to code or a team responsible for purely operational work.

Our visualization represents a continuously changing system, both in its software code and the operational environment hosting it. These are both in the team’s control or teams responsible for developing and operating this system, but there is still something missing. What about customers or users of this application? They’re constantly interacting with it in ways that will influence its development and operation. Let’s add more markers (in green this time) that represent significant shifts in how users interact with our application:

Now that we’ve added humans to our timeline, we can see more of the story. Through this lens, it might become apparent that specific changes in user behavior may have motivated some of the operational changes. The ten-fold increase in average requests per second was probably a motivator in increasing the instance sizes. Or was it? The only people who can answer that are the humans who did the work, and we haven’t added them yet. Let’s add staffing changes to the team or teams responsible for this system. We’ll add these as purple markers:

This new dimension adds a fresh perspective to the system. We now know that we can ask Mary about the motivation for changing the instance size. Mary may not know anything about why the team upgraded MySQL when it did, though. In the world Mary lives in, the team has always been running MySQL 8. We know we can’t ask Aditya about the 10x spike in requests because he left the team before it happened. We also can’t ask the five engineers who joined later because that level of traffic has always been customary for them.

When we add human beings to our visualization, it becomes evident that the system is sociotechnical. The humans are doing the work and have the context behind the changes they are making. But where did they get that context? Often it’s from participating in specific incidents. I’m a big fan of the idea that incidents are unplanned investments. The return on those investments is learning. Therefore, the lessons learned from analyzing incidents can reveal important information about your technical systems, organization, and how various parts interact. Let’s add yellow markers for significant incidents into our timeline:

Incidents provide context for the work we do. If we have questions about the MySQL 8 migration, the answers are probably informed by whatever happened in the “MySQL collation” incident. Visualizing our system this way also makes it clear that incidents can be powerful storytelling mechanisms. Investing in good quality incident write-ups can extend the value of incidents for engineers who join after the fact.

Lenses into a System

Each set of markers we added to our system provided new context to form assumptions and frame our thinking. Everything in the visualization existed whether we were looking or not. It becomes clear when looked at this way that each of these dimensions is inextricably linked. It’s impossible to think holistically about software without thinking about the operational environment, or the users of the system, or the people involved in building and maintaining it. These things come together to create another lens through which we can view the world.

It’s important to point out that the final image here is still incomplete. We’ll never fit all of the contexts into a single model. We could keep going, adding more and more context. A fascinating one, for example, would be marking the beginning of the COVID pandemic, when a team that perhaps was colocated started working remotely, and when stress and risk of burnout increased considerably. Otherwise, we’ll eventually include the whole world, but it’s interesting to continually zoom out and see how a new lens helps frame our perspectives.

Embrace Sociotechnical Perspectives

So our systems are sociotechnical. What do you do about this? I think it’s clear that you start by investing in the social parts of the system. If you want to understand why your systems perform the way they do, it’s necessary to know how expertise is created and distributed amongst the people in your organization. The single best way to do this is to invest in incident analysis.

Many organizations have adopted the practice of doing “post-mortems” or “retrospectives” after incidents. Retrospectives are great! Unfortunately, I think a lot of learning is left on the table by the adoption of template-driven processes that produce shallow understandings of what transpired. I’ve spoken about how I think we can improve this. There are also experts in the field who provide training and consulting in incident analysis. There are also communities and companies dedicated to helping you improve this practice.

Investing in incident analysis will empower teams to discover what they know, and almost as importantly, how they know what they know. It’s a necessary part of understanding the social aspects of your system, which hopefully this post demonstrates, are inextricable from any other part.

Questions I Keep Asking

Sat, 29 Aug 2020 08:14:23 -0500

Software is a big space and in the span of a career, you can move in a bunch of different directions, all interesting to different people. As I approach my 20th year as a professional software person, I thought it’d be interesting to reflect on how I hope to spend the next 20 years. I found it useful to write down questions I keep coming back to and use them as a guide for my continued learning. I’ll be interested to come back to this post and see how it compares to whatever “future me” gets up to.

Some of these questions have answers and I just need to learn more about them. Others are active research areas with no “right” answer. Regardless, they’re things I keep thinking about, reading about, and occasionally applying as a practitioner.

Of course, these aren’t the only things I’m interested in, they’re what I think about the most. If anything, this exercise was a good way to make clear the areas I’m not interested in enough to dedicate serious time to. For example, I think that data visualization is an amazing topic. In a world with infinite time, I’d love to dive into it, but probably not in this career. So, with that said, here are my questions:

1. How can we ensure that data flows through a system in ways that are performant, scalable, and reliable? How do we verify the integrity of data, especially in increasingly large-scale and complex data systems?

This is a broad area, but it’s useful as a guide. My core technical interests gravitate towards backend engineering and this is why. I think that most of what we do when working on 24/7 internet services boils down to accepting requests to retrieve or mutate data. Retrieving data necessitates that we do it quickly and correctly for some definition of correct (I’m really interested in when that definition becomes flexible, see Question 4 below). When mutating data, we want to ensure that a write is either accepted or rejected, i.e., the operation is reliable. It’s rarely acceptable to mysteriously drop writes, it’s sometimes acceptable to repeat writes, and often it’s required that writes happen in a specific order. This is all trivial with one process working at a time but gets really hard when you have large numbers of concurrent operations operating on more data than can possibly fit on one computer, and in a world where you have to be able to tolerate the failure of one or more computers.

The fun thing about this question is how broad it is. You can start at a really high level, with systems design concepts, and deep dive in various directions. For example, load distribution and queuing theory are complete topics in their own right. You can also study the data structures used to store data in memory and on disk, to the protocols used to make decisions about how and where to store data, all the way down to lower-level concepts like filesystem and network performance. At every path, there are tradeoffs to consider and I doubt anyone could ever be considered an expert in all these areas, but it’s really fun stuff to think about and has real implications.

2. How can systems be built to maximize operability? What techniques can be followed to allow operators to observe the behavior of a system, make decisions about its operation, and act on those decisions?

I want to help make life better for those of us who keep things running, as well as the people who depend on the services we build and operate. What are the properties of a system that make it easier to operate compared to the properties of a system that operators fear? Luckily, I have a job that allows me to work directly on one aspect of this problem: Observability. Observability refers to the degree to which a system’s internal state can be inferred by looking at external outputs. Doing this well requires a lot of thought and context.

This question also touches on complexity in systems. It’s intuitive, but often wrong, to try and remove complexity — as system designers, we have this impulse to try and hide complexity, but this can often create dangerous situations for operators. Black boxes can have interesting, negative effects. This video, about why cars rarely crash into buildings in the Netherlands, explains it rather well, I think. In software systems, operators often need to be able to respond to operational failures never considered by the designer. Systems can fail in fundamentally surprising ways, and operators need to be able to act on whatever they are seeing.

I also think about the tunability of a system. Is it possible to affect the internal state of software without deploying a change? For example, feature flags are a useful way of toggling functionality and can be used effectively to handle unexpected failures or conditions that degrade functionality. Traffic control is another area where good complexity can be introduced. The ability to direct small amounts of traffic to alternate versions of running software can be tremendously helpful to operators responsible for rolling out changes safely. Being able to make quick decisions about configuration changes, for example, the version of a piece of software that a part of a system should run is crucial in my experience. Chaos engineering is another area where I think introducing complexity can be helpful.

3. How do operators learn about the systems they are responsible for? How does that knowledge help or hinder them in their duties? How do they teach others who are less familiar with the system? How do teams coordinate and make decisions during high-stress incidents?

I believe these are classic Cognitive Systems Engineering questions. Our software systems don’t run themselves, no matter what certain vendors might want you to believe. People keep systems running, and people are constantly doing things, constantly responding to things, and constantly changing things. Yes, change fail rates are a real thing, but mostly, these the actions that people take are what prevent outages or prevent outages from being worse than they are.

Have you ever worked with someone or a group of people who seemed to have an intuitive sense of what needed to happen when a certain alert fires or a graph doesn’t look right, or when a customer complains about a bad experience? Have you noticed how much you can learn by just being around that person or on their team? I’m fascinated by how operators obtain, continue to shape, and share that knowledge. I’ve heard this process referred to as sense-making and I think it’s an under-appreciated aspect of how we do our jobs.

I also know that we generally have different mental models of systems, and it’s usually true that none of them are correct, so the ways in which we use those different mental models to build shared understanding is fascinating to me.

People have written extensively on this subject, specifically in the context of software operations and outages and I want to dig in a lot more.

4. What are the tradeoffs when considering the needs of consistency, availability, and scalability? In order to guarantee coordination, is it necessary to introduce limits that interfere with a system’s ability to scale? When is coordination unnecessary?

These are classic distributed systems questions and I find them really fun to think about. They come up in practical applications from time to time. If you think about data storage or any part of a distributed system that has to make decisions about how to handle a certain kind of operation, we run into some interesting constraints when considering needs of scale and availability. By scale, I mean handling volumes of data that can’t possibly be stored on a single disk, and by availability, I mean that the system will need to tolerate individual node failures, so data will have to be replicated. In such a system, how many nodes have to agree that a write has been accepted before a client can consider the operation successful? When adding nodes to a system, how is the performance impacted? Is it possible to create a system that requires no coordination, but can make safe guarantees about consistency? This is an active research area, and it’s fun to read papers from companies like Google, Facebook, and Amazon to see how practitioners are thinking about the various tradeoffs or fun ways to cheat. There seems to be a lot of back and forth between academia and industry here. CRDTs, for instance, is an attempt to get around the need for coordination while guaranteeing strong, eventual consistency. The Google Spanner team is no doubt doing some really interesting work here. It’s fascinating and I feel like you could spend a lifetime learning about all the work being done in this area and still not be caught up.

5. How do organizations learn? How do organizations internalize information necessary to guarantee their future success?

So you have a large scale data-driven system that incorporates a number of design decisions intended to prevent or isolate failures. You have groups of people who are constantly learning about and changing the system and coordinating their efforts in surprising and remarkable ways. How does an organization internalize this knowledge and avoid or make less likely certain organization failure modes? How do new members of a team learn about all the folk knowledge held within an organization? I’ve worked in organizations that were dangerously dependent on individuals, and not able to withstand commonplace things like vacation or attrition. These kinds of conditions are hell and lead to burnout for individuals, deteriorating quality for a product, and less reliability for users. I’m fascinated by the differences between teams who disseminate this knowledge well and teams that do not.

The next 20 years

Reading all of this, I could probably summarize my interests as: “building large scale distributed data systems that are safe to operate and discovering ways that teams internalize and share their operational knowledge.”. It’s fun to break it down further though, and useful as a guide for learning. As I described earlier, software engineering is a huge space with a seemingly infinite number of paths to explore. It was a useful exercise for me to write down the questions that seem to guide my learning and my career. I hope the next 20 years of my career afford me more opportunities to study the areas I’ve outlined, learn from my peers, mentors, and coworkers, and generally improve the software I work on. I feel incredibly lucky to work in a space with so many intellectually challenging problems, while also getting to help build products that people find useful.

Production Oriented Development

Mon, 30 Dec 2019 08:14:23 -0500

Throughout my career, I’ve developed some opinions. Some have worn particularly deep ruts, reinforced by years of experience. I tried to figure out what these had in common, and it’s the idea that code in production is the only code that matters. Staging doesn’t matter, code on your laptop doesn’t matter, QA doesn’t matter, only production matters. Everything else is debt.

This perspective probably comes from years sitting in between operations and product development. I strongly believe that teams should optimize for getting code to production as quickly as possible as well as responding to incidents in production.

This idea, and a lot of the practices it implies, can be counter-intuitive or controversial, so I want to dive into them a little further. What follows is a set of practices and principles I believe are true, considering my underlying belief that code working in production is the only code that matters.

1. Engineers should operate their code.

Engineers are the subject matter experts for the code they write and should be responsible for operating it in production. In this context, “operating” means deploying, instrumenting, and monitoring code as well as helping to resolve incidents related to or impacting that code. The responsibility of operating code aligns incentives - it encourages engineers to write code that is observable and easy to debug, and connects them to what customers really care about. It encourages them to be curious about how their code is performing in production. Importantly, engineers should be on-call for their code - being on-call creates a positive feedback loop and makes it easier to know if their efforts in writing production-ready code are paying off. I’ve heard people complain about the prospect of being on-call, so I’ll just ask this: if you’re not on-call for your code, who is?

If you’re not currently on-call for your code but want to be, and can help influence this decision, there are some things you can do. Set up PagerDuty (or similar) schedules for each group of engineers responsible for specific services or parts of your code. A good schedule has 6–8 engineers. There are plenty of variations, but a typical template is to have one-week rotations, where you’ll be on-call for secondary for a week and then primary for a week. Configuring alerts is a separate topic, which probably deserves it’s own blog post entirely, but focus on things that impact your customers (see: Symptom-based alerting) and remember that you’re ultimately responsible for how you respond to alerts, which means you can change them.

There are two talks I’d recommend watching that touch on the topic of configuring alerts: Liz Fong-Jones talks about SLOs in Cultivating Production Excellence and Aditya Mukerjee does a great job talking about techniques for managing alerts in Warning: This Talk Contains Content Known to the State of California to Reduce Alert Fatigue.

2. Buy Almost Always Beats Build

If you can avoid building something, you should. Code is the most expensive way to solve a problem that isn’t addressing a core area of your business. For most small to mid-sized companies, there are open source or better yet, hosted solutions that solve a wide range of common problems. I mean things like git repository hosting (Github, Gitlab, Bitbucket, etc), observability tooling (Honeycomb, Lightstep, etc), managed databases (Amazon RDS, Confluent Kafka, etc), alerting (PagerDuty, OpsGenie, etc) and a whole host of other commodity technologies. This even applies to your infrastructure - if you can help it, don’t roll your own Kubernetes clusters (side note: do you even need to use Kubernetes?), don’t roll your own load balancers if you can use Amazon ELB or ALBs.

Unfortunately, NIH syndrome is very real and some companies get burned badly by this. I’ve seen teams light time and money on fire reinventing components when better, more battle-tested alternatives exist in the market. Those same teams almost always end up spending years contending with the resulting technical debt. If you’re on such a team and have the will and ability to impact change, start rolling back these decisions one by one. Migrate your databases to a managed provider, migrate your feature flagging system to a SaaS tool (i.e. LaunchDarkly). Keep going until the only software you maintain yourselves is the software that delivers value to your customers. You’ll be much, much better off for it.

3. Make Deploys Easy

Deploying should be a frequent and unexciting activity. Engineers should be able to deploy with minimal manual steps and it should be easy to see if the deploy is successful (this requires instrumenting your code for observability, which - tada - is covered above), and it should be easy to roll back a deploy if something doesn’t go well. Deploying frequently implies that deploys are smaller, and smaller deploys are generally easier, faster and safer.

Many teams implement periods where deploys are forbidden - these can be referred to as code freezes, or deploy policies like “Don’t deploy on Fridays”. Having such blackout periods can lead to a pile-up of changes, which increases the overall risk of something going very wrong.

If you’re on a team that fears deploys, dedicate a percentage of your engineering time to improvements in your deployment pipeline until the fear is gone. On a recent team I worked with, we were able to improve deploy times from 3 hours to 30 minutes, which drastically improved the teams’ confidence in the deploy process. A natural side effect of this was that engineers started deploying much more frequently instead of waiting for changes to pile up enough to warrant a “release” (which was synonymous with a deploy).

The book Accelerate has been getting a lot of attention. If you haven’t read it, I’d recommend it. The team behind it also publishes the State of DevOps reports, which are full of well-researched information about what various companies in the industry are doing. It’s not a coincidence that two of the four key metrics that the book focuses on are directly related to this (Deploy Frequency, Change Lead Time). Shipping is your company’s heartbeat.

4. Trust the People Closest to the Knives

The people who work with a system are the ones who understand it best. This applies to any part of the socio-technical systems within which we all work. In the case of software systems, the engineers who deploy every day and are on-call for critical services understand the level of risk they operate in. A sad trend is that managers tend to overestimate their teams’ progress on certain transitions - i.e. cloud-native, DevOps, etc. The higher up the management chain, the larger this overestimation tends to be. Engineers who deploy and get paged when things break know where the bodies are buried and they know what needs the most work. They should, therefore, be the primary stakeholders responsible for prioritizing technical work.

Another manifestation of this principle applies to platform or services teams. If you’re responsible for building some shared component that’s used within your organization (i.e. a messaging system, ci/cd infrastructure, shared libraries or services) there’s an uncomfortable truth lurking for you: the people who use your work know more about it than you do in many cases. They understand implicitly how it serves customers and they know what contortions or hoops they have to jump through to get it to work. Listen to them for clues on how to improve the UX of your services and tools.

5. QA Gates Make Quality Worse

Many teams have a manual QA step that gets performed before deploys. The idea, I guess, is to have someone run automated or manual tests to verify that a set of changes are ready to be released. This sounds like a comforting idea - having a human being (or team of human beings) “verify” a release before it goes out - but it falls victim to several false assumptions and creates some misalignments that do more harm than good.

First of all, if there’s manual work that needs to be done before a deploy can go out, that creates a bottleneck - if you’re making deploys easy, and deploying small changes frequently, no QA team is going to be able to keep up testing every deploy, and will inevitably block teams from deploying. That’s no good. If you have manual tests, automate them and build them into your CI pipeline (if they do deliver value).

Secondly, the teams doing QA often lack context and are under time pressure. They may end up testing “effects” instead of “intents”. For example, I’ve seen QA teams burn time testing that when something happens in a UI, something related happens in a database. What happens when an engineer refactors that UI component and changes the underlying data model? The functionality works, but the test breaks. Because two teams are involved, this takes coordination and time to fix. Similarly, I’ve seen QA teams block deploys because of failing tests when caching was introduced at the CDN layer - a TTL of 5 seconds on an activity feed may not ever be noticed by a user but it might break QA tests causing unnecessary conflicts between product and QA engineers.

Luckily, solving this one is easy. Instead of having a dedicated QA team work on creating manual and automated test cases that run in a fictitious QA environment, reassign that team to work on continuous testing in production. Instead of being a gate for deploys, a QA team could continuously verify that production is working as expected. QA teams are also well situated to lead Chaos Engineering initiatives, where faults are intentionally injected in production. QA engineers could also work on making the CI/CD pipeline more reliable, so that deploys are no longer a nightmare.

6. Boring Technology is Great.

With thanks to Dan McKinley, always strive for boring tech when possible. Systems are inherently unpredictable, and you want a wide area of expertise to fall back on when shit goes sideways. There are also routine operations that you’ll have to do (deploys, database migrations, etc) and it’s Very Nice to have widely used and tested tooling for this stuff. I think of databases most often when I think about this belief. MySQL is a database with many, many quirks, but it is so widely used, that you should still just use it most of the time.

Very few organizations have the bandwidth to debug unique problems. You don’t want unique problems, especially when performing routine operations - i.e. storing bytes on disk, choosing a new leader in a cluster, garbage collecting objects, querying time-series data, etc. Having unique problems will kill a small to medium size team. It will sap you of your creative energy, which is better used creating value for customers who want to pay you monies for your software. Use your innovation tokens wisely!

Using boring technology means you can lean on a large community of users. Shit on it all you want, but there are very few PHP issues that someone else hasn’t already encountered. Nowadays, the same is probably true for sufficiently widely used versions of Ruby on Rails. I often say that I like to be in the 3rd cohort of technology adoption. The 1st cohort is the bleeding edge organization. The 2nd cohort is the people who feel like they can take some risks. Let those two groups go before you, run into all the big problems, and then you can go, benefiting from all of their hard-won experience.

7. Simple Always Wins

I don’t have much to say about this, but we’re all writing YAML and JSON instead of XML and we’re all using HTTP instead of CORBA, RMI, DCOM, XPCOM, etc. Right? In that same spirit, I’d rather debug problems in a LAMP stack than a Microservices architecture any day.

Quick sidebar on Microservices: as with so many trends in tech, they are often sold as a panacea. Let me be clear: Microservices, designed well, solve some specific problems and as with most solutions to complex problems, involve several trade-offs. If you are going in this direction, I do have opinions on how you should do it, but I also think you should hold off for as long as you can.

8. Non-Production Environments Have Diminishing Returns

A more direct heading for this section would be “Non-Production Environments are Bullshit”. Environments like staging or pre-prod are a fucking lie. When you’re starting, they make a little sense, but as you grow, changes happen more frequently and you experience drift. Also, by definition, your non-prod environments aren’t getting traffic, which makes them fundamentally different. The amount of effort required to maintain non-prod environments grows very quickly. You’ll never prioritize work on non-prod like you will on prod, because customers don’t directly touch non-prod. Eventually, you’ll be scrambling to keep this popsicle sticks and duct tape environment up and running so you can test changes in it, lying to yourself, pretending it bears any resemblance to production.

9. Things Will Always Break

It’s impossible, even undesirable, to avoid failure. Lean into the fact that failure is inevitable, and focus on how you respond to it. This means investing in a continuously improving incident response process. There’s no one-size-fits-all for every company and team, but you should have a good idea of what to do when things go wrong, and you should have mechanisms in place to learn from those situations and improve your processes. Invest in Incident Analysis. It’s a huge field with lots of valuable tools and resources for maximizing the return on investment when incidents occur (or don’t!).

This is an area where Chaos Engineering can be helpful. Injecting failures into production can improve confidence in how to respond when a system starts behaving in unexpected ways. Game Days can be a particularly effective way to allow a team of engineers to practice various outage scenarios.

Conclusion

A lot of the beliefs outlined in this post are at least counter-intuitive, if not somewhat controversial, but I’m nevertheless convinced that they’re true. That doesn’t mean my mind cannot be changed, but it is unlikely. If you strongly agree or disagree, I’m on the internets. I’d be very curious to hear about your experiences.

Newbie's Guide to Cyclocross

Sat, 26 Nov 2016 08:14:23 -0500

For most of us in Central Texas, the cyclocross season ended with the Texas State Age & Skill Based Championships in Georgetown, TX. This was my first season racing cross, so I thought I’d take a crack at writing a guide for people thinking about getting into it. Cyclocross is an incredibly fun, completely insane sport. If you’re thinking about racing hopefully this will help.

I can only speak to my own experiences, so the following advice pertains to local USAC races in Texas. Other races may be similar, they may not be. The usual disclaimers apply - I am not an expert, take all of this with a grain of salt. Also keep in mind that I placed in the bottom 50% in every race I entered this year - I’m here to tell you how to survive and have a lot of fun, I cannot give you any advice on how to win!

Skills Clinics

Before your first cyclocross race, try going to a skills clinic or two. As the season nears, keep an eye out for clinics organized by your local bike shop or cycling club. This isn’t strictly necessary (A lot of people who regularly beat me never attended a clinic) but I found it helpful.

I was lucky enough to catch one put on by none other than Katie F’n Compton organized by the Texas Cyclocross Project. Katie discussed topics like choosing tires, tire pressure, race strategies as well as various cornering scenarios. We practiced a lot of cornering, riding off-camber and did a bit of riding on sand. I fell a few times, learned that riding off-camber is really f’in hard, and left feeling a lot more confident on my bike.

If you can’t attend a clinic in your area, go to a park and practice mounting, dismounting, run-ups, starts and riding off-camber. You’ll want to see people do this stuff properly - YouTube is your friend.

Local Practice

After you’ve attended a skills clinic you’ll probably want to ride as often as you can. Most local cycling clubs that cater to CX will host a weekly practice. Talk to folks, look at club Facebook pages, event listings, etc and find one that’s open for anyone to join. In Austin, Bicycle Sports Shop have practices on Wednesday evenings and Embros Bicycle Club have one on most Sunday mornings. Double check those days / times - they might change next year.

Practices, like shop rides, are a great way to learn from more experienced riders and meet people in your local cycling community.

YouTube / GoPro

GoPros are a blessing for those of us who are too nervous, lazy, busy or poor to experience something firsthand. I found it really helpful to watch GoPro videos on YouTube from people who have raced certain courses in previous years. My first race of the season was the Cyclocross Scuffle in Elgin, TX and searching YouTube paid off.

If you want to see what winning a race looks like check out Brandon’s video of the Cat 4/5 race at BSS Six Shooter this year.

What to wear

People talk about skinsuits for cyclocross. I wouldn’t bother. Just some bib shorts and a jersey will do. Do invest in some decent full fingered gloves with some padding though - you’ll be grabbing lots of things (beer, your bikes downtube, your top tube, beer) and you’ll want some padding when you’re on your brake hoods. Other than that, regular cycling attire will do - sunglasses included. You probably won’t need a water cage - if it’s hot and you need water, just stick a bottle in your jersey pocket. Water cages make throwing your bike on your shoulder a little harder anyway.

Registering for your First Race

Okay, so you’re ready to race, now what? In Texas, local races can be found on bikereg.com, txbra.org/events and a few other places. You can pre-register or you can usually show up the day of the race and register onsite. I’d go ahead and pre-register — it’ll be cheaper and some races use pre-registration dates to help determine your starting position.

To participate in a USAC sanctioned race you’ll need a USAC license. You can either buy a one-day license for this specific race or get yourself an annual one. If you know you’ll be doing more than one race in a season you might as well get the annual license. You’ll save money and there are other perks, like being able to use the USAC app and track your race history online. Also, you’ll never get bumped up in categories if you don’t have a regular license.

You might wonder what all these categories mean. All you need to know is that men start at Category 5 and women start at Category 4. You can read more about Cyclocross Categories on the USAC website.

Once you have your license, go ahead and register for your first Men’s Cat 4/5 or Women’s Cat 3/4 race.

Get your Race Number

I’ve participated in running races and triathlons in the past, so was used to showing up just 10-15 minutes before my race begins. Cyclocross isn’t like that. You want to show up early enough to be able to pre-ride the course and pick up your race number. Before you do anything else, find the registration area and show them your USAC license number and some id. You’ll get your race number and some safety pins. Your race number goes on your right with the bottom of the number closer to your stomach (so race officials can easily read it).

Watch a Race & Pre-Ride

I like to show up a few hours before my race starts. You get a chance to chat with people, scout the course, pre-ride and watch another race. A lot of the races will have Men’s Cat 4/5 racing later in the day so if you’re lucky you can catch the pro races to see how it’s supposed to be done.

There’s usually 10-15 minutes between races. If you time it right, you can get on the course and do a pre-ride lap. This is super useful especially on technical courses. You can get a feel for the more technical parts and see if you need to make any adjustments to your tire pressure. I also use the pre-ride to decide if I’m going to ride or run some parts of the course.

Falling

You’ll fall. Don’t worry about it.

Heckling

Heckling is a big part of cyclocross. Consider it mental training - you’ll nail that 20th place finish despite their attempts to phase you!

Mechanical Trouble

You’ll probably have some mechanical trouble during your season. I had a rear derailleur malfunction that resulted in me shifting my chain into my wheel at Six Shooter and I had some trouble with my crank after a crash at Webberville. I’m not looking to podium, so whatever – but some people will keep a spare set of wheels or even a spare bike in the pit for things like this. In that event, you’ll get your broken bike to the pit (you may have to run for a good bit of a lap) and swap out what you need.

This is also where I’ll suggest you get to know the mechanics at your local bike shop. Having a good local with friendly people who care about their customers is really important when you’re gonna mess up your bike on the regular. It’s also a good idea to get your bike checked out a few times during the season just to make sure things like spoke tension are good - cross takes a toll on your bike. I’m lucky enough to live close to Cycleast and Russell, Jacob, Blake and Jarod all helped me out in some way this season. They also built me some amazing wheels for my cross bike. If you’re in Austin, consider giving them your business.

Surviving 30 Minutes of Hell

Cyclocross races are 30-60 minutes of hell. This isn’t like triathlon or a stage race where you pace yourself, it’s going to be full on for the duration of the race if your fitness allows it.

Prepare for this if you can. Do interval work and threshold training during training rides. In Texas most races will be hot (this is supposed to be a winter sport, right?!) so acclimate to the heat through the summer. Running helped me out here. My first race had 11 people DNF (did not finish), so it’s not uncommon for people to get caught off guard by the heat. It’s rough, for sure.

To drive the point in, here’s my heart rate on a regular 25 mile training ride:

And here’s my heart rate at this years Cross of Ages Cat 4/5 race:

So yeah, expect to floor it for the duration of the race.

The Start

About 10-15 minutes before the race begins, you’ll want to get to the staging area. Just look for where the crowd of cyclists are gathering near the start. The race director may or may not go over a few things before the race (confirm the time, number of laps, etc). A few minutes before the race they’ll start doing call-ups. These are usually based on USAC rankings and pre-registration time. After the last call-up, they’ll say “everyone else”. If this is your first race, this will probably be you. One of the hardest things about this sport is getting over your starting position - just focus on picking people off one at a time if you can.

Barriers

Barriers are one of the most common obstacles on a cross course. These are usually roughly 30 - 40cm wood stands. Some rare creatures are capable of bunny hopping these - if you’re able to get this down, it’ll definitely help shave some seconds and get some oohs and aahs from the crowd. For the rest of us mere mortals, you’ll want to swing your leg over your bike, dismount and jump over the barriers carrying your bike like a suitcase, then remount. As with most things, YouTube is your friend here. Also practice. Lots of practice. My worst crash of the season was the result of coming in too hot to some barriers at Webberville CX. I couldn’t find my balance when my feet hit the ground and I went crashing bike first into the ground, bending my crank. That sucked.

Run ups

Most courses will have one or two steep hills you have to get up. The general rule of cyclocross is: do whatever is fastest. If you can ride the hill and get traction and this is the quickest way, do that. If you can’t, you’ll want to dismount, throw the bike on your shoulder and run up the hill. There’s a specific way to hold your bike that involves putting your arm under the down tube and grabbing your handlebars. Obligatory YouTube link.

Here’s me running up a hill at Six Shooter this year, holding my bike the wrong way:

Other skills you’ll want to practice are cornering, riding off camber and dealing with things like ditch crossings. Cornering especially is an extremely important skill - the main advice I can give is keep your head up, look where you want to go and use your bodyweight. You can shave a second off every turn if you get good at this. It’s the thing I struggle with the most and I often find myself stumbling or riding into the tape on sharper turns.

Handups

Handups were a surprise to me. As I mentioned, my first race was the Cyclocross Scuffle in Elgin, TX and it was 100F. I’d raced triathlons before where outside help is a big no-no, but I noticed all these people holding out cups along the course. After crashing in a ditch, I got my bike up and decided to take one of the cups thinking it was water - chugged it and to my surprise discovered that it was actually beer! At first I thought this was a terrible idea (beer dehydrates you!) but it was actually lovely and I think it gave me a bit of a boost. Handups are your friend, especially when you’re losing anyway.

Beware though - at Georgetown I grabbed a cup of something, chugged it and realized it wasn’t beer at all, but pickle juice with some kind of alcohol. That was gross - I was so happy when someone handed me beer later on the course.

Have fun

Cyclocross can be a bit intimidating at first, but hopefully some of what I’ve put here will help you out. It’s an amazingly fun and crazy sport and I’d heartily recommend it to anyone who likes riding a bike and is at least a little crazy.

Photo Credits: Corvin Alstot & Jim Hicks

Race Report - Jack's Generic Triathlon 2016

Tue, 09 Aug 2016 08:14:23 -0500

Early in the year, when I decided to sign up for my first triathlon, I remember thinking that I would do a Sprint and Olympic distance and that’d be it - mission accomplished. That was before I was addicted - a week after that first race, I went looking for more triathlons to sign up for and found Jack’s Generic Triathlon. Jack’s celebrated it’s 14th anniversary this year, so it’s been a staple in the Austin triathlon scene for a long time. The name comes from the race’s emphasis on athletes and the race itself rather than sponsors, big name participants, etc.

JGT is held at Lake Pflugerville, a venue I’m pretty familiar with having raced in the Lake Pflugerville Sprint Distance Triathlon in June. It’s also become a favorite spot of mine for swim / bike / run workouts. Lake Pflugerville is well suited as a venue for multi-sport races. The beach has a good swimming area, most of the roads in the area are low traffic and the lake has a trail that goes around it for about 3 miles. The only difficulty is that the lake is plagued by fast growing hydrilla that can make swimming… interesting.

Race Day

I arrived at Lake Pflugerville at 6:20am and was directed into the parking area (basically the grass beside the parking lot). I got my bike and my bag and headed towards the transition area where volunteers were ready with markers to do body marking. I got my number and headed into transition - volunteers were checking wrist bands to make sure that only athletes were entering the transition area.

Transition closed at 7:00am and athletes started congregating on the beach. After some brief stretching we were able to go for a very quick warmup swim. High Five Events had been at the venue in the days before clearing out the hydrilla and they managed to clear out a nice out and back course. Thank god because that stuff was annoying during practice swims in the lake.

The warmup was a bit hilarious. I swam out to the first buoy, then turned around and saw the horde coming right at me — uh oh, I didn’t think this through! Luckily I managed to get to shore without getting hit in the face by an oncoming swimmer.

The Race

The race started promptly at 7:30am with the open wave, followed by all intermediate women, followed by intermediate men 39 and under, which is my swim start group. Swimmers started one at a time, so once again I avoided the dreaded mass start. The swim course involved two 500m loops with a short (and I mean like 10 feet short) run on the beach between the first and second lap. I managed to stay on course for the most part (only swimming an additional 80 meters according to my Suunto watch) and only got punched in the head a few times. I even managed to pass a few people without incident. I wanted to finish the swim in 20 minutes — I based this on what it would take to make a top 10 finish in my AG last year. I ended up doing it in 21:16, which was 10th amongst M 35-39. So far, so good.

My T1 got off to a rough start - I remembered a specific curb that my bike was close to, unaware that it was just like another curb nearby, so I ran right past my bike and got a little lost - probably wasted about 30 seconds or so. Once I got to my transition area things went smoothly. This was my first race going sockless in my new Giro Inciter tri shoes, which saved me some time. Anyway, off I went on the bike. 2:40, also 9th in my age group.

The bike portion went alright. The course is really great. There are some really nice curving turns on Jesse Bohls Rd and Engelmann Ln that almost feel banked so you can pick up speed as you lean into them. I managed to pass quite a few people on those turns. Overall I only got passed by a few people on expensive tri bikes and huge calf muscles, so I feel like I did alright. My average speed was 19mph, with a time of 1:22:11, 12th in my age group. I’ve got a bit of work to do on the bike, so I’m going to do more interval workouts to try and bring my speed up. Also, the new shoes have one small annoyance - the velcro strap on my left shoe kept rubbing against my bikes crank. Not a huge deal, but I had to reach down and fix it a few times, which probably took a few seconds off.

My T2 went better than the T1. 1:25, 6th in my age group. Going sockless and having elastic laces really helped here - all I had to do was pull off my Giros and snap on my Mizunos and I was off.

I started the run course with a quick stop at the aid station to down some water and gatorade and then I took off, trying to set a sustainable pace. I’ve run this course a bunch of times before and while it’s a great trail around the lake, it’s mentally challenging. There are parts of the course that weave away from the lake and make it seem longer than you might otherwise expect and there’s almost no shade. My plan was to try and ignore how far I had to go and just look down and straight ahead. My brain worked against me a few times here and I ended up thinking a bit too much about the distance and heat. According to the chip time, I finished the run in 53:04 or 8:51/mile. Again, 9th in my age group, but still much slower than I’d like. I feel like I should be able to manage something closer to 8:00/mile based on my training paces. Something to strive for next year. Going sockless in my Mizunos took a toll - in the second lap I felt my feet a lot and started noticing friction burns and blisters. I didn’t let it slow me down but I’m now sold on tri-specific run shoes and will probably shop for some before my next race. I walked a few times during the run, but tried to keep it to aid stations and ran solidly for the last mile. I started to feel nauseous a few times which might have meant I wasn’t hydrating properly. The laps are a little less than the advertised 3 miles - my watch clocked them in at 2.8 or so, so this was actually something like a 5.7 mile run, not a 6 mile run.

Crossing the finish line felt great - I find courses with laps mentally challenging but there’s something great about that final lap when you get to run towards the “Finish” sign instead of the “2nd lap” sign. I crossed the finish line and was handed my finishers medal and water bottle. I saw some familiar faces at the finish line - my training buddy Holland had finished a minute ahead of me and I ran into Shane from RunLab who’s helped me overcome nagging shin splints this season. If you’re in Austin and have any running related injury or performance concerns, check out RunLab, they’re awesome folks. After collecting myself, I headed to the beer tent for the best tasting Fat Tire I’ve ever had.

My total finishing time was 2:40:38, which placed me 10th in my age group - I’m really happy with a top 10 finish and while I’ve got areas that I really want to improve on next year, I’m proud of this time especially for my first year doing tris.

Overall, had a blast and highly recommend this race. It’s really well organized - kudos to the good people at High Five Events, all the other participants in the race and especially to all the volunteers who must’ve woken up at an ungodly hour to help make it all happen. I’ll be back next year looking to nudge closer to the podium.

Race Report - Rookie Tri 2016

Thu, 19 May 2016 08:14:23 -0500

A few weeks ago I competed in my first triathlon - a super sprint distance race aptly named the Rookie Tri. It’s a 300 meter swim, followed by an 11.2 mile bike ending with a 2 mile run. I’m registered to race an Olympic Distance (1500m swim, 40km bike, 10km run) at the CapTex Triathlon at the end of the month, so I figured this would be good practice. I’m so glad I did it, I had a great time and learned a ton.

Before getting into the specific lessons I took away, I’d like to heartily recommend this event. It’s a lot of fun and the organizers go to great lengths to make it accessible to newcomers. Triathlons are intimidating but this one really isn’t - mostly because so many other people are also first timers, so you shouldn’t feel alone. I never once felt like I should have known something I didn’t.

The swim is the most intimidating part of a triathlon for a lot of people. For most newer triathletes I talk to, this is their weakest discipline and the idea of a mass start seems insane to them. The Rookie Tri gets around this by breaking people into divisions when they register and then having staggered starts. Competitive triathletes can register in the “Open” division meaning they’d like to compete with others regardless of age group. This is the only group that actually does a mass swim. After that there are “Veteran” (meaning you’ve done 2 or more triathlons in the past) and “Rookie” (meaning this is your first or second triathlon) divisions who each start the swim 2 people at a time in their age group. It was a very relaxed way to start the race and made the swim portion a lot less terrifying for me.

People had told me ahead of the time that the bike portion would be hilly. I guess I take a certain amount of hills for granted, because I didn’t find it so bad and used the hills to pass people. Cycling is my strongest discipline, so your mileage may vary. The run portion was mostly on grass and involved doing an out and back, so you would pass the people in front of you. There was a short uphill section of the run, which just made it more interesting.

Another way that the Rookie Tri is particularly beginner friendly is that awards are given out separately for rookies and veterans. This makes it a lot easier to win a spot on the podium. I was only competing with other rookies in my age group, and so I managed to come away with a 3rd place finish! That was a great feeling.

All in all I was really happy with how things went. Here’s a list of take aways:

Race belts are a life saver. When you pick up your race packet, you get your bib number and some safety pins. USAT rules state that you must wear your race bib number for the duration of the bike and run, but clearly it wouldn’t be possible to wear it during the swim (it would either get soaked and soggy, or get torn up if you wear a wetsuit). The idea of fiddling with safety pins during a T1 seemed insane to me, so I took some advice from a more experienced triathlete and opted for a race belt that I left in the transition area with the bib number already attached. All I had to do was clip it around my waist when going from the swim to the bike. I bought one made by VeloChampion and it worked great for me.
Pace yourself. I made two pacing mistakes. I underestimated how hard the swim would be and I killed my legs on the bike. Full of adrenaline, I went out at full force when the swim started. Swimming is my weakest discipline and I remember being surprised when I was passing all these much more confident looking people! Then at about the 150 meter mark, holy hell I started to feel it and needed to tread water and take a break for a bit. The realities of open water swimming hit me (the waves!) as well. The second half of the swim was a lot harder than it needed to be as a result. Once on the bike, I pushed pretty hard which felt good at the time. I passed dozens of people and only got passed by one. I paid for it on the run though, so careful here. My run splits were about 8:20/mile which is slow for me on a 2 mile run, I should have been able to pull off 7:00/mile or faster. The lesson I’m taking away from this is that I need to do more brick workouts so I can better understand what I’m capable of on the bike without killing my legs.
Watch your step. The transition area is going to be hectic. I expected that. What I didn’t expect was that I would step on a bunch of burrs with my bare feet and have tiny barbs in my foot for the rest of the race - that sucked! When you’re setting up your transition area in the morning, take a moment to have look around and make sure it’s not in an area with lots of sharp brush, rocks, etc. You’ll be running there straight out of the swim and you’ll be a bit disoriented, so make sure you pick a spot with a clear path. Additionally, practice making your way to your transition area from both directions depending on the course map. I had it down coming from the swim, but got a bit lost in T2 which probably cost me a minute or two.
Finish strong. It’s easy to start to ease off when you see the finish line. You’re exhausted, you’ve been pushing yourself hard, but now isn’t the time to slow down! With just a few hundred feet to go, I noticed too late that a guy behind me was starting to up his pace and now my “finishing photo” is a shot of that other guy crossing the finish line right in front of me. It’s a race, and all in good fun - we even high fived each other right afterwards, but I do wish that I’d crossed the finish line ahead of him. Lesson learned - never again!

All in all it was a really great race and experience. Multisport is intimidating but I’d heartily recommend the Rookie Tri to any newcomers as a perfect way to get your feet wet (literally!). I’ll be there next year, competing in the Veteran category and I can’t wait.

Looking back at 2014

Sat, 24 Jan 2015 11:40:20 +0000

I was in New Orleans, LA at the beginning of 2014. Fitting, as the Crescent City played a huge role in my life this past year. Meg and I were in town planning our wedding and decided to stay and celebrate New Years in our favourite city.

Our normal abode (and future wedding venue), the Maison de Macarty in the Bywater, was fully booked that night so we were temporarily relocated to the Hilton Riverside hotel. After a nice evening out, we decided to get a bottle of wine and go back to our hotel room before the countdown began. This turned out to be a wise move as when the clock struct midnight, we were treated to a spectacular fireworks display over the Mississippi river. It was the perfect way to ring in what would become one hell of an eventful year.

It’s still January, so I reserve the right to pretend that it’s not unsually late for a reflective New Years Day kind of blog post. 2014 was a big year for me so I want to take some time and properly reflect on it. What follows is a list of highlights recorded mostly for my own sake.

The Wedding

On March 8th, shortly after one of the coldest Mardi Gras on record, Meghann and I got married in the backyard of the Maison de Macarty. We were surrounded by people we love in a city we treat as our second home. It was perfect. Seeing so many of our friends and family in New Orleans was a great feeling. We had people travel from Boston, New York, Toronto, Dublin and beyond. It was truly remarkable and I’ll never forget how I felt that week.

Travel

I got a lot of travel in during 2014. As mentioned, I started the year in New Orleans. Meg and I ended up fitting in 4 trips to New Orleans in total. We also spent a long weekend in Austin.

Work sent me to Tokyo in February. My trip overlapped with Music Hack Day, so I got to catch up with old friends from that scene and I managed to see the Crocodiles who were playing their first show in Japan.

In August, our great friends Erin and Andrew hosted us in Maine. We met them at Boston Logan airport, spent a few days meeting their families and seeing where they grew up, then drove back, stopping in New Hampshire and Vermont on the way.

Meg found ridiculously inexpensive flights to Seoul. We spent a week and a half in total with a few days in Tokyo. It was great going to Tokyo for the 2nd time in less than a year. Seoul was great as well. We enjoyed wonderful hospitality from Meg’s cousin Derek, his wife Jill and their new baby Logan who’ve been living there for a while now. Oh and we ran into Kevin who was also taking advantage of ridiculously cheap airfare.

Fitness

In terms of fitness, I’m calling 2014 “pretty good”. I started cycling again (more than just to and from work) and got into running. According to Strava I managed just over 2000km on the bike and about 170km running.

I went for runs in Toronto, San Francisco, New Orleans, Tokyo and Seoul. I participated in the Ride to Conquer Cancer (200km over 2 days) and a 110km organized ride.

The Ride to Conquer Cancer was especially meaningful for me as I rode with my brother, father and brother in law (and 7 other teammates). I’m riding again this year, hoping to raise $2500 for cancer research. Proceeds go to the Princess Margaret Hospital. Ya’ll should donate.

For 2015 I’m looking to double my total ride distance (4000km) and participate in my first 10k organized run.

Work

Professionally, 2014 was hit and miss. I left SoundCloud at the end of 2013 to start a new position at 500px. I managed to do a lot of great work and I’m proud of what my team and I were able to accomplish. We evolved the architecture of the application, introduced Go as a language for backend service development, moved the organization further towards a DevOps culture, and improved response time while reducing infrastructure costs.

At the end of the year though, I decided that 500px was not the right company for me and that it was time to move on. Nothing dramatic, I still love the people and the product and wish them the best. I started 2015 with a new challenge that I’m very excited about, but that’s for different blog post.

I did two conference talks in 2014. One at DevOpsDays Toronto and another at QCon San Francisco.

In Closing

2014 was a great year for me. I married the love of my life, traveled to great places, accomplished some personal goals and generally had a blast. I was surrounded by wonderful friends, family and colleagues. I feel like I totally lucked out.

Reflecting on the past year may seem like naval gazing, and maybe it is a little, but it also helps me appreciate the things that have happened, and focuses my thinking as I make goals for the next year. Here’s to a great 2015.

Cycling for Cancer Research

Tue, 22 Apr 2014 08:14:23 -0500

On June 7th and 8th, I’ll be joining the Ride to Conquer Cancer — a two day ride from Toronto, Ontario to Niagara Falls (that’s just over 200km). The ride raises money for the Princess Margaret Cancer Centre, one of the top 5 cancer research centres in the world. I’m doing this because the work done at centres like Princess Margaret has a direct impact on cancer patients and their loved ones. The ride funds are directed towards research initiatives that are focused on Personalized Cancer Medicine improving detection, diagnosis, targetting and support.

Fundraising

The ride has a $2500 fundraising minimum, and thanks to the generosity of friends and family, I’m sitting at $340 with a mere $2160 to go.

Here’s the thing: I really hate asking people for money, even when it is going to such a worthy institution. Instead I’ve decided to give things away to people who donate certain amounts. I’ve been quite busy as of late, and came to the conclusion that one of the most valuable things I can give is my time.

About Me

I have 15 years of experience working in the tech industry. In that time I’ve worked for companies like SoundCloud and Mozilla, and I’m now the Director of Platform Engineering at 500px (you can see my LinkedIn Profile for details). I have a huge amount of experience writing software and managing teams of Software Developers with an emphasis on platforms (both internal and external APIs).

You can use this time to pick my brain about anything related to technology. If you’re new to software or considering getting into the industry, I’d love to answer your questions or go over your code with you. If you’re starting a product company, I can give you my honest feedback on your idea. Maybe you just want to bounce ideas off me or just chat about tech, I’m happy to do that too.

The Perks

$30 or more $30 will get you a thank you card from me - I know this is paltry, but I appreciate your support so much.
$100 or more For $100 you’ll get the above plus an hour over Skype / Google Hangouts or in person if you’re in the Toronto area.
$500 or more If you want to go nuts, you get a whopping 8 hours of my time for general tech consulting. I’m limiting this to 3 donations and I’ll update here once they’re gone.

Donate Now

Any amount makes a difference. Cancer is a terrible disease that effects so many of us. If you want to take advantage of any of the perks above, simply include details in the Personal Note section when donating. Donate Here.

Great Engineering Teams

Tue, 31 Jan 2012 08:14:23 -0500

Building great teams is hard. We all know this. Nobody has the perfect recipe for creating a great engineering team, but I’m fairly certain that some things are obvious and should be heeded. What follows is a list of observations I’ve made throughout my career about what seems to work and what doesn’t. Perfect excuse for a good old “Do” and “Don’t” list:

Do: Encourage Side Projects

All good developers run into things that piss them off from time to time. Great developers build tools that help them fix the shit that pisses them off. Give developers space and time to work on things not on the roadmap and they’ll probably end up building tools that solve unseen business problems. Intentionally under-load developers (or go one step further and do away with deadlines) and you’ll start to see deploys run more smoothly, metrics start being captured, open source libraries and tools start being published and everyone is happier for it. Allowing developers time for side projects is an intangible, but I’m convinced it’s an investment that pays for itself and then some.

Don’t: Hire Asymmetric Managers

One of my pet peeves is seeing a shop employ a “manager” who’s job is to pester developers and make sure they meet deadlines. The best managers I’ve worked with always ask two questions during check-ins:

How are you doing?
What can I do to help you?

Good managers know that their job is to enable people to do great things by clearing roadblocks and getting the hell out of their way. If you have a manager who is constantly “whipping” their team into shape, things are going to go very wrong. Hire managers who help programmers as much as possible. We’re all on the same team.

Do: Support Async Workflows

Being a software developer involves blocking out large chunks of time to get into periods of deep concentration. Requiring developers to break for meetings or respond to emails / im messages / impromptu conversations immediately breaks concentration and can ruin a big part of the day. It also screws your remote employees if you have any. Zach Holman and Paul Graham have written about this much more elequently than I could hope to.

Don’t: Hire “B"s

Comprimising on hiring is the quickest way to build a shitty team. It can be tempting to hire some “juniors” that will need supervision, and there’s definitely room for people with a variety of experience, but you need to shoot for the best and hire nothing but. Hiring second rate developers will hurt your productivity and effect the morale of your best folks. Good people like to work with good people. Don’t fuck with that.

Do: Encourage a devops culture

As I mentioned, great programmers scratch their own itches. Hire people who know the whole stack. Great programmers should write great code, but also care about how it’s deployed and write or use tools that help you do that effectively and safely. Having a divide between developers and people who make sure their code can be shipped creates two problems:

Your developers stop thinking about ops.
You create an “us” vs “them” attitude. Developers complain about “conservative” ops people and ops people complain about “sloppy” developers.

Bottom line: making sure your production environment is healthy is everybodys job.

Conclusion

Most of this boils down to respect. If you’re shipping software, you’re an engineering company. Engineering companies need to be managed as such. Don’t treat your developers as second tier. Give them what they need, support them and your chances of building a great culture where people want to come and do great work will be greatly increased.

Building Services with Apache Thrift

Mon, 12 Dec 2011 08:14:23 -0500

I’ve always been interested in cross-language service frameworks. I believe in using the best tools for a job, instead of being limited to a specific language or framework, so being able to write components of a service in whatever langauge makes the most sense is attractive to me. In past lives, I’ve developed software that used CORBA, SOAP, COM and XPCOM and found that they all suck in different, significant ways. Because of this, I’ve been interested in the Apache Thrift project originally developed at Facebook.

At its core, Thrift is an interface definition language, a code generation tool and a set of libraries that take care of serialization and implement the transport protocol.

In order to generate code from a thrift service definition, you’ll need to install the thrift compiler. You can install it with homebrew on a mac or download it from the project site.

Thrift services and datatypes are defined using an IDL that should look pretty familiar to anyone who has worked with CORBA or similar technologies. Let’s say you want to create a service that handles creation and storage of user objects. We’ll write the service in Python and a client in Ruby. This is obviously a contrived example, but imagine that we’re using some kind of data store that has a library written in Python and all of our application code is in Ruby. It’d be great to be able to use the data store from our Ruby code without having to develop and maintain our own library.

We start by defining the datatypes and service interface:

struct User {                                                                                                      
  1:string first_name,                                                                                             
  2:string last_name,                                                                                              
  3:string email                                                                                                   
}                                                                                                                  
                                                                                                                   
service UserService {                                                                                              
  bool add_user(1:User user),                                                                                      
  list<User> get_users()                                                                                           
}

Save the above code in a file called users.thrift, then generate the Python code by running thrift --gen py users.thrift. Because we’ll be writing a client in Ruby, we’ll also want to run thrift --gen rb users.thrift to generate the Ruby code.

Next, write the server in Python. This requires a bit of boilerplate and a service handler that gets passed to the Thrift processor. Notice that thrift threw the generated code in a directory called ‘gen-py’. It’s just a Python module, so you can move it wherever you’d like, but in this example I’ll just modify sys.path appropriately.

There are a variety of options for the server itself. For simplicity, we’ll just use TServer.SimpleServer which is a straightforward single threaded server. There’s also a multithreaded server and a forking server.

import os
import sys

sys.path.append(os.path.join(
    os.path.dirname(os.path.abspath(__file__)), 'gen-py'))

from thrift.transport import TSocket, TTransport
from thrift.protocol import TBinaryProtocol
from thrift.server import TServer

from userservice import UserService


class UserServiceHandler(object):

    def __init__(self):
        self.users = []

    def add_user(self, user):
        self.users.append(user)
        return True

    def get_users(self):
        return self.users


def create_server(host='localhost', port=9090):
    handler = UserServiceHandler()
    return TServer.TSimpleServer(
        UserService.Processor(handler),
        TSocket.TServerSocket(host=host, port=port),
        TTransport.TBufferedTransportFactory(),
        TBinaryProtocol.TBinaryProtocolFactory()
    )

if __name__ == '__main__':
    server = create_server()
    server.serve()

Finally we write the client in Ruby. Ignore the boilerplate client code for now, and focus on the part where we create a user, add it using the service, then get a list of users from the service. This part looks like normal Ruby code which is part of what I really like about Thrift. There are no special types to use, users as returned from the get_users method is a simple Array and everything feels really natural.

$:.push('./gen-rb')

require 'rubygems'
require 'thrift'
require 'user_service'
require 'userservice_types'

def create_client(host='localhost', port=9090):
  socket = Thrift::Socket.new(host=host, port=port)
  transport = Thrift::BufferedTransport.new(socket)
  protocol = Thrift::BinaryProtocol.new(transport)
  client = UserService::Client.new(protocol)
  return transport, client
end

user = User.new()
user.first_name = 'Paul'
user.last_name = 'Osman'
user.email = email

transport, client = create_client()
transport.open()
client.add_user(user)

users = client.get_users()
users.each do |user|
    puts "#{user.first_name}, #{user.last_name}, #{user.email}"
end

transport.close()

I found a few good resources out there, but most were either a bit outdated, or too generic or went into areas that I wasn’t immediately interested in. I just wanted the straightest line to seeing what Thrift was all about, so I thought I’d write that up here.

If you’re curious about Thrift, I would encourage you to read this whitepaper before diving any deeper. It provides a good overview of how Thrift works and why it was designed the way it was. I’m glad to see that Thrift is gaining some adoption, appearing in projects like HBase, Storm and FlockDB.

Storing Extra Data in Django Join Tables

Thu, 27 Oct 2011 08:14:23 -0500

I’ve had some people ask me about the through argument supported by Django’s ManyToManyField class. This option supports a very simple use case: when you want to store additional data in a join table.

Imagine, for instance, that we’re building a simple course registration tool. Let’s call the Django app courses. We’ll define the following models:

from django.db import models

class Course(models.Model):
    """A course offered at our school."""
    name = models.CharField(max_length=100)
    code = models.CharField(max_length=5, unique=True)
    description = models.TextField()

    def __unicode__(self):
        return u'%s: %s' % (self.code, self.name)


class Student(models.Model):
    """A student registered with our school."""
    first_name = models.CharField(max_length=50)
    last_name = models.CharField(max_length=50)
    number = models.CharField(max_length=10, unique=True)
    courses = models.ManyToManyField(Course, null=True)
    registered_on = models.DateTimeField(auto_now_add=True)

    def __unicode__(self):
        return u'%s -> %s, %s' % (
            self.number, self.last_name, self.first_name)

When we run syncdb or generate a migration for these models, Django will automatically create the following tables: courses_course, courses_student, and courses_student_courses. The first two should be self-explanatory. The third table will store information about our many-to-many relationship. Each row will associate a student id with a course id.

Now let’s say we want to store the date and time whenever a student registers for a course. We’d want something like the following model:

class Registration(models.Model):
    """Student registration for a course."""
    student = models.ForeignKey(Student)
    course = models.ForeignKey(Course)
    registered_on = models.DateTimeField(auto_now_add=True)

In order to get Django to use our model as a join table, we can use through by changing a single line in our student model:

class Student(models.Model):
    ...
    courses = models.ManyToManyField(Course, null=True,
        through='Registration')
    ...

And that’s it. Now, Django will automatically generate a courses_registration table and use it instead of courses_student_courses, allowing us to store extra data about the registration.

Making Signups Easier With WebFinger

Mon, 04 Oct 2010 08:14:23 -0500

I’ve been thinking about signup processes lately and what we can do with batucada (the next drumbeat.org) to make it easier, while encouraging users to use open, decentralized identity technologies like OpenID.

The Problem

Whenever I come across a new service that I want to create an account on, I dread the idea of creating a new set of credentials and re-entering a bunch of information I have already entered elsewhere. OpenID solves part of this for me, and I’m always happy to see sites supporting it. The problem, however, is that OpenID only solves one piece of the puzzle (authentication) and there are usability problems abound. The latter problem is well known, and people have come up with a bunch of innovative ways of tackling it, including intelligent OpenID selectors.

The core of the problem is that people don’t tend to think of themselves as a URL, which is what OpenID uses as a supplied identifier. It’s been really hard to get the average web user to think of a URL as part of their identity, and frankly, they’re usually pretty hard to remember. There are ways around this, including hosting your own OpenID provider, or using delegation, so that you can choose an easy-to-remember URL, but it’s unrealistic to expect non-technical web users to be able to do this.

The Solution

Enter WebFinger. The idea behind WebFinger is that, given an email-like identifier, an application can discover information about that user (including their OpenID provider). I really like this idea. I think people have an easier time with email addresses than with URLs. A bunch of providers, including AOL, Gmail and Yahoo have implemented WebFinger support, which means that there are a ton of people out there who already have WebFinger identifiers, even if they don’t know it (and they shouldn’t have to know it).

So what would a WebFinger based signup process look like? The first step would be to capture an email address and find out if it is a valid WebFinger identifier. If it is, the next step would be to find out of the user has an OpenID provider. If their email address is not a WebFinger identifier, the workflow would fall back to a more traditional password based signup form. Imagine clicking on a “Sign up” link and being presented with a form asking for your email address.

Once submitted, the web application would perform WebFinger based discovery on the identifier in the background. If an OpenID provider is found, the user could be presented with a friendly message asking them if they’d like to use it to sign in to the site.

If the email address is not a WebFinger identifier, or if the user opts not to use their OpenID, fall back to a more traditional sign up form.

These are obviously very rough, but I’m interested to know what people think of this idea. Specifically how the process could be made easier for non-technical users. A few concerns I have (and note that UX / Usability is certainly NOT my specialty, so help is appreciated!)

A service provider shouldn’t coerce a user into using an OpenID vs creating a new set of credentials, the user must always have a choice. Unfortunately, this means asking another question, and questions are confusing.
Presuming a user has an OpenID, but does not have an active session with their provider, they are going to be sent to their provider to log in after Step Two. This could be confusing to users. We’re sending them into the OpenID “Chasm of Death”. Is there anything that can be done to ease this transition / manage user expectations?
Once users have signed up using an OpenID, how do we let them know that they should use that OpenID from now on? I suppose we could just prompt for an email address on sign in and perform discovery again. This introduces another step for those who aren’t using an OpenID though (step one: input email address, step two on next screen: enter password).
Users could be confused immediately when we ask for their email address and not a password. This is breaking a familiar pattern, which is always potentially jarring.
What about users who know what OpenID is, and have an OpenID that they’d like to use, but don’t have a WebFinger identifier that refers to their OpenID? Again, more questions to ask, which adds more confusion.

Again, I’m certainly not a UX person, so I’d be very curious to hear what people think of this idea and if people have ideas about how to improve the flow.

Google Webfinger

Mon, 01 Feb 2010 08:14:23 -0500

This is pretty old news, but I missed the original announcement and I think it’s pretty interesting.

Google have implemented an alpha of the WebFinger protocol. If you have a Google profile, click on “Edit your profile” and add ‘webfingeralpha’ as an interest. Congrats, your gmail address is now a WebFinger identifier and will resolve to an XRD file containing information about services you use (if you’ve included that information in your Google profile).

This is pretty fun to play around with. Given an email-like identifier, such as evalpaul@gmail.com, get the host-meta XRD file for the domain:

paul@knuth ~ $ curl -i http://gmail.com/.well-known/host-meta
HTTP/1.1 200 OK
Content-Type: application/xrd+xml
Transfer-Encoding: chunked
Date: Mon, 01 Feb 2010 12:36:47 GMT
Expires: Mon, 01 Feb 2010 12:36:47 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
Server: GFE/2.0
...

<?xml version='1.0' encoding='UTF-8'?>
<!-- NOTE: this host-meta end-point is a pre-alpha work in progress.   Don't rely on it. -->
<!-- Please follow the list at http://groups.google.com/group/webfinger -->
<XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0' 
	xmlns:hm='http://host-meta.net/xrd/1.0'>
<hm:Host xmlns='http://host-meta.net/xrd/1.0'>gmail.com<;/hm:Host>
<Link rel='lrdd' 
		template='http://www.google.com/s2/webfinger/?q={uri}'>
	<Title>Resource Descriptor<;/Title>
</Link>
</XRD>

Now that you have the URI template, get the XRD file for the specific user:

curl "http://www.google.com/s2/webfinger/?q=acct%3Aevalpaul%40gmail.com"

And the response:

<?xml version='1.0'?>
<XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0'>
	<Subject>acct:evalpaul@gmail.com<;/Subject>
	<Alias>http://www.google.com/profiles/evalpaul<;/Alias>
	<Link rel='http://portablecontacts.net/spec/1.0' href='http://www-opensocial.googleusercontent.com/api/people/'/>
	<Link rel='http://webfinger.net/rel/profile-page' href='http://www.google.com/profiles/evalpaul' type='text/html'/>
	<Link rel='http://microformats.org/profile/hcard' href='http://www.google.com/profiles/evalpaul' type='text/html'/>
	<Link rel='http://gmpg.org/xfn/11' href='http://www.google.com/profiles/evalpaul' type='text/html'/>
	<Link rel='http://specs.openid.net/auth/2.0/provider' href='http://www.google.com/profiles/evalpaul'/>
	<Link rel='describedby' href='http://www.google.com/profiles/evalpaul' type='text/html'/>
	<Link rel='describedby' href='http://s2.googleusercontent.com/webfinger/?q=evalpaul%40gmail.com&amp;fmt=foaf' type='application/rdf+xml'/>
</XRD>

EDIT: Google have since rolled out WebFinger support for everyone with a Google Profile. You no longer need to add ‘webfingeralpha’ to your interests.

Introduction to mod_python

Mon, 01 Jan 2007 08:14:23 -0500

This article will provide a brief introduction to mod_python, a tour of what you can do with it, and some pointers to further resources should you want to explore it in more depth. I will assume that the reader is comfortable programming in Python (although no specific knowledge is required) and is familiar with Apache and basic web concepts. This article is not intended to be a complete reference for mod_python. Instead it is meant to consolidate information available from other sources, to hopefully whet your appetite and encourage you to read more from the official documentation (links are in the Resources section).

Prerequisites and Assumptions

I am going to assume that you have a working installation of Apache 2.0.47 or higher and have Python 2.2.1 or later installed. To make things simple, I’m going to assume you are working from the same machine that Apache is installed on, so all URLs will have ’localhost’ as the server name. Replace this with your server name if this is not the case.

What is mod_python?

In the bad old days, most web development was done using CGI (Common Gateway Interface). Writing a CGI program meant creating an executable (script or binary) that the web server called to handle a request. The output generated by the CGI program would then be returned to the user via their browser. Think about that process for a second: a) the user requests a page from the web server, perhaps with some arguments sent via GET or POST, b) the web server recognizes that the requested page is handled by a CGI script and invokes the CGI process, c) the CGI program collects information from the web server using some mechanism (usually environment variables), does some processing and prints out a bunch of HTML (usually), if everything went alright, the web server takes the output and sends it to the user.

Sounds pretty cumbersome when you think about it, doesn’t it? It’s not only cumbersome, it’s also slow and very error prone. mod_python saves us from having to go through this process by integrating the Python programming language right into the Apache HTTP server. This provides a much faster way for Apache to execute python handlers, and as an added bonus, gives us complete access to the Apache internals. Imagine mod_python as a little guy (or … a Python?) stuck inside your web server intercepting certain requests and allowing you to do really cool things with them. Okay, so that may not be a very good technical description, but we’ll get to that. Sound interesting? It is!

It’s important to understand that writing applications with mod_python is not the same as writing applications with a server-side scripting language like PHP. Instead, with mod_python you specify handlers in the Apache configuration file(s) that allow you to customize how a request is handled. This allows you to do a variety of neat things like implement protocols other than HTTP, filter the request and response, determine a document’s content-type, etc.

So let’s get mod_python installed and take a tour of how it works. In order to follow along with this tutorial, you’ll need to have Apache and Python installed, and then install mod_python.

Installation

There are a few different ways to install mod_python, depending on what Operating System you are running. If you are using a distribution of Linux that has a decent package management system (any Red Hat / Fedora, Debian or Gentoo based system for instance) then there will likely already be a package available for you to install. On Gentoo I just emerge the mod_python ebuild with the USE flags I want and portage automatically adds configuration files to be included into my Apache configuration. On Red Hat Enterprise Linux I use a mod_python RPM that depends on having the python-devel and httpd-devel packages. For the sake of brevity I’m only going to cover installing from source here. Consult your OS’ documentation to see if there is an easier way for you. (There is a way to install mod_python on Windows but I am not going to cover that here).

Compiling from Source

In order to compile mod_python, you’ll need to grab the latest stable source release from the mod_python website. At the time of this writing, the latest stable release was 3.2.10.

I’m going to assume you have Apache2 already installed (if not you can get it from the Apache website). I am using Apache 2.0.59 but the process should be the same for any version of Apache above 2.0.47. I have Apache installed in /usr/local/apache2. You will need to adjust the path to match your installation.

Once you’ve downloaded the source tarball for mod_python, untar it and run the ‘configure’ script (feel free to run ./configure –help to see what other configuration options are available):

pike:/usr/local/src paul$ tar xfz mod_python-3.2.10.tgz 
pike:/usr/local/src paul$ cd mod_python-3.2.10
pike:/usr/local/src/mod_python-3.2.10 paul$ ./configure --with-apxs=/usr/local/apache2/bin/apxs
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
...

If everything went okay, and you didn’t get any error messages from configure, you should then run ‘make’ to compile mod_python and ‘make install’. You will likely need to run ‘make install’ as root, so you can use “su -c ‘make install’”:

pike:/usr/local/src/mod_python-3.2.10 paul$ make
... 
(a whole bunch of output you can ignore unless you know what you&#8217;re doing)
pike:/usr/local/src/mod_python-3.2.10 paul$ su -c 'make install'
...
(more output that we won&#8217;t concern ourselves with)

Safety Note: Never run a configure script as root. It’s always possible that the host you retrieved the source distribution from has been compromised and therefore you don’t know for sure what could be hiding in that script.

If the compilation process didn’t report any errors, you should be ready to go. Next we have to open the Apache2 configuration file (usually called httpd.conf) and add the following:

LoadModule python_module      modules/mod_python.so

Note: the path to mod_python.so may very. Check your Apache2 installation root and find out if it was actually put there or somewhere else.

As with any time you edit the Apache configuration, you will have to restart Apache before the changes take effect. Now we want to verify that our installation went smoothly. There are a variety of ways to do this, I always like to grab the default headers from Apache to see what is installed. I do this by telnetting into port 80 and typing ‘HEAD / HTTP/1.0’:

pike:/usr/local/src/mod_python-3.2.10 paul$ su -c '/usr/local/apache2/bin/apachectl restart'
pike:/usr/local/src/mod_python-3.2.10 paul$ telnet localhost 80
Trying ::1...
Connected to localhost.
Escape character is '^]'.
HEAD / HTTP/1.0

HTTP/1.1 200 OK
Date: Mon, 01 Jan 2007 04:28:24 GMT
Server: Apache/2.0.59 (Unix) mod_python/3.2.10 Python/2.3.5 PHP/5.2.0
Last-Modified: Sat, 20 Nov 2004 20:16:24 GMT
ETag: "ab5da-2c-4c23b600"
Accept-Ranges: bytes
Content-Length: 44
Connection: close
Content-Type: text/html

Connection closed by foreign host.

As you can see from the “Server” header, mod_python version 3.2.10 is installed. Now to really test it! Open the Apache2 configuration file again and add the following location directive:

<Location /mpinfo>
    SetHandler mod_python
    PythonHandler mod_python.testhandler
</Location>

Restart Apache again and point your browser to http://localhost/mpinfo and you should see a test page with a lot of useful information about your server environment. This information can come in very handy when debugging problems so I tend to keep it around. If you don’t see this page, something must have gone wrong. Go over the instructions again or consult your operating system documentation. If you see the page, congratulations, you have installed mod_python! Now let’s move on and start learning about mod_python.

Handlers

In order to truly understand the power offered by mod_python, a basic understanding of Apache handlers is required. Essentially you can think of a handler as a processing “phase”. Apache gets a request, and then initiates a number of handlers (or “phases”) to do something. The handlers can either be built into Apache, or included as modules. Apache handlers may be configured explicitly, based on either filename extensions or location. Examples of functionality that takes place in handlers may include authenticating a user, invoking a cgi script, getting the server’s status, etc. mod_python allows you to tap into any handler used by Apache. mod_python also provides a few standard handlers to help you with some common tasks.

Publisher Handler

The publisher handler is available so you don’t always have to worry about writing your own handlers and can instead focus on developing your application. It’s very handy. In order to use the publisher handler, you have to something like this to your Apache configuration:

<Directory /usr/local/apache2/htdocs/PublisherExample>
    AddHandler mod_python .py
    PythonDebug On
    PythonHandler mod_python.publisher
</Directory>

Note: The “PythonDebug On” line makes mod_python output errors to the browser when possible, instead of the Apache server logs. This is useful while we’re developing.

Now create a file called ‘ModPythonExample.py’ in the directory ‘PublisherExample’ and type the following:

from time import strftime, localtime

def publisher_example(req):
    req.content_type = 'text/html'
    time_str = strftime("%a %b %d %H:%M:%S %Y", localtime())
    message = "<h1>Hello from mod_python!</h1>"
    message += "<p>The time on this server is %s</p>" % (time_str)
    return message

Because we have modified the Apache configuration we will need to restart it once again. Now point your browser to http://localhost/PublisherExample/ModPythonExample.py/publisher_example and you should see a message telling you what time it is.

As we can see from this example, the publisher handler calls a function and just sends the return value to the client. The function receives a request object as an argument. We set the request object’s content_type to ’text/html’ because we want the output to be handled as HTML. We then construct a message including the current time and date and return it. Notice the structure of the URL. The first part after the directory (ModPythonExample.py) is the name of our file, and the second part (“publisher_example”) is the name of the function to call.

Obviously this is only a simple example. There are many great things you can do with the publisher handler. See the documentation for more information (Resources).

CGI Handler

The CGI Handler is provided as a stepping stone away from traditional CGI. It is not intended as a final solution for using mod_python. Basically, the CGI Handler emulates a CGI environment from within mod_python, allowing you to migrate CGI based Python applications to mod_python with little or no modification. Read the documentation to learn about limitations of this handler. To use it, just add a Directory directive to your Apache config (like we did for the Publisher Handler) and include the following:

SetHandler mod_python
PythonHandler mod_python.cgihandler

Once again, this should not be considered a final solution but can certainly improve the performance of your existing CGI code without too much modification.

Custom Handlers

Using a standard handler may be appropriate in many scenarios, but sometimes you might find it more appropriate to write your own handler. mod_python allows you to do this (of course) and in fact makes it pretty easy! Let’s create the following directory directive in our Apache configuration:

<Directory /usr/local/apache2/htdocs/CustomExample>
    AddHandler mod_python .py
    PythonDebug On
    PythonHandler customexample
</Directory>

This tells Apache that any requests for a file with a .py extension will be served by mod_python. (It is worth noting that the file does not actually have to exist, and that in fact a request for /CustomExample/myfile.py and /CustomExample/myotherfile.py will both be handled the same with this configuration). The “PythonHandler customexample” line tells Apache to hand requests for files with a .py extension to this module (which we will be writing). The actual process goes something like this: a) Apache receives a request for a file in the CustomExample directory that has an extension of .py. b) Apache recognizes that this request is to be handled by mod_python and attempts to import a module called “customexample”. Apache looks for this module in sys.path (with our directory prepended to it so anything in there will be found first). c) Apache will then look for a function called “handler” in the module and execute it, passing the request object as an argument. Okay, enough details, let’s write our handler module. Create a file called “customexample.py” in the “CustomExample” directory:

from mod_python import apache

def handler(req):
    req.content_type = 'text/plain'
    req.write('Hello from mod_python!')
    return apache.OK

A few things to notice here: a) we use the request object to write output to the client instead of just returning content, b) we return a constant from the apache module. The apache.OK constant corresponds to an HTTP 200 response code. Other constants are defined for 404, 302, etc response codes. Of course, this example doesn’t really do anything new, so to demonstrate the real power of writing our own handlers we are now going to create a super simple (and not very secure) MySQL authentication handler. If you have MySQL installed, create a database called “mptutorial” and add the following table and records:

CREATE TABLE `users` (
    `id` int(11) unsigned NOT NULL auto_increment,
    `username` varchar(50) NOT NULL,
    `password` varchar(50) NOT NULL,
    PRIMARY KEY  (`id`),
    UNIQUE KEY `username` (`username`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

INSERT INTO `users` (`username`, `password`) VALUES ('john', MD5('secret'));

Then add the following to your Apache configuration:

<Directory /usr/local/apache2/htdocs/AuthenticateExample>
    AddHandler mod_python .py
    PythonDebug On
    PythonAuthenHandler authuser
    AuthType Basic
    AuthName "Secure Area"
    Require valid-user
</Directory>

Now let’s write our example code. We need to write an authentication handler that retrieves the username and password entered

Note: This example uses the MySQLdb module.

import MySQLdb
from mod_python import apache

def verify_user(username, password):
    db = MySQLdb.Connect(host='localhost',user='mpuser',passwd='mppassword',db='mptutorial')
    cur = db.cursor()
    sql = "SELECT * FROM users WHERE username = '%s' AND password = MD5('%s');" % (MySQLdb.escape_string(username), MySQLdb.escape_string(password))
    cur.execute(sql) 
    results = cur.fetchall()
    db.close()
    return len(results) > 0
    

def authenhandler(req):
    username = req.user
    password = req.get_basic_auth_pw()
    
    if verify_user(username, password):
        return apache.OK
    else:
        return apache.HTTP_UNAUTHORIZED

Restart Apache and point your browser to http://localhost/AuthenticateExample/. You should be prompted with a username and password dialog. Try entering a wrong password, then enter the username ‘john’ and the password ‘secret’. See how mod_python handled the authentication? Neat huh?

Python Server Pages (PSP)

There is one standard handler that I did not cover in the previous section. The PSP Handler allows you to use the PSP class in the mod_python.psp module. PSP stands for Python Server Pages. Python Server Pages allow you to inline Python code in HTML (or any other kind of document) as you would if you were using PHP, ASP (Active Server Pages), JSP (Java Server Pages) or something similar. Some people argue against the practice of mixing markup and code, and I’d be one of them. I personally advocate the use of server pages as a view mechanism with very little control logic. Of course, to demonstrate the functionality of PSPs I will likely break my own rule.

As with any mod_python handler, we have to edit our Apache configuration before we can use Python Server Pages. Open your Apache configuration and add a Directory directive similar to the following:

<Directory /usr/local/apache2/htdocs/PSPExample>
    AddHandler mod_python .psp
    PythonHandler mod_python.psp
    PythonDebug On
</Directory>

This should all look pretty familiar by now! Basically we tell Apache that any files with a .psp extension are handled by mod_python. We then tell Apache that the mod_python.psp module will be the generic handler for these files. Also, because we’ve told Apache that PSP files will have a .psp extension, let’s add index.psp to our DirectoryIndex in Apache’s configuration. Open your Apache configuration file again and find the line that starts with “DirectoryIndex” and add index.psp. Depending on what was there before, that line should now look like this:

DirectoryIndex index.html index.html.var index.php index.psp

We’ve now set up the PSP handler, told Apache to serve index.psp files as directory indexes, all that is left to do is to write some actual code. So restart Apache and let’s start exploring Python Server Pages. Create a file called index.psp in the PSPExample directory and type the following:

<html>
<head>
    <title>Python Server Pages (PSP)</title>
</head>
<body>
<%
import time
%>
Hello world, the time is: <%=time.strftime("%Y-%m-%d, %H:%M:%S")%>
</body>
</html>

Save the file and point your browser to http://localhost/PSPExample. Your server should give you the index.psp file, because we added it to the list of files in DirectoryIndex. If all went well, you should get a message with the current time on your server. If you are familiar with JSP, ASP, etc, then the above code should look very similar. Basically, anything in between the <% %> tags is interpreted as Python code. Whatever is between the <%= %> tags is replaced with the result of the expression. This saves you from typing a lot of write() or print statements.

Indentation can be pretty tricky in Python Server Pages. Because PSP allows you to mix Python code and HTML / XML / Anything else, you often find that you need a way to terminate a for iteration or if statement. There’s a simple, albeit cumbersome way to do this:

    <%
    a = [1,2,3,4,5,6]
    for number in a:
    %>
    This is a number: <%=number %> <br />
    <%
    # this terminates the iteration
    %>
    <h1>Hello</h1>

If we left the comment out of the above example, the Hello header element would be written to the output for each iteration of the list. That’s obviously not what we want, so add a comment to tell PSP that the for iteration has ended.

Conclusion

We’ve just taken a whirl-wind tour of some of the features of mod_python. Leveraging the power of the Python programming language and the Apache HTTP server, mod_python offers an incredible amount of flexibility to a web developer. In the next article in this series, I’m going to start covering some of the more framework oriented approaches to Python Web Development. Until then.

Resources

I have covered a lot of material in this article, but there is still a lot more to mod_python. In order to become truly proficient, you should really take the time to read the official documentation and become more familiar with Apache. The mod_python documentation can be found here. You can also find more examples at the mod_python section of the apache wiki found here.