A coworker asked me if I ever write about difficult bugs I’ve encountered. I hadn’t thought about doing that before, but I like the idea. The conversation made me think about a bug I ran into at Honeycomb that I want to document before I forget the details. Solving this one was a team effort, requiring different insights from different people. It was also the kind of bug I could only imagine solving with good observability and teamwork.
I recently started a new role working on data infrastructure at LinkedIn. Working with data at a vast scale at an organization with a lot of experience in the space is exciting.
The area I work in is described internally as “Big Data.”. That term has been used in many marketing materials and can induce a lot of cringes. For this post, “Big Data” refers to working with large, semi-structured datasets. These can be hosted on distributed filesystems or streamed through a distributed event platform like Kafka. The datasets are typically not pre-loaded into an OTLP or OLAP database.
I recently left my job at Honeycomb. They’re a fantastic company, and I think they will be wildly successful. Still, life circumstances are such that I wanted to explore an opportunity at a much larger technology organization. More on that in another post, perhaps. In this post, I want to reflect on a unique aspect of my employment at Honeycomb: I was the first employee to sit on their Board of Directors as a full voting member. While this is a common practice in some countries, as far as I know, I am the first non-founder/executive person at a venture-backed software company in North America to have had this opportunity (since first publishing, people have pointed out some other companies that have also done this, which is great to see!). Because I left my term early, I only got to participate in two meetings, so these reflections come from that partial sample.
I was having a conversation with another software engineer the other day and they asked for my thoughts on various programming languages and paradigms. The topic came up because I mentioned that Python3 supported Optional types which are a popular feature in many functional programming languages like Haskell, OCaml, and F#.
Conversations like these often turn into statements about the general merits of technology x or y. It can be very tempting to draw specific conclusions. I’ve heard many of these, and have held them myself in the past. Conclusions like “Functional programming makes parallelism easier” or “Static typing reduces bugs” are seductive in their simplicity, but they smooth over the massive surface area that Software Engineers find themselves navigating these days.
Software systems are sociotechnical. I don’t think software professionals spend enough time discussing the challenges that span software and personal aspects. When we look at software through a sociotechnical lens, we begin to appreciate the complexity inherent in software development and operations. The systems we are building and operating are constantly being modified by different people, with different contexts, at different times, who may or may not speak to each other directly. This emergent collaboration can present unique challenges that are fun to navigate.
Software has an ever changing vocabulary of patterns, techniques, concepts and terms. A lot of this can come off like jargon. When people get together and discuss ideas in a professional context, they often rely on a shared context. This is fine as long as all participants are familiar with the specific jargon being used but can become intimidating to people who aren’t. Hopefully there’s enough psychological safety that someone can ask, “I’m not familiar with x, can you explain it?” but that may not always be the case.
Software is a big space and in the span of a career, you can move in a bunch of different directions, all interesting to different people. As I approach my 20th year as a professional software person, I thought it’d be interesting to reflect on how I hope to spend the next 20 years. I found it useful to write down questions I keep coming back to and use them as a guide for my continued learning. I’ll be interested to come back to this post and see how it compares to whatever “future me” gets up to.
Throughout my career, I’ve developed some opinions. Some have worn particularly deep ruts, reinforced by years of experience. I tried to figure out what these had in common, and it’s the idea that code in production is the only code that matters. Staging doesn’t matter, code on your laptop doesn’t matter, QA doesn’t matter, only production matters. Everything else is debt.
This perspective probably comes from years sitting in between operations and product development. I strongly believe that teams should optimize for getting code to production as quickly as possible as well as responding to incidents in production.
For most of us in Central Texas, the cyclocross season ended with the Texas State Age & Skill Based Championships in Georgetown, TX. This was my first season racing cross, so I thought I’d take a crack at writing a guide for people thinking about getting into it. Cyclocross is an incredibly fun, completely insane sport. If you’re thinking about racing hopefully this will help.
I can only speak to my own experiences, so the following advice pertains to local USAC races in Texas. Other races may be similar, they may not be. The usual disclaimers apply - I am not an expert, take all of this with a grain of salt. Also keep in mind that I placed in the bottom 50% in every race I entered this year - I’m here to tell you how to survive and have a lot of fun, I cannot give you any advice on how to win!
Early in the year, when I decided to sign up for my first triathlon, I remember thinking that I would do a Sprint and Olympic distance and that’d be it - mission accomplished. That was before I was addicted - a week after that first race, I went looking for more triathlons to sign up for and found Jack’s Generic Triathlon. Jack’s celebrated it’s 14th anniversary this year, so it’s been a staple in the Austin triathlon scene for a long time. The name comes from the race’s emphasis on athletes and the race itself rather than sponsors, big name participants, etc.