Paul Osman

Great Engineering Teams

2012-01-31T00:00:00-05:00

Building great teams is hard. We all know this. Nobody has the perfect recipe for creating a great engineering team, but I'm fairly certain that some things are obvious and should be heeded. What follows is a list of observations I've made throughout my career about what seems to work and what doesn't. Perfect excuse for a good old "Do" and "Don't" list:

Do: Encourage Side Projects

All good developers run into things that piss them off from time to time. Great developers build tools that help them fix the shit that pisses them off. Give developers space and time to work on things not on the roadmap and they'll probably end up building tools that solve unseen business problems. Intentionally under-load developers (or go one step further and do away with deadlines) and you'll start to see deploys run more smoothly, metrics start being captured, open source libraries and tools start being published and everyone is happier for it. Allowing developers time for side projects is an intangible, but I'm convinced it's an investment that pays for itself and then some.

Don't: Hire Asymmetric Managers

One of my pet peeves is seeing a shop employ a "manager" who's job is to pester developers and make sure they meet deadlines. The best managers I've worked with always ask two questions during check-ins:

How are you doing?
What can I do to help you?

Good managers know that their job is to enable people to do great things by clearing roadblocks and getting the hell out of their way. If you have a manager who is constantly "whipping" their team into shape, things are going to go very wrong. Hire managers who help programmers as much as possible. We're all on the same team.

Do: Support Async Workflows

Being a software developer involves blocking out large chunks of time to get into periods of deep concentration. Requiring developers to break for meetings or respond to emails / im messages / impromptu conversations immediately breaks concentration and can ruin a big part of the day. It also screws your remote employees if you have any. Zach Holman and Paul Graham have written about this much more elequently than I could hope to.

Don't: Hire "B"s

Comprimising on hiring is the quickest way to build a shitty team. It can be tempting to hire some "juniors" that will need supervision, and there's definitely room for people with a variety of experience, but you need to shoot for the best and hire nothing but. Hiring second rate developers will hurt your productivity and effect the morale of your best folks. Good people like to work with good people. Don't fuck with that.

Do: Encourage a devops culture

As I mentioned, great programmers scratch their own itches. Hire people who know the whole stack. Great programmers should write great code, but also care about how it's deployed and write or use tools that help you do that effectively and safely. Having a divide between developers and people who make sure their code can be shipped creates two problems:

Your developers stop thinking about ops.
You create an "us" vs "them" attitude. Developers complain about "conservative" ops people and ops people complain about "sloppy" developers.

Bottom line: making sure your production environment is healthy is everybodys job.

Conclusion

Most of this boils down to respect. If you're shipping software, you're an engineering company. Engineering companies need to be managed as such. Don't treat your developers as second tier. Give them what they need, support them and your chances of building a great culture where people want to come and do great work will be greatly increased.

Building Services with Apache Thrift

2011-12-12T00:00:00-05:00

I've always been interested in cross-language service frameworks. I believe in using the best tools for a job, instead of being limited to a specific language or framework, so being able to write components of a service in whatever langauge makes the most sense is attractive to me. In past lives, I've developed software that used CORBA, SOAP, COM and XPCOM and found that they all suck in different, significant ways. Because of this, I've been interested in the Apache Thrift project originally developed at Facebook.

At its core, Thrift is an interface definition language, a code generation tool and a set of libraries that take care of serialization and implement the transport protocol.

In order to generate code from a thrift service definition, you'll need to install the thrift compiler. You can install it with homebrew on a mac or download it from the project site.

Thrift services and datatypes are defined using an IDL that should look pretty familiar to anyone who has worked with CORBA or similar technologies. Let's say you want to create a service that handles creation and storage of user objects. We'll write the service in Python and a client in Ruby. This is obviously a contrived example, but imagine that we're using some kind of data store that has a library written in Python and all of our application code is in Ruby. It'd be great to be able to use the data store from our Ruby code without having to develop and maintain our own library.

We start by defining the datatypes and service interface:

struct User {                                                                                                      
  1:string first_name,                                                                                             
  2:string last_name,                                                                                              
  3:string email                                                                                                   
}                                                                                                                  
                                                                                                                   
service UserService {                                                                                              
  bool add_user(1:User user),                                                                                      
  list<User> get_users()                                                                                           
}

Save the above code in a file called users.thrift, then generate the Python code by running thrift --gen py users.thrift. Because we'll be writing a client in Ruby, we'll also want to run thrift --gen rb users.thrift to generate the Ruby code.

Next, write the server in Python. This requires a bit of boilerplate and a service handler that gets passed to the Thrift processor. Notice that thrift threw the generated code in a directory called 'gen-py'. It's just a Python module, so you can move it wherever you'd like, but in this example I'll just modify sys.path appropriately.

There are a variety of options for the server itself. For simplicity, we'll just use TServer.SimpleServer which is a straightforward single threaded server. There's also a multithreaded server and a forking server.

import os
import sys

sys.path.append(os.path.join(
    os.path.dirname(os.path.abspath(__file__)), 'gen-py'))

from thrift.transport import TSocket, TTransport
from thrift.protocol import TBinaryProtocol
from thrift.server import TServer

from userservice import UserService


class UserServiceHandler(object):

    def __init__(self):
        self.users = []

    def add_user(self, user):
        self.users.append(user)
        return True

    def get_users(self):
        return self.users


def create_server(host='localhost', port=9090):
    handler = UserServiceHandler()
    return TServer.TSimpleServer(
        UserService.Processor(handler),
        TSocket.TServerSocket(host=host, port=port),
        TTransport.TBufferedTransportFactory(),
        TBinaryProtocol.TBinaryProtocolFactory()
    )

if __name__ == '__main__':
    server = create_server()
    server.serve()

Finally we write the client in Ruby. Ignore the boilerplate client code for now, and focus on the part where we create a user, add it using the service, then get a list of users from the service. This part looks like normal Ruby code which is part of what I really like about Thrift. There are no special types to use, users as returned from the get_users method is a simple Array and everything feels really natural.

$:.push('./gen-rb')

require 'rubygems'
require 'thrift'
require 'user_service'
require 'userservice_types'

def create_client(host='localhost', port=9090):
  socket = Thrift::Socket.new(host=host, port=port)
  transport = Thrift::BufferedTransport.new(socket)
  protocol = Thrift::BinaryProtocol.new(transport)
  client = UserService::Client.new(protocol)
  return transport, client
end

user = User.new()
user.first_name = 'Paul'
user.last_name = 'Osman'
user.email = email

transport, client = create_client()
transport.open()
client.add_user(user)

users = client.get_users()
users.each do |user|
    puts "#{user.first_name}, #{user.last_name}, #{user.email}"
end

transport.close()

I found a few good resources out there, but most were either a bit outdated, or too generic or went into areas that I wasn't immediately interested in. I just wanted the straightest line to seeing what Thrift was all about, so I thought I'd write that up here.

If you're curious about Thrift, I would encourage you to read this whitepaper before diving any deeper. It provides a good overview of how Thrift works and why it was designed the way it was. I'm glad to see that Thrift is gaining some adoption, appearing in projects like HBase, Storm and FlockDB.

Storing Extra Data in Django Join Tables

2011-10-27T00:00:00-04:00

I've had some people ask me about the through argument supported by Django's ManyToManyField class. This option supports a very simple use case: when you want to store additional data in a join table.

Imagine, for instance, that we're building a simple course registration tool. Let's call the Django app courses. We'll define the following models:

from django.db import models

class Course(models.Model):
    """A course offered at our school."""
    name = models.CharField(max_length=100)
    code = models.CharField(max_length=5, unique=True)
    description = models.TextField()

    def __unicode__(self):
        return u'%s: %s' % (self.code, self.name)


class Student(models.Model):
    """A student registered with our school."""
    first_name = models.CharField(max_length=50)
    last_name = models.CharField(max_length=50)
    number = models.CharField(max_length=10, unique=True)
    courses = models.ManyToManyField(Course, null=True)
    registered_on = models.DateTimeField(auto_now_add=True)

    def __unicode__(self):
        return u'%s -> %s, %s' % (
            self.number, self.last_name, self.first_name)

When we run syncdb or generate a migration for these models, Django will automatically create the following tables: courses_course, courses_student, and courses_student_courses. The first two should be self-explanatory. The third table will store information about our many-to-many relationship. Each row will associate a student id with a course id.

Now let's say we want to store the date and time whenever a student registers for a course. We'd want something like the following model:

class Registration(models.Model):
    """Student registration for a course."""
    student = models.ForeignKey(Student)
    course = models.ForeignKey(Course)
    registered_on = models.DateTimeField(auto_now_add=True)

In order to get Django to use our model as a join table, we can use through by changing a single line in our student model:

class Student(models.Model):
    ...
    courses = models.ManyToManyField(Course, null=True,
        through='Registration')
    ...

And that's it. Now, Django will automatically generate a courses_registration table and use it instead of courses_student_courses, allowing us to store extra data about the registration.

New Gig - Joining Krux Digital

2011-10-11T00:00:00-04:00

I'm really excited to announce that I've joined the team at Krux Digital. Krux was founded last year and have since been busy building products that help content publishers responsibly manage their consumer data.

Krux's work touches on two areas very near and dear to my heart: user privacy and the proliferation of third-party Javascript. Writing third-party Javascript (social widgets, analytics, ad scripts, etc) is a great way to build distributable web applications (Ben Vinegar and Anton Koyalyov from Disqus are even writing a book on the subject). I think content publishers deserve good tools to manage and monitor code included on their sites. I'm thrilled that Krux is creating these and I look forward to helping them. I'm also really excited that they're looking way beyond ads.

Krux is based in San Francisco. They've got a great office in SoMa. I won't be moving there though, instead I'll be the first Toronto employee.

Tech-wise, I'll still be writing a lot of Python code. I'm sure I'll get my hands dirty in some less familiar areas though, as they use a lot of cool tech. One of the first things I'm tackling is their deployment process. At the moment, some pieces of Krux's infrastructure are being deployed continuously and some are not. I want to see every project deployed continuously with thorough test coverage and aggressive monitoring. I expect I'll have a few blog posts about that coming up :-)

I'm excited to be joining Krux at such an early stage. They've done a ton of work already, but I feel like they're just getting started. I'll be joining talented folks like Jos Boumans, Nick Sullivan and Krux's founders, Tom Chavez and Vivek Vaidya.

Shipped New Drumbeat.org

2011-02-14T09:17:19-05:00

A week ago we shipped the new version of drumbeat.org. This was a ground up rewrite and represents a significant change in the direction of the site. Before I go into too many details, or start talking about what's next, I want to thank everyone who has committed code to the project so far:

Jaco Joubert - Design
Ned Schwartz - Front end hacking
Brian Brennan - Initial markup and css cutup, front end rescue operative
Cillian O'Ruanaidh - Tester, Django hacker
Guillermo Movia - Markup and css for those welcome and error messages.

So, What's New?

When we decided to rewrite drumbeat.org in Django, we also decided to revisit our goals for the site. The old site tried to do a lot of things and we'd prefer to do fewer things and do them well. We decided that drumbeat.org is a social web application that should allow a user to link to content all over the web and have their activities (blog posts, wiki edits, github commits) aggregated in one place. With that in mind, we built the new drumbeat.org. There's a lot of refining to do, but that's the essence of the new site: find people and projects to follow, create projects, and aggregate activity relevant to your project.

What about my blog posts / pictures / text?

Most data, including user accounts and projects was automatically migrated over. You should even be able to log in with your old password. Some things however, like blog posts, don't exist on the new site. The old site will be available at old.drumbeat.org until March 15th, 2011. You'll be able to salvage whatever content you may have on the site before then.

What now?

Getting to a point where we were comfortable shipping was the first step. We'll now be shipping updates as often as we can. See the (constantly in progress) Roadmap for an idea of what we'd like to do over the next couple of quarters. We're also hiring a full-time front-end dev which will double the number of full-time technical staff we have dedicated to drumbeat.org. As such, I'm planning to start having weekly planning calls that will be open to the community. One week will be dedicated to planning an iteration (deciding on what bugs / features to tackle for a specific iteration) and the next one will serve as a status check-in. 2011 is going to be an exciting time for drumbeat.org. I'm so glad to be out of the building phase and into the shipping phase.

This is really great, but what it's really missing is _____.

I love feedback and I love ideas about what would be useful for drumbeat.org. If you have some feature that you feel is absolutely essential, there are a few things you can do:

Check the Roadmap. If it's already there, we're planning to add it.
Check Bugzilla. It's possible someone has already thought of this and decided to file a bug on it.
Bring it up on the mailing list. This is a good chance to see if other people share your passion for the feature you're thinking of.
File a bug. If you feel like it's worth lobbying for, file a bug requesting the feature.

It's important to note that simply requesting a feature will not make it so. It's entirely possible that a particular feature does not align with the goals of the project and will therefore not be implemented.

I found a bug

Fantastic. This is extremely helpful. If you feel you've found a bug, the process is similar to above, only shorter!

Check Bugzilla. It's possible someone has already filed a bug about the issue you've encountered. If they have, feel free to comment on the bug.
File a bug. Include a description of what you were doing, as well as steps to reproduce (if possible).

That's it! Filing bugs is a great way to help the project.

Drumbeat.org Relaunch

2011-01-03T10:47:45-05:00

Let the countdown begin! We're into the final stretch of the Django rewrite of drumbeat.org, so I thought I'd write a bit about our plans for the next month or so. I'll be staging the application this week and submitting a ticket for security review and working on a deployment plan so we'll be ready to flip the switch when the time comes. Depending on these external factors, we hope to launch by the end of the month. I'll be posting a final list of blockers this week.

Batucada

The new software that will power drumbeat.org is going to feel different. This isn't just a technology change, but a new vision and direction for the site as well. The user experience is much more social; content will be generated by people and projects in your network. Instead of a heavy content site, we're aiming for small pieces loosely joined. Users and projects will be able to add links to external content, and have content from those links show up in the news feeds of people in their network.

What to Expect

We're aiming to make the transition to the new site as seamless as possible for existing drumbeat.org users. At a minimum, user accounts and projects will be moved over to the new site. You will be able to log in with your existing credentials and your projects will already exist. Contributors to projects on the old site will become followers, and updates from your project will appear in their dashboards.

Project and profile metadata (images, bios, etc) will not be moved over. We will be keeping the old, Drupal site around for a limited time, at an alternative URL, so that people can grab their data that doesn't get automatically migrated to the new site.

Future Development

2011 will be a year of rapid iterations on the drumbeat.org website and platform. I'm going to push for us to do releases at a minimum every two weeks with bug fixes and feature enhancements. What we launch later this month will not be perfect and there'll be a lot of work to do, but we'll now be in a position to make constant improvements. Our roadmap and goals will be posted to the wiki as we plan each iteration. Some of the stuff I already know will be in scope for Q1 and Q2: l10n/i18n, improvements to the user experience, addressing scalability and performance issues as they arise, etc.

I'm looking forward to an exciting year for drumbeat.org. As always, ping me if you'd like to get involved!

Status Report - What I've Been Up To

2010-11-15T12:08:02-05:00

I've got a huge backlog of things I've been meaning to blog about, so to hopefully kick things off, here's a short list of things I've been up to in the last few months:

P2PU Development Sprint

I spent a few days in Barcelona (before the Drumbeat festival) working with some very talented developers on a development sprint for the P2PU Website. We split our time between fixing bugs in the existing Drupal code base and working on (as well as deciding) the redevelopment of the site on a new platform. After some refreshingly practical debate, we decided to build on batucada, the Drumbeat.org rewrite I've been leading. Being on the same code base and of course the same platform (Django) means the two teams will be able to share code fairly easily, which is really exciting.

The sprint was a huge success. We had a great venue which we affectionately referred to as the "Hacker Dungeon" (there were no windows!). We fixed some major problems on the current website and left knowing that P2PU has a great core development team and it's really exciting that the batucada and lernanta (codename for the new P2PU code base) teams are going to get to work so closely together (particularly great for me, as I'm on both teams!).

Drumbeat Festival

The Drumbeat Festival, also in Barcelona, was scheduled immediately after the P2PU development sprint. This was a massive event, and I'm incredibly proud to work with the people who helped pull it all together. It was a 3 day event with over 400 participants from over 40 countries in 3 different venues and somehow, it all worked. People self-organized into fairly focused groups, all working on different projects. It was very unconference style, but my sense was that most people stuck with the same group rather than cross-pollinating. The theme this year was Learning, Freedom and the Web. Attendants were from all over, but could roughly be categorized into hackers and educators. Despite this, I was pleasantly surprised to walk away with more new technical contacts than I get at most technical conferences. I also had a few people approach me about volunteering their web development skills for Drumbeat, which was fantastic.

I spent nearly all of my time working in the Badge Lab. Dan Mills, Joshua Gay, a wonderful group of volunteers and myself found a corner to hide in and worked on a functional demo of a federated badge system. More on that below!

Badge Backpack

Thanks to a great group of folks working with us, we were successful in creating a functional demo of a federated badge system. The system uses HTML5 postMessage to enable cross-origin site communication. Badges can be issued by any site, stored in a "backpack" (which simply stores the badge data in localStorage) and consumer sites can show badges stored in the backpack with just a few lines of Javascript. We're continuing development on the idea, and have a couple of Github repositories (here, and here) and of course a mailing list. Thanks to Carlo Frinolli, Johan Bichel, Ana Carvalho and others (so sorry if I didn't catch your name) for joining the code sprint and helping us out!

Batucada / Drumbeat.org

Batucada development is coming along nicely. We've got some new folks contributing code and are laser focused on delivering an initial production release by the end of the year. We're working with a designer to finalize the interface, and are really excited to get this thing going. The existing Drupal based site will be kept available for some time, albeit at a different URL, so people will be able to access it at least even after we launch the new version (signups will be disabled though).

Making Signups Easier With WebFinger

2010-10-04T12:17:12-04:00

I've been thinking about signup processes lately and what we can do with batucada (the next drumbeat.org) to make it easier, while encouraging users to use open, decentralized identity technologies like OpenID.

The Problem

Whenever I come across a new service that I want to create an account on, I dread the idea of creating a new set of credentials and re-entering a bunch of information I have already entered elsewhere. OpenID solves part of this for me, and I'm always happy to see sites supporting it. The problem, however, is that OpenID only solves one piece of the puzzle (authentication) and there are usability problems abound. The latter problem is well known, and people have come up with a bunch of innovative ways of tackling it, including intelligent OpenID selectors.

The core of the problem is that people don't tend to think of themselves as a URL, which is what OpenID uses as a supplied identifier. It's been really hard to get the average web user to think of a URL as part of their identity, and frankly, they're usually pretty hard to remember. There are ways around this, including hosting your own OpenID provider, or using delegation, so that you can choose an easy-to-remember URL, but it's unrealistic to expect non-technical web users to be able to do this.

The Solution

Enter WebFinger. The idea behind WebFinger is that, given an email-like identifier, an application can discover information about that user (including their OpenID provider). I really like this idea. I think people have an easier time with email addresses than with URLs. A bunch of providers, including AOL, Gmail and Yahoo have implemented WebFinger support, which means that there are a ton of people out there who already have WebFinger identifiers, even if they don't know it (and they shouldn't have to know it).

So what would a WebFinger based signup process look like? The first step would be to capture an email address and find out if it is a valid WebFinger identifier. If it is, the next step would be to find out of the user has an OpenID provider. If their email address is not a WebFinger identifier, the workflow would fall back to a more traditional password based signup form. Imagine clicking on a "Sign up" link and being presented with a form asking for your email address.

Once submitted, the web application would perform WebFinger based discovery on the identifier in the background. If an OpenID provider is found, the user could be presented with a friendly message asking them if they'd like to use it to sign in to the site.

If the email address is not a WebFinger identifier, or if the user opts not to use their OpenID, fall back to a more traditional sign up form.

These are obviously very rough, but I'm interested to know what people think of this idea. Specifically how the process could be made easier for non-technical users. A few concerns I have (and note that UX / Usability is certainly NOT my specialty, so help is appreciated!):

A service provider shouldn't coerce a user into using an OpenID vs creating a new set of credentials, the user must always have a choice. Unfortunately, this means asking another question, and questions are confusing.
Presuming a user has an OpenID, but does not have an active session with their provider, they are going to be sent to their provider to log in after Step Two. This could be confusing to users. We're sending them into the OpenID "Chasm of Death". Is there anything that can be done to ease this transition / manage user expectations?
Once users have signed up using an OpenID, how do we let them know that they should use that OpenID from now on? I suppose we could just prompt for an email address on sign in and perform discovery again. This introduces another step for those who aren't using an OpenID though (step one: input email address, step two on next screen: enter password).
Users could be confused immediately when we ask for their email address and not a password. This is breaking a familiar pattern, which is always potentially jarring.
What about users who know what OpenID is, and have an OpenID that they'd like to use, but don't have a WebFinger identifier that refers to their OpenID? Again, more questions to ask, which adds more confusion.

Again, I'm certainly not a UX person, so I'd be very curious to hear what people think of this idea and if people have ideas about how to improve the flow.

Batucada - Tools & Vision

2010-07-07T14:56:48-04:00

I mentioned in the announcement of batucada, that we are dropping Drupal for the next phase of development on the Drumbeat website project. We've decided to build Batucada with the Python web framework django. There are a bunch of reasons why I think django is a good choice, but the primary one is that people are already using it for a number of Mozilla web projects, including AMO and SUMO. Having other people in the organization who are familiar with the framework is helpful, of course, and it means we can reuse work done elsewhere within the webdev team (particularly in fairly generic areas like localization, caching, etc). It also means that our IT department is already familiar with deployment issues.

Django is also simply a great framework. The ability to package up reusable pieces of a project as applications means that we should be able to take advantage of a lot of great work done by folks working on similar web applications, and where we build applications ourselves, we'll be able to release those as separate reusable components. If the work we do can help other people build Django based social web applications that use open web technologies, then we will be further working towards Mozilla's mission of promoting openness, innovation and opportunity on the web.

In order to stay on track, I'm working on a roadmap and a set of core principles that will guide our development of batucada. The roadmap is a work in progress, but I think we've got a good set of principles hashed out:

Social - Batucada will be a social application. People and activities will be at the centre of the experience.
Intelligent - Batucada should be able to suggest connections based on a users activity and profile information.
Data Ownership - Users will control who has access to their personal data and will be able to export their data easily.
Open Web - Batucada will be built using Open Web technologies and will use open standards to integrate with other applications whenever it makes sense to do so.

It goes without saying that Batucada will be Open Source as well of course!

Batucada - Marching towards Drumbeat.org 1.0

2010-07-02T07:34:14-04:00

I'm excited to announce a new phase of development for the Drumbeat website project. We are about to begin development on a new project, code named batucada, that will eventually power drumbeat.org. With batucada, the future of drumbeat.org will be more social and will have the open web at its core.

To understand how we got here, it's important to give a brief history of the drumbeat website. The current alpha and beta phases of development were very experimental. We didn't have a clear idea of what the end result would be. We wanted something that would allow us to experiment and iterate quickly and so building on a CMS was a natural decision. Drupal was a good choice for an open source CMS product. Building the site on Drupal allowed us to get something up and running quickly and gave us a good chance to see how people interacted with the site. We always knew that the technology choices we had made were probably not going to be permanent. Any CMS product is going to introduce tight coupling between code and content changes, which means that developers need access not just to code, but also to the configuration data stored in a database. Providing this in a way that does not compromise users privacy has been a challenge and a barrier for people wanting to contribute code to the project. Additionally, while Drupal's module system is very handy for getting something up and running quickly, at times it makes adding custom functionality and integrations more difficult than if we were using a standard web framework. The lack of unit testing in Drupal also means that we can't create a robust set of test cases for each feature we add to the site, and can't take advantage of things like Continuous Integration, which we're big fans of. I'm aware that many of these issues are going to be further addressed in Drupal 7, but given the amount we're looking to change in the next major release, waiting and upgrading presents its own set of challenges.

We now have dedicated technical staff and a much clearer idea of what drumbeat.org should be, so it's a good time to take the user feedback we've collected from the alpha and beta versions of the site, and apply our experimental results to a production ready, 1.0 product. Batucada will represent a radical shift for drumbeat.org. The vision is a much simpler, user centric site that integrates well with other web applications.

Social and the Open Web

Thinking about where we want to take the site next, one thing is really clear to me, drumbeat.org is a great use case for a social web application: connecting people with projects and local events. Furthermore, if we are to promote the open web, we should be using open web technologies wherever they make sense. Batucada will use open web standards to integrate with other web applications whenever possible. Users should not have to do everything on drumbeat.org. Instead, drumbeat.org should be a "launch pad" for projects, and activities performed on other sites should make their way back to drumbeat.org.

The road-map hasn't been completed yet, but we'll be examining everything from discovery (WebFinger), real-time activity streaming (Activity Streams, PubSubHubbub), distributed updates (OStatus / Salmon, Portable Contacts) and decentralized identity (OpenID).

The next version of drumbeat.org will have social at its core and will be built on Open Web goodness. Where the current version is pretty document centric, the new version will be entirely people centric.

Global Community

The Drumbeat community is not centered in any one country or region. Drumbeat participants come from all corners of the globe. Events have been held in Brazil, Canada, Germany and community members are currently organizing events in France and the United States. Given the global nature of the movement, we will be building in support for localization and internationalization from day one and will be seeking help from localizers who want to contribute to the project. The code name for the project, batucada, is a nod to Drumbeat community members in Brazil who kicked off the first two Drumbeat events.

Open Source and Community Driven

Another thing that is clear is that drumbeat.org should be an open source project. We are Mozilla, we make open source software. For a variety of reasons, we failed to fully capitalize on this with the current version of the site. The next version will be launched as an open source project and we will be actively encouraging ideas and contributions. We want to build a community of people actively building the software behind the project. If this sounds exciting to you, join us! We'll need software developers, front end designers, UX engineers, QA testers, localizers and a whole host of other skill-sets.

I'll be publishing an initial roadmap in the near future and will be looking for feedback.

Using Drumbeat to build the next Drumbeat

Drumbeat.org is all about projects. It makes perfect sense to us to make batucada itself a Drumbeat project. Using existing tools to build new one will allow us to step into the shoes of project maintainers on the site - a perfect way to experience first hand how a future version of the site could better serve the needs of our users. Plus, as a programmer, I like recursion! The batucada project page on Drumbeat is a work in progress, but will be used to facilitate work on the project. Join up on the project page to help build this, and share ideas and feedback.

So there you have it! I'm really excited to kick off this next phase of development. This is the first of a series of posts, in the near future I'll be outlining some of the technical choices we've made (language, framework, etc) as well as a set of values I'm going to pitch for us to follow while we work on this project. Much more to come.

Motivation

2010-05-10T04:24:48-04:00

Reading posts like this make me feel pretty damn good about decisions I've made recently.

"And in this mission, I guess, is the key to what makes Drumbeat special: rather than an initiative, it felt more like there’s a movement building up."

Joining Mozilla

2010-05-02T23:35:44-04:00

I'm very happy to announce that I've taken a position as a Senior Web Developer for the Mozilla Foundation. I'm sure I'll be getting into all sorts of things, but job number one is going to be helping out with the Drumbeat community. If you've never heard of it, Mozilla Drumbeat is a launch-pad and playground for every day web users working on preserving the openness of the web. It's kick-ass and extremely important and I'm proud to be a part of it.

Deciding to leave FreshBooks was a tough call to make. I worked with some amazing people, and I've been able to build Some Really Cool Things. FreshBooks is truly a cool place to work, and I continue to encourage people to check them out. In the end though, the idea of working for a public benefit organization was just too good to pass up. Mozilla is an organization that I've always had a profound amount of respect for and I look forward to helping them promote openness, innovation and opportunity on the web.

I won't be moving (a number of people have asked me this). I'm going to be working out of the Toronto office, which I'm very glad to say has gone through some serious beautification since being written about by Valleywag. (No seriously, it's a lot nicer now. There's even an espresso machine!)

On Implementing WebHooks

2010-05-01T22:25:38-04:00

I wrote about our experiences implementing WebHooks at FreshBooks yesterday. You can read all the gory details here. Quick summary is: use a Message Queue, send HTTP requests asynchronously, have a retry queue, verify ownership of URIs.

#OpenWebTO: Discovery Edition

2010-04-08T00:22:16-04:00

The second #OpenWebTO meetup is happening on Tuesday, April 20th at 7pm. We're at the Centre for Social Innovation again but this time on the 1st floor.

For this meetup, James Walker (walkah) is going to be presenting WebFinger / The Hammer Stack. This will be a fairly in-depth look at the various pieces that make up Hammer Discovery Protocol Stack (XRD, LRDD, /.well-known, etc) and how they fit together. I've spent some time looking at the big picture of this stuff, but have yet to dive into the minutiae, so I'm looking forward to this.

James works for StatusNet and is one of the authors of the OStatus 1.0 Draft.

There's still plenty of space, so if you're interested in discovery, or just want to chat about Open Web stuff, sign up on Guestlist. See you there!

Community and Platform Building

2010-03-16T00:29:25-04:00

In June 2009, I started working for FreshBooks — an online billing service based here in Toronto. My official job title is Team Lead, API & Integrations, but in FreshBooks tradition I'll probably come up with something more clever sounding eventually. When I started, I was mostly doing bug fixes and enhancements> to the API, but my role has become a little higher level now and my focus has shifted to building the platform in general, as well as the community of developers who are using it to add value for our customers.

So the way I see it, there are two facets to my job: platform development (coding) and community development (evangelism, support). On the technical side, this involves building things like our recently announced webhooks implementation, and helping make decisions about the direction of our API. Because we don't want to waste our time inventing Beautiful Fucking Snowflakes, it also involves keeping an eye on developments in various open web communities, and deciding what standards, specifications and ideas can be usefully implemented at FreshBooks. On the community side, things are decidedly less clear to me. I have some thoughts (open source example projects, better library support, more forums for developers to talk, etc) but I'm mostly interested in hearing from others. What do you look for from service providers that have an API? What do you wish that companies would do to help and encourage developers working with their products? What companies do this particularly well, and why? I have some ideas of my own, but I'm really interested in what other people think. You can comment here or hit me up at paul at freshbooks dot com.

#OpenWebTO Meetup

2010-02-23T02:37:38-05:00

On Tuesday, March 2nd at 7pm, I'll be hosting the first #OpenWebTO Meetup at the Centre for Social Innovation. My goal is to get a group of people interested in open web tech together to present, chat, and share ideas. The first meeting will be focused on figuring out what people want out of such a gathering, and what topics in particular are popular. I'm going to do a quick presentation on how I see the current landscape of open web tech and hopefully get some discussion going. Open web technology is a pretty broad topic, so it's probably helpful to say that I'm thinking specifically of protocols like OpenID, OAuth, Portable Contacts, WebFinger, Webhooks / PubSubHubbub, Activity Streams, and similar.

The more the merrier of course, so if you're in the Toronto area and have an interest in open web technologies, come on out and join us! Details are on Guestlist.

Google Webfinger

2010-02-01T12:43:16-05:00

This is pretty old news, but I missed the original announcement and I think it's pretty interesting.

Google have implemented an alpha of the WebFinger protocol. If you have a Google profile, click on "Edit your profile" and add 'webfingeralpha' as an interest. Congrats, your gmail address is now a WebFinger identifier and will resolve to an XRD file containing information about services you use (if you've included that information in your Google profile).

This is pretty fun to play around with. Given an email-like identifier, such as evalpaul@gmail.com, get the host-meta XRD file for the domain:

paul@knuth ~ $ curl -i http://gmail.com/.well-known/host-meta
HTTP/1.1 200 OK
Content-Type: application/xrd+xml
Transfer-Encoding: chunked
Date: Mon, 01 Feb 2010 12:36:47 GMT
Expires: Mon, 01 Feb 2010 12:36:47 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
Server: GFE/2.0
...

<?xml version='1.0' encoding='UTF-8'?>
<!-- NOTE: this host-meta end-point is a pre-alpha work in progress.   Don't rely on it. -->
<!-- Please follow the list at http://groups.google.com/group/webfinger -->
<XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0' 
     xmlns:hm='http://host-meta.net/xrd/1.0'>
  <hm:Host xmlns='http://host-meta.net/xrd/1.0'>gmail.com</hm:Host>
  <Link rel='lrdd' 
        template='http://www.google.com/s2/webfinger/?q={uri}'>
    <Title>Resource Descriptor</Title>
  </Link>
</XRD>

Now that you have the URI template, get the XRD file for the specific user:

curl "http://www.google.com/s2/webfinger/?q=acct%3Aevalpaul%40gmail.com"

<?xml version='1.0'?>
<XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0'>
    <Subject>acct:evalpaul@gmail.com</Subject>
    <Alias>http://www.google.com/profiles/evalpaul</Alias>
    <Link rel='http://portablecontacts.net/spec/1.0' href='http://www-opensocial.googleusercontent.com/api/people/'/>
    <Link rel='http://webfinger.net/rel/profile-page' href='http://www.google.com/profiles/evalpaul' type='text/html'/>
    <Link rel='http://microformats.org/profile/hcard' href='http://www.google.com/profiles/evalpaul' type='text/html'/>
    <Link rel='http://gmpg.org/xfn/11' href='http://www.google.com/profiles/evalpaul' type='text/html'/>
    <Link rel='http://specs.openid.net/auth/2.0/provider' href='http://www.google.com/profiles/evalpaul'/>
    <Link rel='describedby' href='http://www.google.com/profiles/evalpaul' type='text/html'/>
    <Link rel='describedby' href='http://s2.googleusercontent.com/webfinger/?q=evalpaul%40gmail.com&fmt=foaf' type='application/rdf+xml'/>
</XRD>

EDIT: Google have since rolled out WebFinger support for everyone with a Google Profile. You no longer need to add 'webfingeralpha' to your interests.

Social Web Workshop

2009-10-26T04:09:24-04:00

Chris Messina was in Toronto last week and ran a full day workshop on social web technologies at the Centre for Social Innovation. He had delivered a talk the night before called "Identity is the Platform" (slides) and the workshop focused on many of the same topics. The simplest way that I can think of to explain the premise of both is: 1) The web is becoming more social and 2) The data you create on the web has value.

The workshop was fairly interactive with people from the audience asking questions or raising points every so often. It was a pretty diverse group professionally speaking, made up mostly of software developers, technologists, designers and some marketing people. The material wasn't very technical, which I found disappointing, but given the crowd it was the only thing that made sense (I guess for some reason I expected more developers).

The real take-away of the workshop was an overview of various open standards and protocols that can be used together as a sort of Open Web stack. The usual suspects were discussed: OpenID, OAuth, PubSubHubub and a few that were fairly new to me: Portable Contacts, Activity Streams and the Salmon Protocol. Chris also spoke a bit about the Mozilla Drumbeat project which certainly looks interesting.

At the end of the workshop we got into groups and brainstormed about how social web technologies could be used to create a better web. I co-opted my group to talk about how FreshBooks (my employer) could be made more social. Some really interesting ideas were brought up and I think it's a great case study for working with a community that already exists (as opposed to building a community for a new social application).

The workshop was great. I'm glad I had the opportunity to participate and I definitely came away with a few thoughts and ideas. For one, it could be a worthwhile project to document existing or create new open source projects that use open web technologies. Often it's easy to find sample code, but it's usually stripped down and not very useful. Real, concrete examples from open source projects could be extremely useful for developers looking to implement support for open technologies and protocols.

Wrapping C++ Classes in a PHP Extension

2009-04-21T13:51:33-04:00

I have a tutorial at the Zend Developer Zone called Wrapping C++ Classes in a PHP Extension. It walks you through writing a simple PHP extension that wraps a class written in C++. This is a useful thing to be able to do, especially when exposing an already existing library's API to PHP userland scripts, which seems like a fairly common task.

Bad Signs - PHP's "Shut Up" (@) Operator

2009-01-04T03:22:21-05:00

Derick Rethans has a post about PHP's "shut-up" operator (@) and why it should be avoided. He outlines a fairly common debugging scenario and gives some "under-the-hood" explanations on why that particular operator sucks. I couldn't agree more (that it should be avoided) and I want to go further and talk about something that has always bugged me about that operator. In my opinion, it's a hackish, band-aid fix to a much larger, much more worrisome problem: horrendous API design.

First, a disclaimer. I know that there are some historical reasons for a lot of these things (i.e. Exceptions not appearing until PHP5) but that doesn't change the current reality. Consider for instance filesystem functions in the standard extension:

$lines = file("/tmp/file.txt");

If the file "/tmp/file.txt" does not exist, depending on your php.ini settings (because display_errors should always be 'Off' on production systems), you may get something similar to the following output:

Warning: file(/tmp/file.txt) [function.file]: failed to open stream: No such file or directory in /path/to/your/docroot/test.php on line 3

Now the proper thing to do of course would be to verify up front that the file exists and is readable with a call to is_readable() or something similar, but imagine this is a case where there is no way to determine ahead of time if a warning or error will be generated (Derick mentions silencing network errors with stream_socket_client() as one example). In this case, you might be tempted to use the shut-up operator and depend on the return value as the one true way to see if something was succesful or not.

Now compare this with what languages like Python (for example) do, If you were to write similar code in Python:

fp = file("/tmp/file.txt")

If the file did not exist, you'll get an IOError which, if left uncaught, will result in the following being written to standard error:

Traceback (most recent call last):
File "test.py", line 1, in <module>
IOError: [Errno 2] No such file or directory: '/tmp/test.txt'

The difference here of course is that an IOError can be caught, ignored or whatever best suits your needs. It's a part of the language, the API is not just spewing out garbage to standard output or standard error. More importantly, having a function (or technically a constructor) like file() throw an exception follows a consistent design philosophy whereas PHP's error system always leaves me guessing or consulting the docs.

I guess what I'm saying is that I think the shut-up operator is just one of those signs that PHP is suffering from some serious cruft and I don't think those kinds of things bode well for a language.

What drove me away from Apple

2008-11-30T17:32:39-05:00

I recently replaced my Macbook Pro with a Thinkpad x200 running Arch Linux and Windows Vista. After 5 years of using Apple products I decided to switch – to borrow one of their marketing terms – because frankly, I was sick of being abused as a customer and the lock-in was making me less and less comfortable. Here's a list of the things that really did it for me:

Pricing the Macbook Air so high with such low specs.
Releasing an pitiful update to the air along with a nice $100 price increase.
Doing away with the matte screen option.
Pulling the remotes from being included with laptops (admittedly not a huge deal, but felt like salt on a wound).

Is it just me or is Apple becoming a little too arrogant? Anyway, I know Lenovo won't be saints either, but at least in the non-Mac OS X world I can easily switch vendors down the road if I ever need to.

PHP is Lazy

2008-10-30T20:10:54-04:00

Last week I started reading Extending and Embedding PHP. I really like extending scripting languages for a number of reasons, most of all because it's a great way to learn about the more esoteric features of a language. So far I've discovered two neat things about PHP that made me smile, so I thought I'd share them here.

Copy on Write

In an effort to save CPU cycles, PHP uses an optimization strategy called "Copy on Write". Basically what this means is that variable data isn't copied until it needs to be. Using an example similar to the one found in the book:

$a = 50;
$b = $a;
$b += 5;

At the first line, the variable $a is created and memory is allocated for it. When we set $b to equal $a however, you might think that the data in $a is copied to $b. PHP doesn't actually copy the data until the 3rd line when $b is incremented by 5. The value in this is that data is not copied unnecessarily. If, for some reason, we took out the 3rd line and $b was never modified after the initial assignment, the data would actually never have to be copied.

Return Value Used?

The second, admittedly more surprising thing I learned was that PHP provides internal functions a parameter called return_value_used whose value can be inspected to determine whether the return value being prepared by the function is actually being saved or not. Imagine you have a function that does some heavy crunching to compute a return value, and a caller (for some reason beyond us) does this:

<?php
...
myextension_compute_something();
...
?>

In other words, they call the function but don't store the return value. Well, being a thoughtful internal function author, you can check to see that return_value_used evaluates to false and just skip doing any work, saving CPU cycles and time, like so:

PHP_FUNCTION(myextension_compute_something)
{
    if (return_value_used) {
        // some fancy but expensive algorithm
        // ...
        RETURN_DOUBLE(some_value);
    } else {
        RETURN_NULL();
    }
}

ZendCon 2008 - Day 2

2008-09-18T18:23:37-04:00

Harold Goldberg, the CEO of Zend Technologies, started the day off with a talk called "Insights from the Experts: How PHP Leaders Are Transforming High-Impact PHP Applications". The talk was basically a run-down of how PHP is doing out there. (Answer: Fairly well!). There were a few case studies of larger companies using PHP (and of course Zend products) and a few reports / surveys / etc. Nothing particularly groundbreaking but it was good to start the day feeling pumped about being a PHP programmer...

Later on I attended "Tiery-Eyed", a talk given by a Zend employee, Kevin Schroeder on ... you guessed it, tiered application development. It was pretty decent, Kevin walked the audience through the creation of a micro-blogging application using a variety of different backend tiers. I left with an urge to create the smallest application I could think off with one SOAP backend, one XML-RPC backend and one REST backend.

The highlight of the day was Sara Goleman's talk on PHP Extension Writing. This was the most in-depth of the talks I attended and despite a few technical problems, the talk went smoothly. It was pretty broad but still managed to be quite inspiring.

The day ended with a reception in the exhibit hall. Free drinks and food is never a bad way to end a day...

ZendCon 2008 - Day 1

2008-09-16T19:32:51-04:00

Well yesterday was day one of ZendCon 2008. Actually, the conference proper started today, yesterday was just tutorials.

So far I'm pretty impressed. I attended the PHP Developer Best Practices tutorial in the morning and the Quality Assurance in PHP Projects tutorial in the afternoon.

The Best Practices tutorial was a little broad and most of it was focused on topics that were pretty common / basic. I'm not sure exactly what I wanted to get out of it, but for somebody who already uses an SCM system, who already tests (although not as much as I should) and writes api documentation it wasn't the most informative tutorial.

Sebastian Bergmann's QA tutorial was excellent. Of course, as the primary author of PHPUnit, I assumed Sebastian would know what he was talking about and I was right. He offered a really good, in depth look at PHPUnit and covered some of the newer features. He also covered Selenium and specifically Selenium RC in quite a bit of detail.

Looking forward to more sessions... the day of tutorials was a great start, can't wait to see what the conference proper has to offer.

Using YAML with the Zend Framework

2008-02-18T18:23:18-05:00

One of the many great components provided by the Zend Framework is Zend_Config. In a nutshell, this component allows you to access configuration data (from a file, array, etc) through a nested object property based interface. Out of the box, the component supports working with XML and INI files via the Zend_Config_Ini and Zend_Config_Xml Config adapters.

If you're familiar with frameworks like Ruby on Rails or Symfony however, you may notice that YAML support is missing. This is mostly because there is no one YAML parsing library for PHP. If you are willing to introduce another dependency to your application however, there is a really easy way to use Spyc to bridge YAML and Zend_Config. One of the nice things about the Zend_Config component is that it can take data right out of a PHP array (as opposed to an XML or INI file). This gives us quite a lot of flexibility, especially considering that Spyc returns parsed YAML data as a PHP array. Armed with this knowledge, the solution becomes really, really simple:

<?php
require_once 'Zend/Config.php';
require_once 'spyc.php';

$configFile = "config.yml";
$config = new Zend_Config(Spyc::YAMLLoad($confFile));
?>

Remarkably simple isn't it? Now, given the YAML config file:

 webhost: www.example.com
 database: 
     adapter: pdo_mysql
     host: db.example.com
     username: dbuser
     password: secret
     dbname: mydatabase

The above PHP code will create a config object that can then be used like this:

<?php
print $config->webhost;        // prints 'www.example.com'
print $config->database->host; // prints 'db.example.com'
?>

Letting the masses decide...

2008-01-08T06:54:22-05:00

Sho Kuwamoto has been doing some excellent work on implementing improved pluralization support for PHP, Ruby on Rails and AppleScript. When I put together a quick solution for this in PHP, I couldn't help but think that there might be some missing special cases or what have you. Well, Sho has now put together a feeder app for the pluralizer. What a neat experiment in collaborative web production! Go on and give it a try.

The Journey Back to Linux

2008-01-04T17:35:54-05:00

As part of starting my new job I've had to decide what kind of workstation to order. I'm fortunate in that my manager has given me the choice between a generic PC and an Apple machine. There was a time when I used Linux exclusively for all of my desktop and development needs, but about 4 years ago I purchased my first Powerbook and I really haven't touched anything other than Mac OS X on Apple hardware since.

Naturally my first instinct was simple: Get a Mac. I do all of my work on a MacBook Pro and I've grown to appreciate many things about Mac OS X. I fired up my browser and went over to the Apple Store Online to look at my options. What I'd forgotten was that the cost of even the lowest-spec Mac Pro is pretty prohibitive, so I'd have to choose between an iMac and a Mac Mini. Display size is pretty important to me, and the specs on a maxed-out Mini seem so piss-poor, so my best option was a 24" iMac. I've never been a fan of the iMacs (for a variety of reasons) so the fact that this was my best and only option was a bit disappointing to me. Why the hell is it not possible to purchase a Mac Pro for less than $2500 sans-display?

Faced with these limitations from my vendor of choice, I started to consider a PC. After looking around a bit, the prospect of purchasing a PC and running Linux again started to become strangely compelling. Perhaps years of limiting my hardware choices has taken it's toll on me after all. As one is want to do when faced with such a conundrum, I decided to do a little bit of soul searching and figure out exactly what it was (other than the price) that was attracting me to the option of a Lintel PC.

Now before anybody says anything, I'm perfectly aware that I could run Linux on Apple hardware, but I really don't consider that an option. I acknowledge that Apple hardware is generally of good quality, but I really do think it's overpriced. I'm fine paying a fee for a machine that runs OS X but I just can't justify the cost in order to just run Linux on the thing. If I'm going to run Linux, it's going to be on a Frankenstein-ish PC and that's that.

After some thought I decided that the big thing drawing me towards the Lintel option was choice. I didn't realize how much I missed piecing together a system, choosing a processor, shopping for a motherboard and all the fun things that Apple shields me from. I started to imagine my dream system, running Gentoo Linux (with XGL), maybe compiling a xen patched kernel and running a few virtual machines... it all made me feel... happy. The other unexpected realization I made was this: Running Linux on a PC I specced out and built myself makes me feel more... nerdy. For some reason, and I know that this is very unscientific, I just feel more encouraged to tinker on Linux... to try things out, to learn about new corners of the OS or a particular development environment or programming language or whatever the case may be...

Maybe it's the unpolished nature (kind of like living in a fixer-upper house) that does it, but Linux just makes me want to hack more. Apple hardware and Mac OS X is very good at making me feel comfortable or at home in the way a warm bath does... or the smell of bread baking, but it just doesn't inspire me to hack. At least... not anymore. So, we'll see if I still feel this way after a couple of months of customizing my system, emerging portage ebuilds, configuring USE flags, etc. But for now, this is the way I'm going and I'm a little excited to get back into it.

Disclaimer: I'm not leaving OS X behind. I still have my MacBook Pro, I suspect I'll still do a lot of my word processing in iWork and well, it's hard to do anything with Cocoa on Linux... I'm just diversifying my desktop experience.

Perforce Knowledge Base Launched

2007-11-04T10:21:59-05:00

In case you missed the announcement, the Perforce Knowledge Base is now available. Click here for more information about the technology driving the site.

Perspectives on SCM

2007-06-20T20:33:40-04:00

So I know this is old news by now, but Linus Torvalds gave a talk last month on git, the version management system he wrote. He spends quite a lot of time bashing other SCM systems, particularly Perforce and Subversion. He writes off CVS entirely, which is perhaps warranted, but I thought his views on the other two products were way off. He doesn't really explain why he thinks that Perforce "should just go away", so I can't really comment on that, but curiously he does go on to complain about things that Perforce actually does rather well (like branching and merging). He also says clearly that he "is not an SCM person", so I guess that explains it.

For a much more informative and useful presentation on SCM, check out Laura Wingerd's (someone who is an SCM person) presentation at Google Talks:

Using Smarty with the Zend Framework

2007-05-15T19:15:22-04:00

So it appears that the Zend Framework is approaching a 1.0 release, which is very exciting. The last time I worked with the ZF, I found the capabilities of the Zend_View class to be a little bit limited. Luckily, somebody has written a tutorial on using the Smarty template engine with the Zend Framework. Read more about it here.

PHP Pluralize Method

2007-03-03T19:06:41-05:00

I recently found myself wanting a rails-esque pluralize function like that found in the Rails Inflector class. After inspecting the Rails implementation, and playing around a bit, I was able to get a PHP version working as a Smarty variable modifier. Thank goodness for the rails version, I have no idea how I would go about listing all the various kinds of singular / plural nouns. Where would one go for a list of those kinds of things?


class MyClass 
{
    public static function conditionallyPluralize( $string, $count )
    {
        if ( intval( $count ) !== 0 )
            return MyClass::pluralize( $string );

        return $string; 
    }

    public static function pluralize( $string ) 
    {

        $plural = array(
            array( '/(quiz)$/i',               "$1zes"   ),
        array( '/^(ox)$/i',                "$1en"    ),
        array( '/([m|l])ouse$/i',          "$1ice"   ),
        array( '/(matr|vert|ind)ix|ex$/i', "$1ices"  ),
        array( '/(x|ch|ss|sh)$/i',         "$1es"    ),
        array( '/([^aeiouy]|qu)y$/i',      "$1ies"   ),
        array( '/([^aeiouy]|qu)ies$/i',    "$1y"     ),
            array( '/(hive)$/i',               "$1s"     ),
            array( '/(?:([^f])fe|([lr])f)$/i', "$1$2ves" ),
            array( '/sis$/i',                  "ses"     ),
            array( '/([ti])um$/i',             "$1a"     ),
            array( '/(buffal|tomat)o$/i',      "$1oes"   ),
            array( '/(bu)s$/i',                "$1ses"   ),
            array( '/(alias|status)$/i',       "$1es"    ),
            array( '/(octop|vir)us$/i',        "$1i"     ),
            array( '/(ax|test)is$/i',          "$1es"    ),
            array( '/s$/i',                    "s"       ),
            array( '/$/',                      "s"       )
        );

        $irregular = array(
        array( 'move',   'moves'    ),
        array( 'sex',    'sexes'    ),
        array( 'child',  'children' ),
        array( 'man',    'men'      ),
        array( 'person', 'people'   )
        );

        $uncountable = array( 
        'sheep', 
        'fish',
        'series',
        'species',
        'money',
        'rice',
        'information',
        'equipment'
        );

        // save some time in the case that singular and plural are the same
        if ( in_array( strtolower( $string ), $uncountable ) )
        return $string;

        // check for irregular singular forms
        foreach ( $irregular as $noun )
        {
        if ( strtolower( $string ) == $noun[0] )
            return $noun[1];
        }

        // check for matches using regular expressions
        foreach ( $plural as $pattern )
        {
        if ( preg_match( $pattern[0], $string ) )
            return preg_replace( $pattern[0], $pattern[1], $string );
        }
    
        return $string;
    }

}

Works like a charm!

EDIT: Sho Kuwamoto took this example and ran with it, creating a really robust and complete pluralization solution for PHP (and ActionScript!). Check out his improvements here.

PAM Authentication for Apache, Trac and SVN

2007-01-04T20:45:38-05:00

I’m going to describe how to get two of my favourite development tools, Trac and Subversion, to authenticate against PAM (Pluggable Authentication Module). For those who might not be familiar, Trac is a wiki that has integrated SCM and issue tracking. It’s written in Python and it’s incredibly useful. I’ve been using Subversion for version control for a long time and Trac is perfect for documenting a project as you go, and for keeping track of tasks and bugs. It’s access control settings allow you to specify levels of access for different users, so I often create one group for me and any other developers, and another group for my clients. The clients group can update the wiki, create tickets and get reports or look at milestones, etc. The developers group can administer all aspects of the site. Anyway, as useful as Trac and SVN are, I started to get really sick of handling authentication for them. I used to set up htpasswd files for each repository and wiki and it got to be a real pain, especially when I wanted my clients or developers to also have email, shell access, etc. So I decided to try and get mod_pam_auth working so I could use existing system accounts for Trac and Subversion access (over SSL of course).

mod_auth_pam is an Apache module that implements Basic authentication on top of the Pluggable Authentication Module. Unfortunately, as the project page says, the module is no longer being maintained which is unfortunate, but it works well enough with Apache 2.0.

Installing the module is pretty straightforward if you’re familiar installing Apache modules. I won’t go into too much detail, but as usual you’ll need to load the module in your Apache configuration:

LoadModule auth_pam_module modules/mod_auth_pam.so
LoadModule auth_sys_group_module modules/mod_auth_sys_group.so

Once that’s done you can easily set up basic authentication with PAM. Because Basic authentication involves sending a username + password combination in plain text, this setup should not be used without SSL. Within my VirtualHost configuration, I define separate location configs for each trac site and svn repository. It all looks something like this:

<VirtualHost 123.321.123.321:443>
    ServerName host.domain.tld
    SSLEngine On
    ...
    # Trac config
    <Location /trac>
       SetHandler mod_python
       PythonHandler trac.web.modpython_frontend 
       PythonOption TracEnvParentDir /var/lib/trac
       PythonOption TracUriRoot /trac
    </Location>

    <Location "/trac/tracsiteone/login">
       AuthPAM_Enabled On
       AuthType Basic
       AuthName "trac site # 1"
       Require user paul
    </Location>

    <Location "/trac/tracsitetwo/login">
       AuthPAM_Enabled On
       AuthType Basic
       AuthName "trac site # 2"
       Require group developers
    </Location>

    # Subversion config
    <Location /svn>
       DAV svn
       SVNParentPath /var/svn
       SVNListParentPath On
       SVNAutoVersioning On
    </Location>

    <Location "/svn/repositoryone">
       AuthPAM_Enabled On
       AuthType Basic
       AuthName "Repo # 1"
       Require user paul
    </Location>

    <Location "/svn/repositorytwo">
       AuthPAM_Enabled On
       AuthType Basic
       AuthName "Repo # 2"
       Require group developers
    </Location>
</VirtualHost>

So what we have now is two SVN repositories and two Trac wikis. For the first trac wiki and the first subversion repository, only the user 'paul' is given access. For the second, any valid user in the 'developers' group has access. Unfortunately there’s an issue with shadow passwords and this module and I’m not entirely happy with the work-around so I may have to edit this setup to use mod_authnz_external or maybe I’ll eventually move to LDAP. Regardless, I find this works well enough for now and saves a lot of hassle maintaining separate authentication files.

Xen Hosting

2007-01-03T20:40:56-05:00

I recently made the decision to move from a shared hosting account to a dedicated server. I was very happy with my shared hosting provider (WebFaction) but it’s not uncommon for me to have several websites or trac wikis or subversion repositories all going at once. I need these to be accessible to me and my clients at all times and my home DSL connection just isn’t dependable enough for hosting. I started looking into dedicated hosting packages but decided to save myself about $80 / month and go for a VPS (Virtual Private Server) package.

When it comes to virtualization, I'm a pretty big fan of Xen. If anyone is interested in the subject and hasn’t already, it’s really worth checking out. At a really high level, here’s how it works: basically you have a DOM0 (Domain-0) patched kernel that you run on your physical machine which you allocate a specific amount of RAM to. Once you have your DOM0 set up correctly you can install any number (RAM permitting) of guest hosts running a DOMU (unprivileged domain) patched kernel. You specify how much RAM you want to allocate to each DOMU and there are a number of ways to handle filesystems. The DOM0 (host OS) manages access to hardware and other low-level stuff and basically makes it completely transparent when working within a DOMU. It comes with some handy userland tools to manage guest hosts and there are 3rd party packages like enomalism (Yes I just plugged my former employer!) that make it really simple.

So I started shopping around for hosting companies offering VPS packages that used Xen and I found one that looked a little nickle and dime, but were local, had reasonable prices and offered my favourite Linux distribution (Gentoo!). I figured I’d give them a shot. One week and 14 hours of downtime later I canceled my account, got a refund and signed up for an account with RimuHosting. Now these people know how to do support! Within minutes of signing up I got an email with a link to setup my PayPal subscription. I setup my subscription and less than an hour later had my account details. First things first, I tried to connect to my host via SSH and got a timeout... good opportunity to try out their support! I emailed their tech support and we were able to figure out that the problem was a lack of a reverse DNS entry on the VPS' IP address. I didn’t know that Mac OS X won’t open an SSH connection in these circumstances. Okay, easy to solve... there’s a handy utility in their control panel to do just that. Once that was working I was up and running and started to set up Apache, Postfix and all the other wonderful software I use.

Now I’m happily hosting several websites (each with MySQL or PostgreSQL databases), Trac wikis, SVN repositories, I’m running AWStats, hosting my mail, and generally loving the VPS life, all for at least a 5th of what a dedicated package would cost!

Introduction to mod_python

2007-01-01T19:56:10-05:00

This article will provide a brief introduction to mod_python, a tour of what you can do with it, and some pointers to further resources should you want to explore it in more depth. I will assume that the reader is comfortable programming in Python (although no specific knowledge is required) and is familiar with Apache and basic web concepts. This article is not intended to be a complete reference for mod_python. Instead it is meant to consolidate information available from other sources, to hopefully whet your appetite and encourage you to read more from the official documentation (links are in the Resources section).

Prerequisites and Assumptions

I am going to assume that you have a working installation of Apache 2.0.47 or higher and have Python 2.2.1 or later installed. To make things simple, I’m going to assume you are working from the same machine that Apache is installed on, so all URLs will have 'localhost' as the server name. Replace this with your server name if this is not the case.

What is mod_python?

In the bad old days, most web development was done using CGI (Common Gateway Interface). Writing a CGI program meant creating an executable (script or binary) that the web server called to handle a request. The output generated by the CGI program would then be returned to the user via their browser. Think about that process for a second: a) the user requests a page from the web server, perhaps with some arguments sent via GET or POST, b) the web server recognizes that the requested page is handled by a CGI script and invokes the CGI process, c) the CGI program collects information from the web server using some mechanism (usually environment variables), does some processing and prints out a bunch of HTML (usually), if everything went alright, the web server takes the output and sends it to the user.

Sounds pretty cumbersome when you think about it, doesn’t it? It’s not only cumbersome, it’s also slow and very error prone. mod_python saves us from having to go through this process by integrating the Python programming language right into the Apache HTTP server. This provides a much faster way for Apache to execute python handlers, and as an added bonus, gives us complete access to the Apache internals. Imagine mod_python as a little guy (or ... a Python?) stuck inside your web server intercepting certain requests and allowing you to do really cool things with them. Okay, so that may not be a very good technical description, but we’ll get to that. Sound interesting? It is!

It’s important to understand that writing applications with mod_python is not the same as writing applications with a server-side scripting language like PHP. Instead, with mod_python you specify handlers in the Apache configuration file(s) that allow you to customize how a request is handled. This allows you to do a variety of neat things like implement protocols other than HTTP, filter the request and response, determine a document’s content-type, etc.

So let’s get mod_python installed and take a tour of how it works. In order to follow along with this tutorial, you’ll need to have Apache and Python installed, and then install mod_python.

Installation

There are a few different ways to install mod_python, depending on what Operating System you are running. If you are using a distribution of Linux that has a decent package management system (any Red Hat / Fedora, Debian or Gentoo based system for instance) then there will likely already be a package available for you to install. On Gentoo I just emerge the mod_python ebuild with the USE flags I want and portage automatically adds configuration files to be included into my Apache configuration. On Red Hat Enterprise Linux I use a mod_python RPM that depends on having the python-devel and httpd-devel packages. For the sake of brevity I’m only going to cover installing from source here. Consult your OS’ documentation to see if there is an easier way for you. (There is a way to install mod_python on Windows but I am not going to cover that here).

Compiling from Source

In order to compile mod_python, you’ll need to grab the latest stable source release from the mod_python website. At the time of this writing, the latest stable release was 3.2.10.

I’m going to assume you have Apache2 already installed (if not you can get it from the Apache website). I am using Apache 2.0.59 but the process should be the same for any version of Apache above 2.0.47. I have Apache installed in /usr/local/apache2. You will need to adjust the path to match your installation.

Once you’ve downloaded the source tarball for mod_python, untar it and run the 'configure' script (feel free to run ./configure --help to see what other configuration options are available):

pike:/usr/local/src paul$ tar xfz mod_python-3.2.10.tgz 
pike:/usr/local/src paul$ cd mod_python-3.2.10
pike:/usr/local/src/mod_python-3.2.10 paul$ ./configure --with-apxs=/usr/local/apache2/bin/apxs
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
...

If everything went okay, and you didn’t get any error messages from configure, you should then run 'make' to compile mod_python and 'make install'. You will likely need to run 'make install' as root, so you can use "su -c 'make install'":

pike:/usr/local/src/mod_python-3.2.10 paul$ make
... 
(a whole bunch of output you can ignore unless you know what you’re doing)
pike:/usr/local/src/mod_python-3.2.10 paul$ su -c 'make install'
...
(more output that we won’t concern ourselves with)

Safety Note: Never run a configure script as root. It’s always possible that the host you retrieved the source distribution from has been compromised and therefore you don’t know for sure what could be hiding in that script.

If the compilation process didn’t report any errors, you should be ready to go. Next we have to open the Apache2 configuration file (usually called httpd.conf) and add the following:

LoadModule python_module      modules/mod_python.so

Note: the path to mod_python.so may very. Check your Apache2 installation root and find out if it was actually put there or somewhere else.

As with any time you edit the Apache configuration, you will have to restart Apache before the changes take effect. Now we want to verify that our installation went smoothly. There are a variety of ways to do this, I always like to grab the default headers from Apache to see what is installed. I do this by telnetting into port 80 and typing 'HEAD / HTTP/1.0':

pike:/usr/local/src/mod_python-3.2.10 paul$ su -c '/usr/local/apache2/bin/apachectl restart'
pike:/usr/local/src/mod_python-3.2.10 paul$ telnet localhost 80
Trying ::1...
Connected to localhost.
Escape character is '^]'.
HEAD / HTTP/1.0

HTTP/1.1 200 OK
Date: Mon, 01 Jan 2007 04:28:24 GMT
Server: Apache/2.0.59 (Unix) mod_python/3.2.10 Python/2.3.5 PHP/5.2.0
Last-Modified: Sat, 20 Nov 2004 20:16:24 GMT
ETag: "ab5da-2c-4c23b600"
Accept-Ranges: bytes
Content-Length: 44
Connection: close
Content-Type: text/html

Connection closed by foreign host.

As you can see from the "Server" header, mod_python version 3.2.10 is installed. Now to really test it! Open the Apache2 configuration file again and add the following location directive:

<Location /mpinfo>
    SetHandler mod_python
    PythonHandler mod_python.testhandler
</Location>

Restart Apache again and point your browser to http://localhost/mpinfo and you should see a test page with a lot of useful information about your server environment. This information can come in very handy when debugging problems so I tend to keep it around. If you don’t see this page, something must have gone wrong. Go over the instructions again or consult your operating system documentation. If you see the page, congratulations, you have installed mod_python! Now let’s move on and start learning about mod_python.

Handlers

In order to truly understand the power offered by mod_python, a basic understanding of Apache handlers is required. Essentially you can think of a handler as a processing "phase". Apache gets a request, and then initiates a number of handlers (or "phases") to do something. The handlers can either be built into Apache, or included as modules. Apache handlers may be configured explicitly, based on either filename extensions or location. Examples of functionality that takes place in handlers may include authenticating a user, invoking a cgi script, getting the server’s status, etc. mod_python allows you to tap into any handler used by Apache. mod_python also provides a few standard handlers to help you with some common tasks.

Publisher Handler

The publisher handler is available so you don’t always have to worry about writing your own handlers and can instead focus on developing your application. It’s very handy. In order to use the publisher handler, you have to something like this to your Apache configuration:

<Directory /usr/local/apache2/htdocs/PublisherExample>
    AddHandler mod_python .py
    PythonDebug On
    PythonHandler mod_python.publisher
</Directory>

Note: The "PythonDebug On" line makes mod_python output errors to the browser when possible, instead of the Apache server logs. This is useful while we’re developing.

Now create a file called 'ModPythonExample.py' in the directory 'PublisherExample' and type the following:

from time import strftime, localtime

def publisher_example(req):
    req.content_type = 'text/html'
    time_str = strftime("%a %b %d %H:%M:%S %Y", localtime())
    message = "<h1>Hello from mod_python!</h1>"
    message += "<p>The time on this server is %s</p>" % (time_str)
    return message

Because we have modified the Apache configuration we will need to restart it once again. Now point your browser to http://localhost/PublisherExample/ModPythonExample.py/publisher_example and you should see a message telling you what time it is.

As we can see from this example, the publisher handler calls a function and just sends the return value to the client. The function receives a request object as an argument. We set the request object’s content_type to 'text/html' because we want the output to be handled as HTML. We then construct a message including the current time and date and return it. Notice the structure of the URL. The first part after the directory (ModPythonExample.py) is the name of our file, and the second part ("publisher_example") is the name of the function to call.

Obviously this is only a simple example. There are many great things you can do with the publisher handler. See the documentation for more information (Resources).

CGI Handler

The CGI Handler is provided as a stepping stone away from traditional CGI. It is not intended as a final solution for using mod_python. Basically, the CGI Handler emulates a CGI environment from within mod_python, allowing you to migrate CGI based Python applications to mod_python with little or no modification. Read the documentation to learn about limitations of this handler. To use it, just add a Directory directive to your Apache config (like we did for the Publisher Handler) and include the following:

SetHandler mod_python
PythonHandler mod_python.cgihandler

Once again, this should not be considered a final solution but can certainly improve the performance of your existing CGI code without too much modification.

Custom Handlers

Using a standard handler may be appropriate in many scenarios, but sometimes you might find it more appropriate to write your own handler. mod_python allows you to do this (of course) and in fact makes it pretty easy! Let’s create the following directory directive in our Apache configuration:

<Directory /usr/local/apache2/htdocs/CustomExample>
    AddHandler mod_python .py
    PythonDebug On
    PythonHandler customexample
</Directory>

This tells Apache that any requests for a file with a .py extension will be served by mod_python. (It is worth noting that the file does not actually have to exist, and that in fact a request for /CustomExample/myfile.py and /CustomExample/myotherfile.py will both be handled the same with this configuration). The "PythonHandler customexample" line tells Apache to hand requests for files with a .py extension to this module (which we will be writing). The actual process goes something like this: a) Apache receives a request for a file in the CustomExample directory that has an extension of .py. b) Apache recognizes that this request is to be handled by mod_python and attempts to import a module called "customexample". Apache looks for this module in sys.path (with our directory prepended to it so anything in there will be found first). c) Apache will then look for a function called "handler" in the module and execute it, passing the request object as an argument. Okay, enough details, let’s write our handler module. Create a file called "customexample.py" in the "CustomExample" directory:

from mod_python import apache

def handler(req):
    req.content_type = 'text/plain'
    req.write('Hello from mod_python!')
    return apache.OK

A few things to notice here: a) we use the request object to write output to the client instead of just returning content, b) we return a constant from the apache module. The apache.OK constant corresponds to an HTTP 200 response code. Other constants are defined for 404, 302, etc response codes. Of course, this example doesn’t really do anything new, so to demonstrate the real power of writing our own handlers we are now going to create a super simple (and not very secure) MySQL authentication handler. If you have MySQL installed, create a database called "mptutorial" and add the following table and records:

CREATE TABLE `users` (
    `id` int(11) unsigned NOT NULL auto_increment,
    `username` varchar(50) NOT NULL,
    `password` varchar(50) NOT NULL,
    PRIMARY KEY  (`id`),
    UNIQUE KEY `username` (`username`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

INSERT INTO `users` (`username`, `password`) VALUES ('john', MD5('secret'));

Then add the following to your Apache configuration:

<Directory /usr/local/apache2/htdocs/AuthenticateExample>
    AddHandler mod_python .py
    PythonDebug On
    PythonAuthenHandler authuser
    AuthType Basic
    AuthName "Secure Area"
    Require valid-user
</Directory>

Now let’s write our example code. We need to write an authentication handler that retrieves the username and password entered Note: This example uses the MySQLdb module.

import MySQLdb
from mod_python import apache

def verify_user(username, password):
    db = MySQLdb.Connect(host='localhost',user='mpuser',passwd='mppassword',db='mptutorial')
    cur = db.cursor()
    sql = "SELECT * FROM users WHERE username = '%s' AND password = MD5('%s');" % (MySQLdb.escape_string(username), MySQLdb.escape_string(password))
    cur.execute(sql) 
    results = cur.fetchall()
    db.close()
    return len(results) > 0
    

def authenhandler(req):
    username = req.user
    password = req.get_basic_auth_pw()
    
    if verify_user(username, password):
        return apache.OK
    else:
        return apache.HTTP_UNAUTHORIZED

Restart Apache and point your browser to http://localhost/AuthenticateExample/. You should be prompted with a username and password dialog. Try entering a wrong password, then enter the username 'john' and the password 'secret'. See how mod_python handled the authentication? Neat huh?

Python Server Pages (PSP)

There is one standard handler that I did not cover in the previous section. The PSP Handler allows you to use the PSP class in the mod_python.psp module. PSP stands for Python Server Pages. Python Server Pages allow you to inline Python code in HTML (or any other kind of document) as you would if you were using PHP, ASP (Active Server Pages), JSP (Java Server Pages) or something similar. Some people argue against the practice of mixing markup and code, and I’d be one of them. I personally advocate the use of server pages as a view mechanism with very little control logic. Of course, to demonstrate the functionality of PSPs I will likely break my own rule.

As with any mod_python handler, we have to edit our Apache configuration before we can use Python Server Pages. Open your Apache configuration and add a Directory directive similar to the following:

<Directory /usr/local/apache2/htdocs/PSPExample>
    AddHandler mod_python .psp
    PythonHandler mod_python.psp
    PythonDebug On
</Directory>

This should all look pretty familiar by now! Basically we tell Apache that any files with a .psp extension are handled by mod_python. We then tell Apache that the mod_python.psp module will be the generic handler for these files. Also, because we’ve told Apache that PSP files will have a .psp extension, let’s add index.psp to our DirectoryIndex in Apache’s configuration. Open your Apache configuration file again and find the line that starts with "DirectoryIndex" and add index.psp. Depending on what was there before, that line should now look like this:

DirectoryIndex index.html index.html.var index.php index.psp

We’ve now set up the PSP handler, told Apache to serve index.psp files as directory indexes, all that is left to do is to write some actual code. So restart Apache and let’s start exploring Python Server Pages. Create a file called index.psp in the PSPExample directory and type the following:

<html>
<head>
    <title>Python Server Pages (PSP)</title>
</head>
<body>
<%
import time
%>
Hello world, the time is: <%=time.strftime("%Y-%m-%d, %H:%M:%S")%>
</body>
</html>

Save the file and point your browser to http://localhost/PSPExample. Your server should give you the index.psp file, because we added it to the list of files in DirectoryIndex. If all went well, you should get a message with the current time on your server. If you are familiar with JSP, ASP, etc, then the above code should look very similar. Basically, anything in between the <% %> tags is interpreted as Python code. Whatever is between the <%= %> tags is replaced with the result of the expression. This saves you from typing a lot of write() or print statements.

Indentation can be pretty tricky in Python Server Pages. Because PSP allows you to mix Python code and HTML / XML / Anything else, you often find that you need a way to terminate a for iteration or if statement. There’s a simple, albeit cumbersome way to do this:

<%
a = [1,2,3,4,5,6]
for number in a:
%>
This is a number: <%=number %> <br />
<%
# this terminates the iteration
%>
<h1>Hello</h1>

If we left the comment out of the above example, the Hello header element would be written to the output for each iteration of the list. That’s obviously not what we want, so add a comment to tell PSP that the for iteration has ended.

Conclusion

We’ve just taken a whirl-wind tour of some of the features of mod_python. Leveraging the power of the Python programming language and the Apache HTTP server, mod_python offers an incredible amount of flexibility to a web developer. In the next article in this series, I’m going to start covering some of the more framework oriented approaches to Python Web Development. Until then.

Resources

I have covered a lot of material in this article, but there is still a lot more to mod_python. In order to become truly proficient, you should really take the time to read the official documentation and become more familiar with Apache. The mod_python documentation can be found here. You can also find more examples at the mod_python section of the apache wiki found here.