About Dan Crow

CTO, Songkick

Recent talks on Songkick Engineering

Since I joined Songkick a little over four years ago, our development team has done some amazing things. Our technology, process and culture have improved an enormous amount.

We’ve always been eager to share our progress on this blog and elsewhere, and we often talk about what we’ve learned and where we are still trying to improve.

Here are some recent talks given by members of our team discussing various aspects of how we work.

Processing email with Google App Engine for Java

Google App Engine (GAE) is all kinds of cool. It hosts your Java and Python apps and provides a set of APIs that give you the power, stability and scalability of Google’s infrastructure. It’s dead simple to deploy apps and it’s reasonably priced – starting at free for apps that don’t need too many resources.

But, it’s a platform, which means you write code specifically for App Engine and you have to rely on Google’s API and services for much of your functionality. Mostly this works great, but sometimes there are bugs or subtleties in how App Engine works which can cause you pain.

At Songkick I maintain an internal application for processing snippets. The idea is cribbed from Google, where I used to work. Each week Songkick employees email in a short summary of what they did that week and what they plan to do next week. It’s a really useful way of getting a quick update on everything that’s going on in the company.

The Snippets app accepts the emails, parses them and compiles them into a weekly digest which it mails out to everyone at the company. It also makes the digests available on an internal web page and has some nice reminder emails it sends out. It’s written in Java and runs on GAE.

All pretty straightforward and simple to implement. Except that email processing in GAE is quirky and poorly documented. Although GAE implements the standard JavaMail API, it’s pretty hard to find clear examples of how to make this work in the general case. So, here’s some code that shows how we do it, and some of the pitfalls to look out for. This covers the basics of parsing emails using JavaMail and the specifics of handling incoming email in GAE.

You should also read Google’s own documentation on handling GAE email. I will point out where you need to diverge from Google’s advice.

Receiving Email in Google App Engine

In order for your GAE app to handle incoming email, you must first configure your instance properly.

Start by enabling inbound emails for your app. If you don’t do this none of the rest of the code will work. Add the following lines to the appengine-web.xml file (in /war/WEB-INF/ in your GAE app folder structure)

<inbound-services>
  <service>mail</service>
</inbound-services>

Next, configure a servlet to handle incoming emails in your web.xml file (also in /war/WEB-INF/). Add lines like this:

<servlet>
  <servlet-name>mailhandler</servlet-name>
  <servlet-class>com.yourappname.MailHandlerServlet</servlet-class>
</servlet>
<servlet-mapping>
  <servlet-name>mailhandler</servlet-name>
  <url-pattern>/_ah/mail/*</url-pattern>
</servlet-mapping>
<security-constraint>
  <web-resource-collection>
    <url-pattern>/_ah/mail/*</url-pattern>
  </web-resource-collection>
  <auth-constraint>
    <role-name>admin</role-name>
  </auth-constraint>
</security-constraint>

The only change you have to make is to replace com.yourappname.MailHandlerServlet with the path to the mail handling servlet class you build. I’ll cover what this class does in the next section. You can optionally change the <servlet-name> if you want, but make sure the name is the same in both the <servlet> and <servlet-mapping> classes.

Your application is now setup to receive emails.

Two important notes

Email handling only really works when you app is deployed to GAE. You can simulate sending email when your app is running in the development server, but this does not let you test all the edge cases that come up when handling email. This is a massive pain. In practice you have to test in production.

To get email to your app, it has to be addressed to string@appid.appspotmail.com. For example, if your app is called mailinator, emails you send to any @mailinator.appspotmail.com address will be routed to MailHandlerServlet.

Pro tip: set up an email alias that redirects from a more friendly address to your @mailinator.appspotmail.com email address.

Writing the mail handling servlet

Now your app is configured to receive email, you need to handle incoming emails in a servlet. Assuming you configured <servlet-class> to be com.yourappname.MailHandlerServlet, then create a new Java class called MailHandlerServlet under com.yourappname in your app’s src folder.

Your MailHandlerServlet class extends javax.servlet.http.HttpServlet and handles Post requests sent to it. The basic class looks like:

package com.yourappname;

import java.io.IOException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

@SuppressWarnings("serial")
public class MailHandlerServlet extends HttpServlet {
  public void doPost(HttpServletRequest req, HttpServletResponse resp)
    throws IOException {
  }
}

When an email arrives for your app, the doPost method is called and you can handle the email.

Up to now, we’ve mostly followed Google’s own documentation. Now we diverge.

Google tells you to use the standard JavaMail APIs to parse your emails. This will not work for emails sent from some clients, including hotmail and Mac OS X’s Mail.app. Ouch. Emails sent from these addresses will throw java.io.IOException: Truncated quoted printable data. For the gory details, see this Google Groups thread.

If you want to handle emails from a wide variety of clients, use the code at the bottom of that thread to handle emails. I copied that code (kindly supplied by user moca) into a MimeUtils class which you can download from Pastebin here. You’ll also need to download the Apache Commons IO package to use MimeUtils.

Once you have MimeUtils, processing emails is fairly straightforward. Here is the basic MailHandlerServlet class:

package com.yourappname;

import java.io.IOException;

import javax.mail.MessagingException;
import javax.mail.internet.MimeMessage;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

@SuppressWarnings("serial")
public class MailHandlerServlet extends HttpServlet {
  public void doPost(HttpServletRequest req, HttpServletResponse resp)
    throws IOException {
  try {
    MimeMessage message = MimeUtils.createMimeMessage(req);

    if (processMessage(message)) {
      Debug.log("Incoming email handled");
    } else {
      Debug.log("Failed to handle incoming email");
    }
  } catch (MessagingException e) {
    Debug.log("MessagingException: " + e);
    e.printStackTrace();
  }
 }
}

I use a utility class called Debug to write to the GAE logs. You could replace all calls to Debug.log() with System.err.println().

This code extracts the incoming email from the HttpServletRequest using MimeUtils, then passes it to a method called processMessage. The rest is exception handling and logging.

So, what does processMessage do? It is responsible for parsing the email. Here is a slightly modified version of the processMessage method we use. This is designed to extract the text of the email body. Email attachments are ignored. It’s fairly easy to extend the code to extract and save attachments if you need them.

I use the standard JavaMail API for email parsing, but handling multipart messages correctly is a little tricky. A multipart mail is a hierarchical data structure: each part of the multipart mail can potentially also be a multipart message, so you have to recurse through the parts until you find the one you want. This is handled in the aptly named handlePart method below:

package com.yourappname;

import java.io.IOException;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Date;

import javax.mail.BodyPart;
import javax.mail.Message;
import javax.mail.MessagingException;
import javax.mail.Multipart;
import javax.mail.internet.MimeMessage;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

@SuppressWarnings("serial")
public class MailHandlerServlet extends HttpServlet {
  public void doPost(HttpServletRequest req, HttpServletResponse resp)
      throws IOException {
    try {
      MimeMessage message = MimeUtils.createMimeMessage(req);

      if (processMessage(message)) {
        Debug.log("Incoming email handled");
      } else {
        Debug.log("Failed to handle incoming email");
      }
    } catch (MessagingException e) {
      Debug.log("MessagingException: " + e);
      e.printStackTrace();
    }
  }

  private boolean processMessage(MimeMessage message) {
    String date = getMessageDate(message);
    String from = "unknown";

    try {
      from = message.getFrom()[0].toString();
      Object content = MimeUtils.getContent(message);

      if (message.getContentType().startsWith("text/plain")) {
        processMail(from, date, (String) content);
        return true;
      } else if (content instanceof Multipart) {
        Multipart mp = (Multipart) content;
        for (int i = 0; i < mp.getCount(); i++) {
          if (handlePart(from, date, mp.getBodyPart(i))) {
            return true;
          }
        }
        return false;
      } else {
        Debug.log("Unable to process message content - unknown content type");
      }
    } catch (IOException e) {
      Debug.log("Exception handling incoming email " + e);
    } catch (MessagingException e) {
      Debug.log("Exception handling incoming email " + e);
    } catch (Exception e) {
      Debug.log("Exception handling incoming email " + e);
    }

    return false;
  }

  private boolean handlePart(String from, String date, BodyPart part)
      throws MessagingException, IOException {
    if (part.getContentType().startsWith("text/plain")
        || part.getContentType().startsWith("text/html")) {
      processMail(from, date, (String) part.getContent());
      return true;
    } else {
      if (part.getContent() instanceof Multipart) {
        Multipart mp = (Multipart) part.getContent();
        Debug.log("Handling a multipart sub-message with " + mp.getCount() + " sub-parts");
        for (int i = 0; i < mp.getCount(); i++) {
          if (handlePart(from, date, mp.getBodyPart(i))) {
            return true;
          }
        }
        Debug.log("No text or HTML part in the multipart mime sub-message");
      }
      return false;
    }
  }

  private String getMessageDate(Message message) {
    Date when = null;
    try {
      when = message.getReceivedDate();
      if (when == null) {
        when = message.getSentDate();
      }
      if (when == null) {
        return null;
      }
    } catch (MessagingException e) {
      Debug.log("Cannot get message date: " + e);
      e.printStackTrace();
      return null;
    }

    DateFormat format = new SimpleDateFormat("EEE, d MMM yyyy HH:mm:ss");
    return format.format(when);
  }
}

Download the code from Pastebin

The one remaining method is processMail which handles the parsed email. I will leave implementing this up to you. In the case of our snippet system, we store the parsed email in the GAE datastore (using the excellent Objectify for convenience), ready for later use.

If you implement this code, you have fully working GAE email handling. MailHandlerServlet processes emails sent from a very wide variety of clients and services. It extracts the date, sender and email body from the incoming email and passes it to processMail for handling.

Pretty straightforward when you know how, but hard to get right if you are unfamiliar with the bugs in GAE and the subtleties of handling mutlipart emails via JavaMail.

The poetry of doors

This is not an article about Jim Morrison. This is about the other sort of door:

door: (n) a hinged, sliding, or revolving barrier at the entrance to a building, room, or vehicle, or in the framework of a cupboard.

More specifically it’s about locking and unlocking doors, and what that might teach us about usability design.

The Songkick office door gets a new lock

We’ve recently upgraded the Songkick front door from a traditional hardware lock to a fancy new electronic entry system. This is great. I no longer have to fumble through my keys, finding the right one, fitting it into the old and gnarly lock, which half the time stuck, while holding a laptop, a latte and a donut in the other hand. Now I have a sleek keyfob which is much smaller and lighter than the key it replaces. I just swipe the fob near the electronic reader and I’m in. No more spilled lattes or dropped donuts. Huzzah.

Ingress, egress, but not egrets

So, getting into to the Songkick office is much improved. What about getting out? Is it just as easy?

Sadly, the answer is “no”.

The lock is there to stop unauthorized people getting into the office and stealing our lovely stuff. Having the door locked is necessary, but it should be as easy as possible for authorized people to get in. The electronic lock is meets these requirements nicely. Mission accomplished.

But, making it hard to leave the office is not a goal. Once I’m inside, getting out should be as easy as possible – especially if there is an emergency.

So does the new Songkick electronic lock makes leaving easy? It does not. Here’s a view of the door from the inside. How would you open the door?

As you approach a door, you’re naturally looking right at it: people tend to look at the object they want to operate in order to figure out how to operate it (well, d’uh). So you’ll look at the door and see the small brass handle next to the window. Here’s our first problem – in fitting the new lock we left the old handle in place, but it doesn’t work anymore. It’s very confusing for people who have used the door before and it trips up new users because it’s placed exactly where you’d expect the mechanism to unlock the door to be.

So here’s Lesson One: don’t leave behind old stuff that doesn’t work any more, it distracts users from the new, working stuff.

Okay, so after some fumbling, we realize that the handle is in fact just there to mock us. How do we open the door?

The wall to the right of the door is covered in stuff. There’s a phone, for example. Don’t worry, you don’t have to call a doorman to get permission to exit, that’s the entry phone for the door downstairs. Below that are two similar sized boxes. If you’re trying to exit, you will end up looking through all these objects to the right and eventually notice the box marked “Press to exit”, with a light switch on it. Really? This is how I open the door? Yes, yes it is.

What is wrong with this arrangement? Here are a few problems:

  • The exit switch is in the wrong place. To open the door I have to hunt for the switch, because its not in the obvious place: on the door. Lesson two: put the control that operates an object on the object itself.
  • The switch is lost in the clutter of signs and switches. It isn’t easy to pick out from the jumble of other non-door related stuff. Everyone needs to open the door, very few need to read Songkick’s fire alarm details, so move the fire alarm and its notice somewhere else and make the switch more visible. Lesson three: make the important, frequently used controls stand out from the clutter.
  • I can only tell that this is the door switch by reading the label. If it weren’t for the words “Press to exit”, I’d have no clue this would unlock the door. The switch is a standard light switch, why would I believe it opens the door and not, say, turn on the lights. Lesson four: Don’t use a control that is associated with one operation for a different operation; its confusing.

But for all these detailed flaws in the choice of switch and its location, there is an even better way to solve this design problem. We don’t need a switch for the door at all. All I should have to do is push the door open: that is a natural and obvious mechanism for opening this door.

So the real design for the door cuts through all the confusion of the current “design” and just lets me do the obvious thing. How much better would that be?

Interestingly, a door that pushes open is better both for experienced users and novice users – in this case both Songkick employees who used the old mechanism and infrequent visitors to our office. People often complain that designers tend to think only about one group or the other, but often simple, thoughtful design can cater well to both groups.

Stop moaning already

There are two counter arguments to all this:

First, maybe it’s hard for the lock mechanism to distinguish between a push to open it from the inside, and a pull from the outside which should not open the door. That’s an engineering problem, not a design one – we should start with the design that works best for users and figure out if there is a plausible engineering solution, not the other way around. Only compromise the design if you really can’t solve the engineering problem. I can guarantee you that is not what’s happened here. I’ve used doors that unlock on a push and are secure in the other direction. It’s really not rocket science.

Second objection: why all this fuss about a door? People can figure it out, you’re overthinking this, it will take you a day or two to learn to press that exit button. Stop sweating the small stuff.

In design, whether of doors or software, the small stuff matters a lot. When people talk about Apple’s products being polished, they mean that Apple pay attention to exactly these sorts of details. Great design is all about making the user experience as simple as possible, but no simpler. This is not the iPad of door mechanisms. This might just be the Maylong M-150 of doors.

Its amazing what you can learn from a door.

Further reading

One of my favorite design books is Don Norman’s classic The Psychology of Everyday Things (later editions were renamed The Design of Everyday Things, which rather spoiled everyone’s habit of referring to the book as POET). Norman looks at how people use everyday objects like teapots and doors and examines how their design can help, or hinder, their use. Its a short, simple, well written book that anyone can enjoy, not just design nerds.

I was lucky enough to work briefly with Don when we were both at Apple in the 1990s. He’s a very smart guy. POET is still one of the best, most accessible books on design. Anyone who builds products, or is interested in learning more about how the objects in our world work, should read it.

Update

The awesome team at Songkick took pity on me and provided this helpful solution:

Songkick rocks!

What we believe in

Recently, the Songkick development team wrote down some of the principles we follow when building software. We originally did this for the latest Silicon Milkroundabout job fair. I thought I’d share them with you, to give you a flavor of what it’s like to work here.

Ship early, ship often

Our development process is optimized for speed of building, releasing and iterating. Five minutes is way too long to wait for a build, so we work hard to make it as fast as possible. Shipping fast means we deliver more great features to our users. We practice continuous integration and continuous deployment. We have high test coverage and we run tests automatically. If the build is green, ship it to production; if it’s red, fix it now.

Architecture is fundamental

Simple, versatile software abstractions are the key to building maintainable software. Our service oriented architecture is easy to extend, and is scalable and elegant. Each service has a clearly defined role and a simple, stable API. New clients are quickly assembled from these services. Client teams focus on creating awesome products for their users: our iPhone app looks different from our Spotify client, which looks different from our website, but they are all built on the same services. Each team can operate like they are their own startup.

Automate away the trivial

We have 10 engineers supporting more than six million users. To manage this, we automate away the operational burden. Services restart automatically in production. Our machines are managed by Puppet, our deployments by Jenkins and Capistrano. Our production monitoring tells us if something goes wrong, so we can fix it fast. Automation and good tools make our lives easier.

Bad code kills startups

It’s too easy to be lazy: don’t reuse that existing method, roll yet another version; don’t refactor, you can do that another day; don’t remove old code, it’s not like it gets in the way. Over months and years, these small indiscretions add up and can strangle a startup like bindweed. We don’t let this happen, we hold ourselves to a higher standard and actively tend our codebase.

Explore, listen, learn

Every engineer spends time learning. Recently a group of us did the Stanford Machine Learning course together. Everyone gets a budget and work time to attend conferences, to speak or listen. Getting better at what you do is a key part of your job. We hold regular dojos and tech talks to share and explore new technologies and ideas, and regular retrospectives to make sure we always do better next time.

Work together

We value a broad range of skills and a broad range of perspectives. Our engineers come from across the globe and have many different technical interests. Some of us dig machine learning, others are into client side coding, we have testing gurus and security experts, some are architecture geeks. We share a common passion for building the best live music service in the world. We respect our differences and believe that makes us better.

Always be playing

What fun would it be if we knew all the answers? We’re always looking for new ideas and new approaches. So we hold regular innovation days where everyone in the company can try out any idea. It might be a new product, a novel automation tool, a better user interface, a different way of building software. These are not just idle experiments: we use many of them in our day to day lives or launch them to our users.

Hire the best

Songkick is an outstanding place to work because we have an outstanding team. We’ve hired the best and the brightest from around the world: the UK, France, Germany, New Zealand, America, Brazil, South Africa, Portugal; from universities such as Imperial, Cambridge, Durham, and Stanford; and from companies including Apple, Google, Yahoo!, The Guardian and the BBC. The things we look for in great engineers are skills, passion and a desire to be part of a great team.