The client side of SOA

This article is part of a series on Songkick’s migration to a service-oriented architecture. The full series:

Following on from my previous article on what our backend services look like, it’s time to talk about the client side. How do our user-facing applications use the services, and how is it different from using ActiveRecord?

The nice thing about Rails is it doesn’t force you into using ActiveRecord. If you do, then a lot of conveniences are made available to you, but you’re really free to do whatever you want in your Rails controllers. So, instead of speaking to ActiveRecord models, our applications make HTTP calls to several backend services.

HTTP, do you speak it?

The first bit of the problem is, how do we make HTTP calls? We want this to be extremely convenient for people writing application code, which means avoiding as much boilerplate as possible. We don’t want application code cluttered with stuff like this:

uri = URI.parse("http://accounts-service/users/#{name}")
http = Net::HTTP.new(uri.host, uri.port)
response = http.request_get(uri.path)
if response.code == '200'
  JSON.parse(response.body)
else
  raise NotFound
end

when we could just write:

http_client.get("/users/#{name}").data

And that’s the simple case. When making HTTP calls, you have to deal with a lot of complexity: serializing parameters, query strings vs entity bodies, multipart uploads, content types, service hostname lookups, keep-alive or not, response parsing and several classes of error detection: DNS failure, refused connections, timeouts, HTTP failure responses, user input validation errors, malformed or interrupted output formats… and good luck changing all that if you want to change which HTTP library you want to use.

So, the first thing we did is create an abstract HTTP API with several implementations, and released it as open-source. Songkick::Transport gives us a terse HTTP interface with backends based on Curb, HTTParty and Rack::Test, all with the same high-level feature set. This lets us switch HTTP library easily, and we’ve used this to tweak the performance of our internal code.

You use it by making a connection to a host, and issuing requests. It assumes anything but a 200, 201, 204 or 409 is a software error and raises an exception, otherwise it parses the response for you and returns it:

http = Songkick::Transport::Curb.new('http://accounts-service')
user = http.get('/users/jcoglan').data
# => {'id' => 18787, 'username' => 'jcoglan'}

Songkick::Transport also has some useful reporting facilities built into it, for example it makes it easy to record all the backend service requests made during a single call to our user-facing Rails app, and log the total time spent calling services, much like Rails does for DB calls. More details in the README.

Who needs FakeWeb?

The nice thing about having a simple flat API for doing HTTP means it’s really easy to test clients built on top of Songkick::Transport, as opposed to something like FakeWeb that fakes the whole complicated Net::HTTP interface. In each application, we have clients built on top of Songkick::Transport that take an HTTP client as a constructor argument. When they make an HTTP call, they wrap the response data in a model object, which allows the application to shield itself from potential changes to the API wire format.

module Services
  class AccountsClient
    def initialize(http_client)
      @http = http_client
    end
    
    def find_user(username)
      data = @http.get("/users/#{username}").data
      Models::User.new(data)
    end
  end
end

module Models
  class User
    def initialize(data)
      @data = data
    end

    def username
      @data['username']
    end
  end
end

This approach makes it really easy to stub out the response of a backend service for a test:

before do
  @http   = mock('Transport')
  @client = Services::AccountsClient.new(@http)
end

it "returns a User" do
  response = mock('Response', :data => {'username' => 'jcoglan'})
  @http.stub(:get).with('/users/jcoglan').and_return(response)
  @client.find_user('jcoglan').username.should == 'jcoglan'
end

It also makes mock-based testing really easy:

it "tells the service to delete a User" do
  @http.should_receive(:delete).with('/users/jcoglan')
  @client.delete_user('jcoglan')
end

Being able to stub HTTP calls like this is very powerful, especially when query strings or entity bodies are involved. Your backend probably treats foo=bar&something=else and something=else&foo=bar the same, and it’s much easier to mock/stub on such parameter sets when they’re expressed as a hash, as in

http.get '/', :foo => 'bar', :something => 'else'

rather than as an order-sensitive string:

http.get '/?foo=bar&something=else'

It’s also worth noting that the models are basically inert data objects, and in many cases they are immutable values. They don’t know anything about the services, or any other I/O device, they just accept and expose data. This means you can use real data objects in other tests, rather than hard-to-maintain fakes, and still your tests run fast.

Convenience vs flexibility

Nice as it is to be able to choose which HTTP implementation you use, most of the time the application developer does not want to write

http   = Songkick::Transport::Curb.new('http://accounts-service')
client = Services::AccountsClient.new(http)
user   = client.find_user(params[:username])

every time they need to look up a record. The flexibility helps with testing and deployment concerns, but it’s not convenient. So, we put a layer of sugar over these flexible building blocks that means most of the things an application needs to do are one-liners. We have a Services module that provides canonical instances of all the service clients; it deals with knowing which hostnames to connect to, which HTTP library to use, and which client object to construct for each service.

module Services
  def self.accounts
    @accounts ||= begin
      http = Songkick::Transport::Curb.new('http://accounts-service')
      AccountsClient.new(http)
    end
  end
end

With this layer of sugar, getting a user account is one line:

user = Services.accounts.find_user(params[:username])

In our Cucumber tests, we tend to stub out methods on these canonical instances, or make a Services method return an entirely fake instance. The cukes are not complete full-stack tests; they are integration tests of the current project, rather than of the entire stack, and the lack of backend I/O keeps them very fast. The stability of the underlying service APIs means we aren’t taking a big risk with these fakes, and we have a few acceptance tests that run against our staging and production sites to make sure we don’t break anything really important.

What about error handling?

We want it to be as easy as possible to deal with errors, since messy error handling can hamper the maintainability of a project and introduce mistakes that make things harder for end users. For this reason, we made anything but 200, 201, 204 or 409 from a backend raise an exception, for example if the accounts service returns a 404 for this call, an exception is raised:

Services.accounts.find_user('santa_claus')

The exception raised by Songkick::Transport contains information about the request and response. This means you can put a catch-all error handler in your Rails or Sinatra app to catch Songkick::Transport::HttpError, and forward the 404 from the backend out to the user. The removes a lot of error handling code from the application.

In some cases though, you don’t want this behaviour. For example, say we’re rendering an artist’s page and we have a sidebar module showing related artists. If the main artist gives a 404, then the whole page response should be a 404. But if we can’t get the related artists, or their profile images, then we don’t want the whole page to fail, just that sidebar module. Such cases tend to be the minority in our applications, and it’s easy enough to catch the service exception and render nothing if the services backing a non-core component fail. Using an object model of our user interface helps to isolate these failures, and we hope to cover that in a future post.

Repeat after me: sometimes, you should repeat yourself

One open question when we moved to this model was: should we maintain client libraries for each service, or just make whatever calls we need in each application? The DRY principle suggests the former is obviously the best, but it’s worth asking this question if you do a project like this.

We went with the latter, for several reasons. First, since the services and Songkick::Transport encapsulate a lot of business and wire logic, the client and model classes in each application end up being pretty thin wrappers, and it isn’t hard to build just what you need in each project. Second, we got burned by having too many things depending on in-process Ruby APIs, where any change to a shared library would require us to re-test and re-start all downstream applications. This coupling tended to slow us down, and we found that sharing in-process code isn’t worth it unless it’s encapsulating substantial complexity.

Each application is free to tweak how it interacts with the service APIs, without affecting any other application, and this is a big win for us. It means no change to one application can have side effects or block work on another application, and we have’t actually found ourselves reinventing substantial pieces of logic since that’s all hidden behind the HTTP APIs.

And finally, having per-application service clients gives you a really accessible picture of what data each application actually relies on. Having one catch-all domain library made this sort of reasoning really difficult, and made it hard to assess the cost of changing anything.

Wrapping up

So that’s our architecture these days. If you decide to go down this route, remember there’s no ‘one right way’ to do things. You have to make trade-offs all the time, and the textbook engineering answer doesn’t always give your team the greatest velocity. Examine why you’re making each change, focus on long-term productivity, and you won’t go far wrong.

Processing email with Google App Engine for Java

Google App Engine (GAE) is all kinds of cool. It hosts your Java and Python apps and provides a set of APIs that give you the power, stability and scalability of Google’s infrastructure. It’s dead simple to deploy apps and it’s reasonably priced – starting at free for apps that don’t need too many resources.

But, it’s a platform, which means you write code specifically for App Engine and you have to rely on Google’s API and services for much of your functionality. Mostly this works great, but sometimes there are bugs or subtleties in how App Engine works which can cause you pain.

At Songkick I maintain an internal application for processing snippets. The idea is cribbed from Google, where I used to work. Each week Songkick employees email in a short summary of what they did that week and what they plan to do next week. It’s a really useful way of getting a quick update on everything that’s going on in the company.

The Snippets app accepts the emails, parses them and compiles them into a weekly digest which it mails out to everyone at the company. It also makes the digests available on an internal web page and has some nice reminder emails it sends out. It’s written in Java and runs on GAE.

All pretty straightforward and simple to implement. Except that email processing in GAE is quirky and poorly documented. Although GAE implements the standard JavaMail API, it’s pretty hard to find clear examples of how to make this work in the general case. So, here’s some code that shows how we do it, and some of the pitfalls to look out for. This covers the basics of parsing emails using JavaMail and the specifics of handling incoming email in GAE.

You should also read Google’s own documentation on handling GAE email. I will point out where you need to diverge from Google’s advice.

Receiving Email in Google App Engine

In order for your GAE app to handle incoming email, you must first configure your instance properly.

Start by enabling inbound emails for your app. If you don’t do this none of the rest of the code will work. Add the following lines to the appengine-web.xml file (in /war/WEB-INF/ in your GAE app folder structure)

<inbound-services>
  <service>mail</service>
</inbound-services>

Next, configure a servlet to handle incoming emails in your web.xml file (also in /war/WEB-INF/). Add lines like this:

<servlet>
  <servlet-name>mailhandler</servlet-name>
  <servlet-class>com.yourappname.MailHandlerServlet</servlet-class>
</servlet>
<servlet-mapping>
  <servlet-name>mailhandler</servlet-name>
  <url-pattern>/_ah/mail/*</url-pattern>
</servlet-mapping>
<security-constraint>
  <web-resource-collection>
    <url-pattern>/_ah/mail/*</url-pattern>
  </web-resource-collection>
  <auth-constraint>
    <role-name>admin</role-name>
  </auth-constraint>
</security-constraint>

The only change you have to make is to replace com.yourappname.MailHandlerServlet with the path to the mail handling servlet class you build. I’ll cover what this class does in the next section. You can optionally change the <servlet-name> if you want, but make sure the name is the same in both the <servlet> and <servlet-mapping> classes.

Your application is now setup to receive emails.

Two important notes

Email handling only really works when you app is deployed to GAE. You can simulate sending email when your app is running in the development server, but this does not let you test all the edge cases that come up when handling email. This is a massive pain. In practice you have to test in production.

To get email to your app, it has to be addressed to string@appid.appspotmail.com. For example, if your app is called mailinator, emails you send to any @mailinator.appspotmail.com address will be routed to MailHandlerServlet.

Pro tip: set up an email alias that redirects from a more friendly address to your @mailinator.appspotmail.com email address.

Writing the mail handling servlet

Now your app is configured to receive email, you need to handle incoming emails in a servlet. Assuming you configured <servlet-class> to be com.yourappname.MailHandlerServlet, then create a new Java class called MailHandlerServlet under com.yourappname in your app’s src folder.

Your MailHandlerServlet class extends javax.servlet.http.HttpServlet and handles Post requests sent to it. The basic class looks like:

package com.yourappname;

import java.io.IOException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

@SuppressWarnings("serial")
public class MailHandlerServlet extends HttpServlet {
  public void doPost(HttpServletRequest req, HttpServletResponse resp)
    throws IOException {
  }
}

When an email arrives for your app, the doPost method is called and you can handle the email.

Up to now, we’ve mostly followed Google’s own documentation. Now we diverge.

Google tells you to use the standard JavaMail APIs to parse your emails. This will not work for emails sent from some clients, including hotmail and Mac OS X’s Mail.app. Ouch. Emails sent from these addresses will throw java.io.IOException: Truncated quoted printable data. For the gory details, see this Google Groups thread.

If you want to handle emails from a wide variety of clients, use the code at the bottom of that thread to handle emails. I copied that code (kindly supplied by user moca) into a MimeUtils class which you can download from Pastebin here. You’ll also need to download the Apache Commons IO package to use MimeUtils.

Once you have MimeUtils, processing emails is fairly straightforward. Here is the basic MailHandlerServlet class:

package com.yourappname;

import java.io.IOException;

import javax.mail.MessagingException;
import javax.mail.internet.MimeMessage;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

@SuppressWarnings("serial")
public class MailHandlerServlet extends HttpServlet {
  public void doPost(HttpServletRequest req, HttpServletResponse resp)
    throws IOException {
  try {
    MimeMessage message = MimeUtils.createMimeMessage(req);

    if (processMessage(message)) {
      Debug.log("Incoming email handled");
    } else {
      Debug.log("Failed to handle incoming email");
    }
  } catch (MessagingException e) {
    Debug.log("MessagingException: " + e);
    e.printStackTrace();
  }
 }
}

I use a utility class called Debug to write to the GAE logs. You could replace all calls to Debug.log() with System.err.println().

This code extracts the incoming email from the HttpServletRequest using MimeUtils, then passes it to a method called processMessage. The rest is exception handling and logging.

So, what does processMessage do? It is responsible for parsing the email. Here is a slightly modified version of the processMessage method we use. This is designed to extract the text of the email body. Email attachments are ignored. It’s fairly easy to extend the code to extract and save attachments if you need them.

I use the standard JavaMail API for email parsing, but handling multipart messages correctly is a little tricky. A multipart mail is a hierarchical data structure: each part of the multipart mail can potentially also be a multipart message, so you have to recurse through the parts until you find the one you want. This is handled in the aptly named handlePart method below:

package com.yourappname;

import java.io.IOException;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Date;

import javax.mail.BodyPart;
import javax.mail.Message;
import javax.mail.MessagingException;
import javax.mail.Multipart;
import javax.mail.internet.MimeMessage;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

@SuppressWarnings("serial")
public class MailHandlerServlet extends HttpServlet {
  public void doPost(HttpServletRequest req, HttpServletResponse resp)
      throws IOException {
    try {
      MimeMessage message = MimeUtils.createMimeMessage(req);

      if (processMessage(message)) {
        Debug.log("Incoming email handled");
      } else {
        Debug.log("Failed to handle incoming email");
      }
    } catch (MessagingException e) {
      Debug.log("MessagingException: " + e);
      e.printStackTrace();
    }
  }

  private boolean processMessage(MimeMessage message) {
    String date = getMessageDate(message);
    String from = "unknown";

    try {
      from = message.getFrom()[0].toString();
      Object content = MimeUtils.getContent(message);

      if (message.getContentType().startsWith("text/plain")) {
        processMail(from, date, (String) content);
        return true;
      } else if (content instanceof Multipart) {
        Multipart mp = (Multipart) content;
        for (int i = 0; i < mp.getCount(); i++) {
          if (handlePart(from, date, mp.getBodyPart(i))) {
            return true;
          }
        }
        return false;
      } else {
        Debug.log("Unable to process message content - unknown content type");
      }
    } catch (IOException e) {
      Debug.log("Exception handling incoming email " + e);
    } catch (MessagingException e) {
      Debug.log("Exception handling incoming email " + e);
    } catch (Exception e) {
      Debug.log("Exception handling incoming email " + e);
    }

    return false;
  }

  private boolean handlePart(String from, String date, BodyPart part)
      throws MessagingException, IOException {
    if (part.getContentType().startsWith("text/plain")
        || part.getContentType().startsWith("text/html")) {
      processMail(from, date, (String) part.getContent());
      return true;
    } else {
      if (part.getContent() instanceof Multipart) {
        Multipart mp = (Multipart) part.getContent();
        Debug.log("Handling a multipart sub-message with " + mp.getCount() + " sub-parts");
        for (int i = 0; i < mp.getCount(); i++) {
          if (handlePart(from, date, mp.getBodyPart(i))) {
            return true;
          }
        }
        Debug.log("No text or HTML part in the multipart mime sub-message");
      }
      return false;
    }
  }

  private String getMessageDate(Message message) {
    Date when = null;
    try {
      when = message.getReceivedDate();
      if (when == null) {
        when = message.getSentDate();
      }
      if (when == null) {
        return null;
      }
    } catch (MessagingException e) {
      Debug.log("Cannot get message date: " + e);
      e.printStackTrace();
      return null;
    }

    DateFormat format = new SimpleDateFormat("EEE, d MMM yyyy HH:mm:ss");
    return format.format(when);
  }
}

Download the code from Pastebin

The one remaining method is processMail which handles the parsed email. I will leave implementing this up to you. In the case of our snippet system, we store the parsed email in the GAE datastore (using the excellent Objectify for convenience), ready for later use.

If you implement this code, you have fully working GAE email handling. MailHandlerServlet processes emails sent from a very wide variety of clients and services. It extracts the date, sender and email body from the incoming email and passes it to processMail for handling.

Pretty straightforward when you know how, but hard to get right if you are unfamiliar with the bugs in GAE and the subtleties of handling mutlipart emails via JavaMail.

SOA: what our services look like

This article is part of a series on Songkick’s migration to a service-oriented architecture. The full series:

Since I began mentioning to people that Songkick is migrating its user-facing Rails app and lots more supporting components to service-oriented architecture, I’ve been asked many times to explain how we’re doing it. Truth is, it took us a while to figure this out. Any departure from the Rails Way suddenly requires all sorts of dangerous things like Decisions and Creativity and Disagreement. What we have today is the mostly-stable result of rounds of iteration, trial-and-error and debate.

What do you mean by services, exactly?

When we say SOA, we mean we’re replacing all the ActiveRecord-based data access and business logic in our applications with a number of orthogonal web services:

  • event-listings handles data relating to concerts, artists, venues, and so on
  • accounts handles users’ account data and authentication
  • taste-imports processes sets of artists uploaded by various sources of user taste data
  • caltrak handles users’ taste data and calendar generation
  • attendance stores concerts users have said they are going to
  • media handles and stores file uploads – photos, videos and the like
  • recommendations determines sets of similar artists

These are all just Sinatra apps that return JSON for the most part. They encapsulate all the business logic previously held in our ActiveRecord models, indeed they are still based on these models at present. But they don’t simply mirror the ActiveRecord APIs: they reflect how data is used rather than how it’s stored.

ActiveRecord models tend to reflect the design of normalized databases, which reflect the static properties of entities involved. Let’s take an example. Say I ask you to design an ActiveRecord schema for modelling concerts. Most people would come up with something close to our actual model, which is:

  • A polymorphic type Event with two subtypes Concert and Festival
  • The Event class has a date property and optionally an end_date
  • The Event belongs to a Venue, which belongs to a City, which belongs to a MetroArea, which belongs to a Country, and all of these entities have a name
  • The Event has many Performances
  • Each Performance belongs to an Artist and has a billing, either headline or support
  • All Artists have a name, and other metadata like popularity

This makes sense as a database design, but doesn’t reflect how the data is used. Usually, when dealing with an event, you want all of the above information, which means accessing about seven tables. Hope you didn’t miss a JOIN somewhere!

So, we could have exposed all these as distinct resources in our services, with links from each resource to those related to it, but that would be a giant waste of HTTP requests when you always want all this information all at once. It also makes it harder to write client code for the common case – you’d need to write code to follow all those links in every app you build on top of such a service. That’s what I mean when I say the services should reflect how data is used rather than how it is stored. Here’s a request I just made to find out all about Grandaddy’s upcoming show at the Shepherds Bush Empire in September.

$ curl appserver:9101/events/12511498

{
    "id":           12511498,
    "type":         "Concert",
    "status":       "ok",
    "path":         "/concerts/12511498-grandaddy-at-o2-shepherds-bush-empire",
    "date":         "2012-09-04",
    "startTime":    "2012-09-04T19:00:00+0000",
    "endDate":      null,
    "upcoming":     true,
    "profileImage": {"id": 523306, "type": "Image"},
    "series": null,
    "performances": [{
        "artist": {
            "id":           63366,
            "name":         "Grandaddy",
            "path":         "/artists/63366-grandaddy",
            "popularity":   0.044921,
            "active":       true,
            "profileImage": {"id": 523306, "type": "Image"},
            "upcomingEventsCount": 11
        },
        "id":       24380668,
        "billing":  "headline"
    }],
    "venue": {
        "id":         38320,
        "internalId": 38320,
        "name":       "O2 Shepherd's Bush Empire",
        "path":       "/venues/38320-o2-shepherds-bush-empire",
        "smallCityLongName": "London, UK",
        "unknown":    false
    }
}

Everything an app wants to know about an event, in one HTTP call. I’m reminded of this quote, which always springs to mind when I’m putting a boundary between my business logic and a user interface:

Remember that the job of your model layer is not to represent objects but to answer questions. Provide an API that answers the questions your application has, as simply and efficiently as possible. Sometimes these answers will be painfully specific, in a way that seems “wrong” to even a seasoned OO developer.

ORM is an anti-pattern

As well as encapsulating common queries, the services encapsulate operations we often need to perform, such as recording that someone likes At The Drive-In, or creating a new concert. The services are not a thin skin over the database, they encapsulate all our domain logic so it does not get replicated in various applications. The amount of code you need in an app in order to access this logic is fairly minimal, and I’ll explain what it looks like in a future post.

What is this buying us?

The core maintainability problem with a large monolithic application, like our songkick-domain library, is that internal coupling tends to creep in over time, rendering it hard to change one thing without affecting a lot of unexpected components. Every time you commit a change to the monolithic core, all the apps depending on it need re-testing and re-starting.

Monolithic database abstractions in particular are problematic because they’re coupled to, well, a monolithic database. If you have everything in one big MySQL DB, chances are parts of that DB are under much heavier load than others. It’s hard to add more capacity in this situation without replicating the whole database; you’d rather have your data split into chunks that can be horizontally scaled independently. This both makes scaling easier and reduces cost, since you’re not wasting DB machine capacity on lots of data that probably doesn’t need replicating (yet).

Creating a set of decoupled services gives us a way to deal with that: by creating an explicit boundary layer that’s designed to be kept stable, we can change the internals of the services without breaking the apps downstream, and do it faster than if the apps were still coupled to the Ruby implementation of this logic. As our applications are moved off of ActiveRecord and onto these service APIs, the volume of code coupled to our models is going down by orders of magnitude, so we can more easily chip away at these models and begin to split them up, assigning them just to the services that need them.

I mentioned in my previous post that, because of the amount of coupled Ruby code living in one process, we’ve been stuck on old versions of Ruby, Rails and other libraries for some time. Splitting our code up like this greatly reduces the amount of code living in the same process, and makes it easier for us to upgrade our dependencies, or totally change what language or hosting platform a service runs on.

The boundary creates an awareness among the team that this is a deliberate stable API, and makes the abstraction boundary more obvious than it is with bag of Ruby APIs that all live in the same process. But we can only do this because we understand the problem domain sufficiently. We’ve been working on Songkick for five years, and so we have a much better understanding of how to divide the domain up than when we started. Of course, when you start a project, you have no idea about half the stuff that’s going to end up in it, so this migration should be seen as refactoring, rather than cookie-cutter architecture to adopt from day one.