Catalog: Increasing visibility for our Android UI tests

Getting automatic feedback from tests is extremely important when building any kind of software. At Songkick, our code is tested, validated, and reported through Jenkins CI.
The pipeline around our Android app includes static analysis, unit tests and instrumentation tests running on real devices and emulators.
Previously, we used square/spoon to run our instrumentation tests. It did a great job, with support for screenshots and LogCat recordings. But recently we had to skip it because it was conflicting with another library, LogCat recording stopped working, and it was taking too long to run all of our tests (around 15 minutes for our entire test suite).
So we moved to the official connected{Variant}AndroidTest tasks. Despite being much faster (around 8 minutes for the same test suite), we were missing the logs. When a test was failing, we couldn’t check the logs for more details. So we started re-running our tests and losing trust in them.

Introducing Catalog

Catalog is a Gradle plugin for Android. When added to your project, it runs with connected{Variant}AndroidTest tasks. At the end of the tests, it generates a report per device in app/build/outputs/androidTest-results/:

Screen Shot 2016-06-20 at 17.15.08

Why should I use it?

  • Catalog is built on top of Android build tools, we are not introducing any new test tasks
  • It will give you more confidence in your tests
  • It is lightweight (basically 8 simple classes)
  • It is fast, it won’t add any significant overhead to your build time

Get started

To include the plugin in your project, just add these lines in your app/build.gradle:

buildscript {
    repositories {
        jcenter()
    }
    dependencies {
        classpath 'com.songkick:catalog:0.1.1'
    }
}

apply plugin 'com.android.application'
apply plugin 'com.songkick.catalog'

How does it work?

Catalog consists of two gradle tasks:

  • recordConnected{Variant}AndroidTest: runs before connected{Variant}AndroidTest and connects to Adb to record the LogCat for the current application.
  • printConnected{Variant}AndroidTest: runs after connected{Variant}AndroidTest and gathers the recorded logs and prints a txt and a html file into app/build/outputs/androidTest-results/.

Going forward

We are starting small with Catalog, but we would love suggestions and feedback. If you like the plugin, please create a pull request or post an issue. We have a few ideas to make it even more awesome, like:

  • show the status of the test (failure/success/ignored)
  • generate a html file listing all devices
  • add support for screenshots

Anything is possible, feel free to contribute: https://github.com/songkick/catalog

Comments are closed.