Testing on the CI

Note: article posted originally on the Bitrise blog. If you have never used Bitrise before, please see this real cool article first.
This article is part of a series about testing on Android, please find the full list of articles of the series here:
1. A guide to write your tests for your Android apps
2. Write your tests for your Android libraries and Gradle plugins
3. Testing on the CI
4. Dealing with emulator issues and Android UI test failures on CI

It’s your time to set up your testing on Bitrise! In the third article of our Android testing series, we will give you some hints on how to effectively run a virtual device on your CI.

Now this is where the fun begins. I do not want to describe why you need CI/CD, or what the benefits are, I will just show you how to do it on Bitrise.

As we learned in the previous articles, some test types require an instrumentation (device) to run. Usually, most developers plug in their phone or launch a virtual device if they are testing with their local environment. But when it comes to CI, it becomes less trivial. Ultimately, in most cases you have the following options:

1. Starting an emulator in your CI
2. Using a cloud-based test infrastructure provider, such as Firebase Testlab (https://firebase.google.com/products/test-lab )
3. Running your own server, connecting devices to it, and running the tests on them.

‍

Each has its own ups and downs, let’s see them:

1. Involves easy setup, especially where your CI solution provides massive support for it. For example, with Bitrise, you can just use a dedicated step to do it, the AVD Manager step. The downside is that the performance of such virtual devices are usually worse than running the tests on an actual device or on your local machine.
2. This is also relatively easy to set up and similarly to the option above, Bitrise supports it with the Virtual Device Testing For Android Step. This does provide better performance, but it is more cumbersome to access the devices themselves, and will cost additional money if the free plan does not suit your needs. Still, you will be able to choose from a wide variety of devices, what otherwise would be really expensive to buy them for yourself.
3. This solution can provide a very good performance, but as you might guess, you have to invest in it, buy the required hardware, and create the required infrastructure for it. This could be difficult and risky, not to mention that you have to maintain it as well. I would recommend this only, if you have a really strong reason to not to go with option 1 or 2.

How to effectively run a virtual device on your CI?

1. Time is money

The first thing I would like to mention is that every time you run a build in Bitrise, a new machine is used. This means that you have to launch your virtual device every time.

Launching virtual devices takes significant time, which could make your CI builds take more time. What I can recommend is to try to run some things in parallel, for example while your app is being built, make sure that the virtual device is being launched. Launch them as soon as possible, so there will be the least amount of time that your CI will spend on waiting. To be sure that your virtual device is ready for the tests, use the Wait for Android Emulator Step. This Step will hold your build until the emulator is ready, place it in your Workflow just before you will run the tests.

2. Discover issues early

This walks hand in hand with “time is money”: the sooner we discover something, the sooner we will be able to fix it. Riddle time! Can you think of something that you test on the CI, but not on your local machine?

If your answer was CI Workflow, the settings or similar, you were right. To be honest, it is quite saddening when you push your code changes, expect some test results, but the CI fails, as it is misconfigured.

Luckily, you can easily prevent this: just update your Gradle files to validate the configuration. There could be a lot of opinions when this validation should run, because a lot can depend on development habits. I will give you a few options, but it will be the reader’s task to tailor it to their needs.

Let’s start with the validation. It can be done with a few lines of code in your build.gradle:

exec {
   commandLine("bitrise", "validate")
}

You can also decide to run it after/before a specific task has been run. You just have to wrap the code in a task and configure the task hook:

// Create the task
task validateCiConfig() {
	exec {
            commandLine("bitrise", "validate")
	}
}
// Make sure the project has been evaluated
afterEvaluate {
      // Get the tasks
      Task taskWhatIamLookingFor = tasks.findByName("nameOfTheTask")
      Task validateCiConfigTask =
           tasks.findByName("validateCiConfig")
      // Add the dependency on the task
      taskWhatIamLookingFor.dependsOn(validateCiConfigTask)
}

See the Gradle documentation for more options of task ordering.

3. Test results are not everything

We expect tests to provide some kind of result, but test results often lack some important details. Have you ever seen that your test failed due to the application crashing? In most cases you can guess the reason, but the best is to have the details as well. Therefore, you should check the output of Logcat too.

This is relatively easy in Bitrise, just add a Script Step at the end of your Workflow with the following content:

mkdir -p /bitrise/src/my-application/build/reports
adb logcat -d > $BITRISE_SOURCE_DIR/my-application/build/reports/logcat.txt

This will dump the Logcat to the file on the given path (and of course create the given directory, if it does not exist). Also, make sure to set is_always_run property to true for this Step.

One last thing you have to do is to export this somehow. For example, you can export it with the Deploy to Bitrise.io Step. A tip from the experienced: I prefer exporting the reports as a zip file, and when I set the name of the output file, I always postfix it with the build number, so I won’t be confused when I will have already downloaded a gazillion of build report zip files. To achieve these, set the Compress the artifacts into one file field of the step to true, the Name of the compressed artifact (without .zip extension) to <your_zip_name>_$BITRISE_BUILD_NUMBER.

Summary

Now it’s your time to set up your testing on Bitrise! Let me know how it went. Also, stay tuned for my upcoming articles: in the next one, I will describe how to deal with UI test issues and flakiness on Bitrise.