Prerequisites

In order to execute tests on Android devices marathon will need Android SDK installed. Devices are expected to be connected to local machine by any means supported by the adb (local usb connection, local emulator or TCP/IP).

CLI

To indicate to CLI that you’re using a vendor config for android you have to specify the type in the root of the Marathonfile configuration as following:

vendorConfiguration:
  type: "Android"
  additional_option1: ...
  additional_option2: ...

Required options

Android SDK path

If you’re using gradle plugin then this option is automatically detected.

If you have an ANDROID_HOME environment variable then this option is automatically detected by the CLI as well.

If this is not the case then you have to specify this option manually:

  • vendorConfiguration:
      type: "Android"
      androidSdk: "/usr/share/local/android"
    

APK paths

Single module testing

Application APK path

If you’re using gradle plugin then this option is automatically detected. If this is not the case then you have to specify this option manually.

  • vendorConfiguration:
      type: "Android"
      applicationApk: "app/build/outputs/apk/debug/app-debug.apk"
    

Test application APK path

If you’re using gradle plugin then this option is automatically detected. If this is not the case then you have to specify this option manually.

  • vendorConfiguration:
      type: "Android"
      testApplicationApk: "app/build/outputs/apk/androidTest/debug/app-debug.apk"
    

Multi module testing

Marathon supports testing multiple modules at the same time (e.g. your tests are across more than one module):

  • vendorConfiguration:
      type: "Android"
      outputs:
      - application: "android-app/app/build/outputs/apk/debug/app-debug.apk"
        testApplication: "android-app/app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk"
      - testApplication: "android-library/library/build/outputs/apk/androidTest/debug/library-debug-androidTest.apk"  
    

Each entry consists of testApplication in case of library testing and application + testApplication for application testing.

This option is not available for users of gradle plugin.

Optional

Automatic granting of permissions

This option will grant all runtime permissions during the installation of the application. This works like the option -g for adb install command. By default, it’s set to false.

  • vendorConfiguration:
      type: "Android"
      autoGrantPermission: true
    
  • marathon {
        autoGrantPermission = true
    }
    
  • marathon {
        autoGrantPermission = true
    }
    

ADB initialisation timeout

This option will allow you to increase/decrease the default adb init timeout of 30 seconds.

  • vendorConfiguration:
      type: "Android"
      adbInitTimeoutMillis: 60000
    
  • marathon {
        adbInitTimeout = 100000
    }
    
  • marathon {
        adbInitTimeout = 100000
    }
    

Device serial number assignment

This option allows to customise how marathon assigns a serial number to devices. Possible values are:

  • automatic
  • marathon_property
  • boot_property
  • hostname
  • ddms
  • vendorConfiguration:
      type: "Android"
      serialStrategy: "automatic"
    
  • marathon {
        serialStrategy = SerialStrategy.AUTOMATIC
    }
    
  • marathon {
        serialStrategy = SerialStrategy.AUTOMATIC
    }
    

Notes on the source of serial number:

marathon_property - Property name marathon.serialno

boot_property - Property name ro.boot.serialno

hostname - Property name net.hostname

ddms - Adb serial number(same as you see with adb devices command)

automatic - Sequantially checks all available options for first non empty value.

Priority order:

Before 0.6: marathon_property -> boot_property -> hostname -> ddms -> UUID

After 0.6: marathon_property -> ddms -> boot_property -> hostname -> UUID

Install options

By default, these will be -g -r (-r prior to marshmallow). You can specify additional options to append to the default ones.

  • vendorConfiguration:
      type: "Android"
      installOptions: "-d"
    
  • marathon {
        installOptions = "-d"
    }
    
  • marathon {
        installOptions = "-d"
    }
    

Screen recorder configuration

By default, device will record a 1280x720 1Mbps video of up to 180 seconds if it is supported. If on the other hand you want to force screenshots or configure the recording parameters you can specify this as follows:

  • vendorConfiguration:
      type: "Android"
      screenRecordConfiguration:
        preferableRecorderType: "screenshot"
        videoConfiguration:
          enabled: false
          width: 1080
          height: 1920
          bitrateMbps: 2
          timeLimit: 300
        screenshotConfiguration:
          enabled: false
          width: 1080
          height: 1920
          delayMs: 200
    
  • marathon {
      screenRecordConfiguration = ScreenRecordConfiguration(
        RecorderType.SCREENSHOT,
        VideoConfiguration(
                false, //enabled
                1080, //width
                1920, //height
                2, //Bitrate in Mbps
                300 //Max duration in seconds
            ),
            ScreenshotConfiguration(
                false, //enabled
                1080, //width
                1920, //height
                200 //Delay between taking screenshots
            )
        )
    }
    
  • marathon {
        screenRecordConfiguration = ScreenRecordConfiguration(
            RecorderType.SCREENSHOT,
            VideoConfiguration(
                false, //enabled
                1080, //width
                1920, //height
                2, //Bitrate in Mbps
                300 //Max duration in seconds
            ),
            ScreenshotConfiguration(
                false, //enabled
                1080, //width
                1920, //height
                200 //Delay between taking screenshots
            )
        )
    }
    

Clear state between test executions

By default, marathon does not clear state between test batch executions. To mitigate potential test side-effects, one could add an option to clear the package data between test runs. Keep in mind that test side-effects might be present. If you want to isolate tests even further, then you should consider reducing the batch size.

Since pm clear resets the permissions of the package, the granting of permissions during installation is essentially overridden. Marathon doesn’t grant the permissions again. If you need permissions to be granted and you need to clear the state, consider alternatives like GrantPermissionRule

  • vendorConfiguration:
      type: "Android"
      applicationPmClear: true
      testApplicationPmClear: true
    
  • marathon {
        applicationPmClear = true
        testApplicationPmClear = true
    }
    
  • marathon {
        applicationPmClear = true
        testApplicationPmClear = true
    }
    

Instrumentation arguments

If you want to pass additional arguments to the am instrument command executed on the device like execute only “SMALL” tests:

  • vendorConfiguration:
      type: "Android"
      instrumentationArgs:
        size: small
    
  • marathon {
        instrumentationArgs { 
            set("size", "small")
        }
    }
    
  • marathon {
        instrumentationArgs { 
            set("size", "small")
        }
    }
    

Allure-kotlin support

If you want to enable on-device collection of allure’s reports, you can use the following option:

  • vendorConfiguration:
      type: "Android"
      allureConfiguration:
        enabled: true
    
  • marathon {
        allureConfiguration {
            enabled = true
        }
    }
    
  • marathon {
        allureConfiguration {
            enabled = true
        }
    }
    

Configuration above works out of the box for allure-kotlin 2.3.0+.

Additional configuration parameters include pathRoot which has two options: EXTERNAL_STORAGE that is usually the /sdcard/ on most of the devices and APP_DATA which is usually /data/data/$appPackage/. Besides the expected path root you might need to provide the relativeResultsDirectory: this is the relative path to pathRoot. The default path for allure-kotlin as of 2.3.0 is /data/data/$appPackage/allure-results. Please refer to allure’s documentation on the usage of allure.

Starting with allure 2.3.0 your test application no longer needs MANAGE_EXTERNAL_STORAGE permission to write allure’s output, so there is no need add any special permissions there.

Enabling this option effectively creates two allure reports for each test run:

  • one from the point of view of the marathon test runner
  • one from the point of view of on-device test execution

The on-device report gives you more flexibility and allows you to:

  • Take screenshots whenever you want
  • Divide large tests into steps and visualise them in the report
  • Capture window hierarchy and more.

All allure output from devices will be collected under $output/device-files/allure-results folder.

Vendor module selection

The first implementation of marathon for Android relied heavily on AOSP’s ddmlib. For a number of technical reasons we had to write our own implementation of the ADB client named adam.

The ddmlib’s implementation is going to be deprecated in marathon 0.7.0 and by default adam is going to be handling all communication with devices.

By 0.8.0, ddmlib is going to be removed completely unless we find major issues.

All the features supported in ddmlib’s implementation transparently work without any changes. We ask you to test adam prior to the removal of ddmlib and submit your concerns/issues.

  • vendorConfiguration:
      type: "Android"
      vendor: ADAM
    
  • marathon {
        vendor = com.malinskiy.marathon.config.vendor.VendorConfiguration.AndroidConfiguration.VendorType.ADAM
    }
    
  • marathon {
        vendor = com.malinskiy.marathon.config.vendor.VendorConfiguration.AndroidConfiguration.VendorType.ADAM
    }
    

Timeout configuration

With the introduction of adam we can precisely control the timeout of individual requests. Here is how you can use it:

  • vendorConfiguration:
      type: "Android"
      vendor: ADAM
      timeoutConfiguration:
        # ISO_8601 duration
        shell: "PT30S"
        listFiles: "PT1M"
        pushFile: "PT1H"
        pushFolder: "PT1H"
        pullFile: "P1D"
        uninstall: "PT1S"
        install: "P1DT12H30M5S"
        screenrecorder: "PT1H"
        screencapturer: "PT1S"
    
  • marathon {
        timeoutConfiguration {
            shell = Duration.ofSeconds(30)
        }
    }
    
  • marathon {
        timeoutConfiguration {
            shell = Duration.ofSeconds(30)
        }
    }
    

Please keep in mind that this timeout configuration is only for adam vendor, ddmlib will not pick up these settings.

Sync/pull files from device after test run

Sometimes you need to pull some folders from each device after the test execution. It may be screenshots or logs or other debug information. To help with this marathon supports pulling files from devices at the end of the test batch execution. Here is how you can configure it:

  • vendorConfiguration:
      type: "Android"
      fileSyncConfiguration:
        pull:
        - relativePath: "my-device-folder1"
          aggregationMode: TEST_RUN
          pathRoot: EXTERNAL_STORAGE
        - relativePath: "my-device-folder2"
          aggregationMode: DEVICE
          pathRoot: EXTERNAL_STORAGE
        - relativePath: "my-device-folder3"
          aggregationMode: DEVICE_AND_POOL
          pathRoot: EXTERNAL_STORAGE
        - relativePath: "my-device-folder4"
          aggregationMode: POOL
          pathRoot: EXTERNAL_STORAGE
    
  • marathon {
      fileSyncConfiguration {
        pull.add(
          new FileSyncEntry(
            "my-device-folder1",
            AggregationMode.TEST_RUN,
            PathRoot.EXTERNAL_STORAGE
          )
        )
        pull.add(
          FileSyncEntry(
            "my-device-folder2",
            AggregationMode.DEVICE,
            PathRoot.EXTERNAL_STORAGE
          )
        )
        pull.add(
          FileSyncEntry(
            "my-device-folder3",
            AggregationMode.DEVICE_AND_POOL,
            PathRoot.EXTERNAL_STORAGE
          )
        )
        pull.add(
          FileSyncEntry(
            "my-device-folder4",
            AggregationMode.POOL,
            PathRoot.EXTERNAL_STORAGE
          )
        )
      }
    }
    
  • marathon {
      fileSyncConfiguration {
        pull.add(
          FileSyncEntry(
            "my-device-folder1",
            AggregationMode.TEST_RUN,
            PathRoot.EXTERNAL_STORAGE
          )
        )
        pull.add(
          FileSyncEntry(
            "my-device-folder2",
            AggregationMode.DEVICE,
            PathRoot.EXTERNAL_STORAGE
          )
        )
        pull.add(
          FileSyncEntry(
            "my-device-folder3",
            AggregationMode.DEVICE_AND_POOL,
            PathRoot.EXTERNAL_STORAGE
          )
        )
        pull.add(
          FileSyncEntry(
            "my-device-folder4",
            AggregationMode.POOL,
            PathRoot.EXTERNAL_STORAGE
          )
        )
      }
    }
    

Please pay attention to the path on the device: if path root is EXTERNAL_STORAGE which is the default value if you don’t specify anything, then relativePath is relative to the Environment.getExternalStorageDirectory() or the EXTERNAL_STORAGE envvar. In practice this means that if you have a folder like /sdcard/my-folder you should specify /my-folder as a relative path.

Starting with Android 11 your test application will require MANAGE_EXTERNAL_STORAGE permission:

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android">
    <uses-permission android:name="android.permission.MANAGE_EXTERNAL_STORAGE"/>
    ...
</manifest>

Marathon will automatically grant this permission before executing tests if you pull/push files from devices with EXTERNAL_STORAGE path root.

If you don’t want to add any permissions to your test application, you can also use the path root APP_DATA. This will automatically transfer the files from your application’s private folder, e.g. /data/data/com.example/my-folder.

Push files to device before each batch execution

Sometimes you need to push some files/folders to each device before the test execution. Here is how you can configure it:

  • vendorConfiguration:
      type: "Android"
      fileSyncConfiguration:
        push:
        - path: "/home/user/folder"
        - path: "/home/user/testfile.txt"
    
  • marathon {
      fileSyncConfiguration {
        push.add(new FilePushEntry("/home/user/folder"))
        push.add(new FilePushEntry("/home/user/testfile.txt"))
      }
    }
    
  • marathon {
      fileSyncConfiguration {
        push.add(FilePushEntry("/home/user/folder"))
        push.add(FilePushEntry("/home/user/testfile.txt"))
      }
    }
    

By default, pushing will be done to LOCAL_TMP path root which refers to the /data/local/tmp. You can also push files to EXTERNAL_STORAGE. Currently, pushing to APP_DATA is not supported.

Test parser

Test parsing (collecting a list of tests expected to execute) can be done using either a local test parser, which uses byte code analysis, or a remote test parser that uses an Android device to collect a list of tests expected to run. Both have pros and cons listed below:

YAML type Gradle class Pros Const
“local” LocalTestParser Doesn’t require a booted Android device Doesn’t support runtime-generated tests, e.g. named parameterized tests. Doesn’t support parallelising parameterized tests
“remote” RemoteTestParser Supports any runtime-generated tests, including parameterized, and allows marathon to parallelise their execution Requires a booted Android device for parsing. If you need to use annotations for filtering purposes, requires test apk changes as well as attaching a test run listener for parser

Default test parser is local. If you need to parallelize the execution of parameterized tests or have complex runtime test generation (custom test runners, e.g. cucumber) - remote parser is your choice.

For annotations parsing using remote test parser test run is triggered without running tests (using -e log true option). Annotations are expected to be reported as test metrics, e.g.:

INSTRUMENTATION_STATUS_CODE: 0
INSTRUMENTATION_STATUS: class=com.example.FailedAssumptionTest
INSTRUMENTATION_STATUS: current=4
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: numtests=39
INSTRUMENTATION_STATUS: stream=
com.example.FailedAssumptionTest:
INSTRUMENTATION_STATUS: test=ignoreTest
INSTRUMENTATION_STATUS_CODE: 1
INSTRUMENTATION_STATUS: com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer.v2=[androidx.test.filters.SmallTest(), io.qameta.allure.kotlin.Severity(value=critical), io.qameta.allure.kotlin.Story(value=Slow), org.junit.Test(expected=class org.junit.Test$None:timeout=0), io.qameta.allure.kotlin.Owner(value=user2), io.qameta.allure.kotlin.Feature(value=Text on main screen), io.qameta.allure.kotlin.Epic(value=General), org.junit.runner.RunWith(value=class io.qameta.allure.android.runners.AllureAndroidJUnit4), kotlin.Metadata(bytecodeVersion=[I@bdf6b25:data1=[Ljava.lang.String;@46414fa:data2=[Ljava.lang.String;@5d4aab:extraInt=0:extraString=:kind=1:metadataVersion=[I@fbb1508:packageName=), io.qameta.allure.kotlin.Severity(value=critical), io.qameta.allure.kotlin.Story(value=Slow)]
INSTRUMENTATION_STATUS_CODE: 2
INSTRUMENTATION_STATUS: class=com.example.FailedAssumptionTest
INSTRUMENTATION_STATUS: current=4
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: numtests=39
INSTRUMENTATION_STATUS: stream=.
INSTRUMENTATION_STATUS: test=ignoreTest

To generate the above metrics you need to add a JUnit 4 listener to your dependencies:

dependecies {
  androidTestImplementation("com.malinskiy.adam:android-junit4-test-annotation-producer:${LATEST_VERSION}")
}

Then you need to attach it to the execution. One way to attach the listener is using am instrument parameters, e.g. -e listener com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer. Below you will find examples for configuring a remote test parser:

  • vendorConfiguration:
      type: "Android"
      testParserConfiguration:
        type: "remote"
        instrumentationArgs:
          listener: "com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer"
    
  • marathon {
      testParserConfiguration = TestParserConfiguration.RemoteTestParserConfiguration(
        mapOf(
          "listener" to "com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer"
        )
      )
    }
    

Keep in mind that instrumentationArgs should include a listener only for the test parser. During the actual execution there is no reason to produce test annotations.

Test access configuration

Marathon supports adam’s junit extensions which allow tests to gain access to adb on all devices and emulator’s control + gRPC port. See the docs as well as the PR for description on how this works.

  • vendorConfiguration:
      type: "Android"
      testAccessConfiguration:
        adb: true
        grpc: true
        console: true
        consoleToken: "cantFoolMe"
    
  • marathon {
      testAccessConfiguration = TestAccessConfiguration(
        adb = true,
        grpc = true,
        console = true,
        consoleToken = "cantFoolMe"
      )
    }
    

Multiple adb servers

Default configuration of marathon assumes that adb server is started locally and is available at 127.0.0.1:5037. In some cases it may be desirable to connect multiple adb servers instead of connecting devices to a single adb server. An example of this is distributed execution of tests using test access (calling adb commands from tests). For such scenario all emulators should be connected via a local (in relation to the emulator) adb server. Default port for each host is 5037.

In order to expose the adb server it should be started on all or public network interfaces using option -a. For example, if you want to expose the adb server and start it in foreground explicitly on port 5037: adb nodaemon server -a -P 5037.

This functionality is only supported by vendor adam because ddmlib doesn’t support connecting to a remote instance of adb server.

  • vendorConfiguration:
      type: "Android"
      adbServers:
        - host: 127.0.0.1
        - host: 10.0.0.2
          port: 5037
    
  • marathon {
      adbServers = listOf(
        AdbEndpoint(host = "127.0.0.1"),
        AdbEndpoint(host = "10.0.0.2", port = 5037)
      )
    }
    

Extra applications APK path

Install extra apk before running the tests if required, e.g. test-butler.apk

  • vendorConfiguration:
      type: "Android"
      extraApplicationsApk: 
        - "app/build/outputs/apk/androidTest/debug/extra.apk"
        - "app/build/outputs/apk/androidTest/debug/extra_two.apk"
    
  • marathon {
      extraApplications = listOf(
        File(project.rootDir, "test-butler-app-2.2.1.apk")
        File("/home/user/other-apk-with-absolute-path.apk"),
      )
    }
    

Keep in mind that for Gradle, the extraApplications parameter will affect all the testing apk configurations in a single module.

ScreenCapture API

Marathon supports automatic pull of screenshots taken via ScreenCapture API

To enable marathon to pull screenshots you need to use a custom ScreenCaptureProcessor called AdamScreenCaptureProcessor.

Firstly, add com.malinskiy.adam:android-junit4-test-annotation-producer:${LATEST_VERSION} to your test code.

Secondly, enable AdamScreenCaptureProcessor in your tests. You can do this manually:

Screenshot.addScreenCaptureProcessors(setOf(AdamScreenCaptureProcessor()))

or using JUnit4 rule AdamScreenCaptureRule:

class ScreenshotTest {
    ... 
  
    @get:Rule
    val screencaptureRule = AdamScreenCaptureRule()

    @Test
    fun testScreencapture() {
        ...
        Screenshot.capture().process()
        ...
    }
}

That’s it, you’re done. No need for custom configuration on Marathon’s side: everything should be picked up automatically.

More information on this custom ScreenCaptureProcessor can be found here.

Enable window animations

By default, marathon uses --no-window-animation flag. Use the following option if you want to enable window animations:

  • vendorConfiguration:
      type: "Android"
      disableWindowAnimation: false
    
  • marathon {
      disableWindowAnimation = false
    }