Logs

There are multiple logs and reports available in marathon in order to understand what’s going on. Let’s check all of them.

Stdout log

The first and easiest one to get is the stdout log that is printed during the execution of the test run. In order to have more information you can enable the debug mode via configuration.

Device emulator-5554 connected. Healthy = true
Device Unknown booted!
Installing application output to emulator-5554
Installing application output to emulator-5556
Uninstalling com.example from emulator-5554

Run instrumentation tests /Users/xxx/Development/marathon/sample/android-app/app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk for app /Users/xxx/Development/marathon/sample/android-app/app/build/outputs/apk/debug/app-debug.apk
Output: /Users/xxx/Development/marathon/sample/android-app/app/build/reports/marathon/debugAndroidTest
Ignore failures: false
Scheduling 28 tests
com.example.ClassIgnoredTest#testAlwaysIgnored, com.example.FailedAssumptionTest#failedAssumptionTest, com.example.FailedAssumptionTest#ignoreTest, com.example.MainActivityFlakyTest#testTextFlaky, com.example.MainActivityFlakyTest#testTextFlaky1, com.example.MainActivityFlakyTest#testTextFlaky2, com.example.MainActivityFlakyTest#testTextFlaky3, com.example.MainActivityFlakyTest#testTextFlaky4, com.example.MainActivityFlakyTest#testTextFlaky5, com.example.MainActivityFlakyTest#testTextFlaky6, com.example.MainActivityFlakyTest#testTextFlaky7, com.example.MainActivityFlakyTest#testTextFlaky8, com.example.MainActivitySlowTest#testTextSlow, com.example.MainActivitySlowTest#testTextSlow1, com.example.MainActivitySlowTest#testTextSlow2, com.example.MainActivitySlowTest#testTextSlow3, com.example.MainActivitySlowTest#testTextSlow4, com.example.MainActivityTest#testText, com.example.MainActivityTest#testText1, com.example.MainActivityTest#testText2, com.example.MainActivityTest#testText3, com.example.MainActivityTest#testText4, com.example.MainActivityTest#testText5, com.example.MainActivityTest#testText6, com.example.MainActivityTest#testText7, com.example.MainActivityTest#testText8, com.example.MainActivityTest#testText9, com.example.ParameterizedTest#test
device emulator-5554 associated with poolId omni
pool actor omni is being created
add device emulator-5554
initialize emulator-5554

Installing com.example, /Users/xxx/Development/marathon/sample/android-app/app/build/outputs/apk/debug/app-debug.apk to emulator-5554
Success
Installing instrumentation package to emulator-5554
Uninstalling com.example.test from emulator-5554
Installing com.example.test, /Users/xxx/Development/marathon/sample/android-app/app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk to emulator-5554
Success
Prepare installation finished for emulator-5554
tests = []
Starting recording for ClassIgnoredTest.testAlwaysIgnored

....

terminate emulator-5554
Allure environment data saved.
Marathon run finished:
Device pool omni:
	23 passed, 2 failed, 3 ignored tests
	Flakiness overhead: 0ms
	Raw: 23 passed, 2 failed, 3 ignored, 0 incomplete tests
Total time: 0H 0m 57s

Raw json log

In case you want to produce a custom report or you want to push metrics based on the results of the execution raw json is probably your best option. Each test is written as a json object inside an array. Keep in mind that this report shows retries, so you basically have full access to what happened during the execution.

This log file can be found at the $output/test_result/raw.json

{
  "package": "com.example",
  "class": "SomeTest",
  "method": "checkSomethingActuallyWorks",
  "deviceSerial": "XXXXXXXXXXXX",
  "ignored": false,
  "success": true,
  "timestamp": 1548142665055,
  "duration": 13370
}

Reports

Html report

This report can be found at $output/html/index.html

Device pools are separated on the main page of the report:

html report home page

The pool part of the report shows test result (success, failure, ignored), video/screencapture if it’s available and also device’s log. Filtering by test name and class is supported:

html report home page

Timeline log

This report is now part of html report but if you need to view it separately it can be found at $output/html/timeline/index.html

Timeline helps identify potential misbehaviours visually helping infrastructure teams identify faulty devices and also helping developers identify tests which affect the whole batch. For example you have a test which doesn’t cleanup properly after execution and all the tests after this one will fail in the same batch.

timeline report home page

Tip: Hovering over a test batch will show you the name of the test.

JUnit4 report

Xml files are written according to the Junit4 specification to integrate with existing tools such as Continuous Integration servers or third-party tools such as Danger.

All the reports for respective pools are placed at $output/tests/${poolName}/*.xml.

Allure report

allure report helps identify multiple possible problems during test execution. Be aware that marathon generates only the data files necessary to compile the actual report via commands line or plugin options available for allure.

allure report home page

One of the best use cases for this report is the grouping of problems which helps to identify if your tests have a specific issue that is relevant for a large number of tests. Another useful feature is the ability to see all the retries visually and with video/screencapture (if available). Allure also provides the ability to see flaky tests by storing history of the runs.

The allure jsons can be found at $output/allure-results

Dashboards

Marathon is able to leverage the power of InfluxDB and Graphite to store and fetch the metrics about the test runs. There are sample dashboards for InfluxDB and Graphite that you can use in grafana to visualise this information in order to better understand what’s going on.

grafana influxdb dashboard