Photo by Bannon Morrissy on Unsplash

iOS CI Separating build & test jobs on Gitlab

Usually when running automated test on CI we will have testing suites that run (like unit test/integration/etc..). When developing mobile applications with Swift we need to compile our testing targets before we can run them, usually the simpler solution is to run them together as a single CI job. I will try to explore the option of breaking building and testing into separate CI jobs.

TL;DR

If you have a single testing target you are probably better off building and testing at the same CI job, if you have more it is a different story.

Tools

I will be using the following tools to for running the build & test

• Gitlab for CI and Hosting
• Fastlane for writing the xCode actions

Build for testing

When writing a test for an iOS app on xCode we have the option to build for testing.

Build for testing

Running this will compile the app into a testable package that contains the code from the testing target that is defined in the scheme that we are building.

Scheme settings

This type of package that is produced by the build for testing is.xctest , and xcode will create an .xctest package for each testing target in the scheme.

The packages can be quite large as the contain all the app code and dependancies as well as the testing code and config.

Using fastlane for building

fastlane gives us a very convenient tool to use called scan, which essentially wraps xcode-build and other API’s into a very easy to use tool. we can run scan as follows in order to create a build for testing for a specific scheme.

lane :build_for_testing do
   scan(scheme: "BuildAndTest"
   derived_data_path: "build",
   build_for_testing: true )
end

• scheme: defines which scheme to use
• derived_data_path: changes the default derivedData folder path for the build, this allows us easy access to the build artifacts
• build_for_testing: builds the scheme for testing

Testing compiled packages

Now that we have the compile .xctest files we can run the tests.

We will use scan again to run the tests.

lane :test do
   scan(scheme: "BuildAndTest"
   derived_data_path: "build",
   test_without_building: true )
end

• derived_data_path: must be the same as the build lane so that the script will be able to find the compiled packages.
• test_without_building: will attempt to run the tests for a provided compiled package.

Running build & test in separate job on CI

If we were to run both lanes in the same CI job as follows it would build and test the our code

build_and_test:
   script:
   - fastlane build_for_testing
   - fastlane test

But say that we want to run the testing in a separate job, all we will have to do is create 2 stages (because testing depends on building to finish and they can’t run at the same time), and make sure to pass the .xctest packages to the testing job. it would look something like this:

stages:
- build
- test
build:
   stage: build
   script: 
   - fastlane build_for_testing
   artifacts:
      when: on_success
      paths:
      - build/**/*.xctest
test:
   stage: test
   script:
   - fastlane test

What we did here is we defined 2 jobs build and test . build will run first, creating the .xctest files and uploading then to the gitlab pipeline artifact storage (if built finish successful). Then test will run, the artifact are part of the pipeline, they maintain their original folder structure, and are available to all jobs that are running in the following stage by default, so test will have the build folder with our .xctest files available for running, and all it does is running the tests.

Conclusions

pros —

• Better separation of CI pipeline and actions, making them more atomic, which makes it easier to understand if a pipeline failed and where just by looking at the pipeline
• Re-run flaky tests — if we have a flaky test suite we can re-run very quickly because we do not need to wait for the code to compile.

cons —

• It can take more time — from my testing it takes about 1–2~ more minutes for the whole process of running a testing suite, the reason being that we need to upload and download big artifacts at the start and end of every build + checking out the git branch again, those process together take about 1–2 minutes.

I think that this can be very beneficial in certain cases, imagine that you have 3 testing suites in your project, you can compile them all in a single job, that will produce 3 different testing packages where each can be tested in its own job, that would actually save time because we only have to build once, and re-running a test would be very fast.