2014-10-02

In the previous post I mentioned a code pattern frequently used in tests:

Run the RunLoop until something specific happens.

I’ve written and rewritten this method, making the same mistakes a few times. Here are two of the latest implementations I had ended up with:

Variant A:

- (void) runUntilBlock:(BOOL(^)())block timeout:(NSTimeInterval)timeout
{
    NSDate * endDate = [NSDate dateWithTimeIntervalSinceNow:timeout];
    do {
        CFTimeInterval quantum = 0.0001;
        CFRunLoopRunInMode(kCFRunLoopDefaultMode, quantum, false);
    } while( [timeoutDate timeIntervalSinceNow]>0.0 && !block() );
}

(Don’t do that.)

Variant B:

- (void) runUntilBlock:(BOOL(^)())block timeout:(NSTimeInterval)timeout
{
    NSDate * endDate = [NSDate dateWithTimeIntervalSinceNow:timeout];
    while( [timeoutDate timeIntervalSinceNow]>0.0 && !block() ) {
        CFRunLoopRunInMode(kCFRunLoopDefaultMode, DBL_MAX, true);
    }
}

(Don’t do that, either.)

Neither felt particularly satisfying:

  • Variant A runs the RunLoop for 0.0001 second no matter what, then checks for completion and timeout, then runs the RunLoop again, etc. This is me doing polling, but with a small delay, because I was told that polling is bad.
  • Variant B is a bit better: it runs the RunLoop, blocking until one event is processed (that’s what the third parameter set to true is for), then checks for completion and timeout. However, there again, why am I handling a timeout myself? And why do I need a while loop? Isn’t that precisely the job of CFRunLoop?

The biggest difference between the two variants if the polling mechanism. Typically, the stop condition is expected to be fulfilled after some code runs on the same thread as the RunLoop, which means this code would be called out by the RunLoop. In variant B, that’s the “processed event” that makes CFRunLoopRunInMode return: as soon as the code that fulfills the condition has run, CFRunLoop returns and we check the condition.

On the other hand, if the condition we’re testing only becomes true as the result of code running on other threads, we need to check for completion regularly, and there may be no solution but polling.1

XCTestExpectation

In Xcode 5, Apple added Asynchronous unit testing, with the new class XCTestExpectation and a category on XCTestCase. It’s really great, and it comes with a few very nice helpers for KVO or NSNotification-based tests. However, it has a few limitations2:

  • I can only be used for asynchronous tasks with an explicit callback. The test code has to send the -[XCTestExpectation fulfill] message for the test to pass. This is simply impossible in some cases: for example, if the asynchronous task being tested simply changes the value of a flag, there’s no client callback to do this. Arguably, that’s a sign of a bad API, or that you’re trying to test the wrong things. Unfortunately, some of the things I want to do are precisely this kind of “Black Box” tests. (More on that below.)
  • It can only fail by timing out. If the task finishes, but with an error, should it really send the fulfill message? In my opinion, running the app asynchronously should be not be a feature of the test framework.

CTT

To write the Capitaine Train iOS app, we’ve setup a large suite of tests including high-level integration tests. For example, here’s the simulator running the -test_login scenario.

The early versions of these tests were written using KIF. KIF is really powerful, but after a while, its syntax became too heavy to write dozens of scenarios.

KIF:

[tester enterText:@"testuser@capitainetrain.com" intoViewWithAccessibilityLabel:@"Email Address"];
[tester enterText:@"testpassword" intoViewWithAccessibilityLabel:@"Password"];
[tester tapViewWithAccessibilityLabel:@"Sign In"];

[tester waitForTappableViewWithAccessibilityLabel:@"Search"];

So after a while, we wrote our own system, dubbed CTT. In CTT, the same test looks like:

CTT:

[[CTT match:@"Email Address"] enterText:@"testuser@capitainetrain.com"];
[[CTT match:@"Password"] enterText:@"testpassword"];
[CTT tap:@"Sign In"];

[CTT match:@"Search" options:CTTMatchOptionsTappable];

The whole point of this post (and hopefully, the next ones) is to make me extract CTT, document it, and open-source it.3

By I digress. The hardest thing with Integration testing is to run the app normally, while waiting for specific views to appear. We need to run the RunLoop, until an arbitrary condition becomes true.

CTTRunLoopRunUntil

Here’s what I believe is a correct implementation of CTTRunLoopRunUntil:4

Boolean CTTRunLoopRunUntil(Boolean(^fulfilled_)(), Boolean polling_, CFTimeInterval timeout_)
{
    // Loop Observer Callback
    __block Boolean fulfilled = NO;
    void (^beforeWaiting) (CFRunLoopObserverRef observer, CFRunLoopActivity activity) =
    ^(CFRunLoopObserverRef observer, CFRunLoopActivity activity) {
        assert(!fulfilled); //RunLoop should be stopped after condition is fulfilled.
        // Check Condition
        fulfilled = fulfilled_();
        if(fulfilled) {
            // Condition fulfilled: stop RunLoop now.
            CFRunLoopStop(CFRunLoopGetCurrent());
        } else if(polling_) {
            // Condition not fulfilled, and we are polling: prevent RunLoop from waiting and continue looping.
            CFRunLoopWakeUp(CFRunLoopGetCurrent());
        }
    };
    
    CFRunLoopObserverRef observer = CFRunLoopObserverCreateWithHandler(NULL, kCFRunLoopBeforeWaiting, true, 0, beforeWaiting);
    CFRunLoopAddObserver(CFRunLoopGetCurrent(), observer, kCFRunLoopDefaultMode);
    
    // Run!
    CFRunLoopRunInMode(kCFRunLoopDefaultMode, timeout_, false);

    CFRunLoopRemoveObserver(CFRunLoopGetCurrent(), observer, kCFRunLoopDefaultMode);
    CFRelease(observer);

    return fulfilled;
}

The idea is to use of CFRunLoopObservers. As mentioned in the previous post, CFRunLoopObservers are used to setup callbacks at different checkpoints of the RunLoop’s loop:

/* Run Loop Observer Activities */
typedef CF_OPTIONS(CFOptionFlags, CFRunLoopActivity) {
    kCFRunLoopEntry = (1UL << 0),
    kCFRunLoopBeforeTimers = (1UL << 1),
    kCFRunLoopBeforeSources = (1UL << 2),
    kCFRunLoopBeforeWaiting = (1UL << 5),
    kCFRunLoopAfterWaiting = (1UL << 6),
    kCFRunLoopExit = (1UL << 7),
    kCFRunLoopAllActivities = 0x0FFFFFFFU
};

By observing kCFRunLoopBeforeWaiting, we can test for completion on every loop of the RunLoop. Before sleeping (i.e. waiting for an event), the RunLoop has called everything there was to call. That’s the right time to test for completion. This variant also solves the “active polling” scenarios: if the polling_ flag is set, the RunLoop actually never sleeps and run continuously; fulfilled_ is checked on every pass. And of course, contrary to most implementations, including my own, there’s no “minimal delay”, and no additional code to handle the loop or the timeout. That should do the trick.

If you see an error or a possible improvement, please do let me know. Here’s the code for CTTRunLoopRunUntil on github; the repository is pretty bare right now; hopefully, I’ll gradually add all the functionality of CTT into it. It’ll involve poking at UIView internals, accessibility and keyboard private APIs, general UIKit hackery, and some smarty-pants C macros. Stay tuned.

  1. To be exact: if an asynchronous task calls back client code on a secondary thread, the correct way to make the main thread check for completion is by waking its RunLoop using CFRunLoopWakeUp(), not to use polling. Polling is only necessary if the asynchronous task doesn’t call back client code at all

  2. Additionally, as there’s no XCAssert macro, it looks completely different from the rest of the XCUnit API. I wonder how it prints the error at the correct line without using __FILE__ and __LINE__

  3. By the way, if you want to help me, we’re hiring

  4. Until I realize there’s another significant flaw.