A new testing harness about mobile HOT 15 OPEN

Expanded out of #22

from mobile.

XCTest is now OSS, could make life easier.

from mobile.

The open-source implementation of XCTest is a simple program that prints to stdout. Last we spoke, this project involved IPC, some reverse-engineering of iOS simulator internals, and more--open-source XCTest isn't going to be of much help there.

from mobile.

^ This is a legit point

from mobile.

Alright so, based on what I learned in code spelunking in Injection for Xcode, here's what I think should happen.

Let's just call it instatests for now. Create a gem called cocoapods-instatests.

Why a CocoaPods plugin? Well we need to add code to the app to start listening out for , people shouldn't need to have it on every computer and ideally it shouldn't get shipped with your app's source, sounds exactly like cocoapods-keys.

The code inside the pod needs to:

• Wait until an XCTest session is about to close, check for any instatests servers.
• If it doesn't find one, close like normal.
• If one exists, create a connection ( Injection uses sockets, could we use NSXPC?)
• Receive NSBundles, run the code as tests in XCTest
• The code can be added entire as a #ifdef DEBUG so it won't go to prod

This plugin has a command pod instatests which starts up a guard-like server. This detects saves, and creates a bundle representing the changes. These get passed to the pod's server, ran, then it gets test results back, these can use NSNotifications to let you know if it passed/failed.

Once a working build is done, then I can look at standardizing all the communication points.

from mobile.

could we use NSXPC?

Depends. How will the bundles with new code be compiled? By Xcode or do you want to do a stand-alone thing, possibly with a Clang compilation database?

from mobile.

Simplest feels to be the same as injection - read the build log, find the section for that file, and reuse that. In injection that .o is linked into a blank bundle, which gets passed across the connection.

Though I do want to consult Diamond's reloaded before making the first approach

from mobile.

Current feels on a name, xctestjam

from mobile.

Don't think XPC is probably the right communication method, weirdly enough, JSCore should have a working websocket, could use that as the intermediary for the client/server to bi-directionally communicate. Meta.

from mobile.

Got a web socket to connect between a ruby webserver and a WKWebView websocket - looks like the simplest way to start

from mobile.

Now that I've started this, https://github.com/orta/cocoapods-xcautotest/

I'd like to try scope 1.0 as "runs tests, reliably" and 2.0 as "runs tests reliably, in the background, on a separate sim."

This looks a bit tough, but definitely doable, as it looks like I'll have to create ruby bindings for the frameworks generated by https://github.com/facebook/FBSimulatorControl - they have a command-line tool but it looks like it depends on having Carthage, which I'd rather not have as a dependency.

from mobile.

but it looks like it depends on having Carthage

Seems to only be for testing (OCMock).

from mobile.

Nah, it is used for the the command-line tool itself - the frameworks themselves are dependecy-less - https://github.com/facebook/FBSimulatorControl/tree/master/fbsimctl

from mobile.

@orta @alloy As far as Carthage goes we need it for two reasons:

• For OCMock tests for the Frameworks. I'm all for ripping out this dependency by replacing with hand-written doubles.
• ForSwifter with fbsimctl, which can run with an embedded HTTP Server.

If you need to build the Frameworks (XCTestBoostrap/FBSimulatorControl/FBDeviceControl), then you don't need any use of Carthage whatsoever. If you want to use the fbsimctl command-line you'll need it for building the fbsimctl executable. If you have some suggestions here, please let me know.

If you're interested in linking against the XCTestBootstrap framework from Ruby with a FFI and avoiding the fbsimctl executable, you should look at FBTestManagerReporter. These are the Delegate methods that Xcode/xcodebuild would get if launched via these tools. The XCTest protocols themselves are really quite good and should be more than sufficient for making a custom reporter. And for Simulators, XCTestBootstrap enables you to run against multiple booted Simulators on the same host.

from mobile.

Awesome - thanks @lawrencelomax 👍

from mobile.