mobilenativefoundation / bluepill Goto Github PK
View Code? Open in Web Editor NEWBluepill is a reliable iOS testing tool that runs UI tests using multiple simulators on a single machine
License: BSD 2-Clause "Simplified" License
Bluepill is a reliable iOS testing tool that runs UI tests using multiple simulators on a single machine
License: BSD 2-Clause "Simplified" License
Listing tests by default just lists the test bundles, which I think is kind of useless. Had I not looked at the code and noticed that values greater than 1 indicate verbose mode which lists all the tests in the bundle, I would have tried to find a different solution.
I think listing all of the tests in the bundle should be default, and values greater than 1 indicate "quiet" mode if the other functionality is desired.
The various outputs of bluepill have become quite a mess. I'm opening this issue as a place for discussion about what the different output modes should be. My proposal is as follows:
--quiet
only prints ERROR
messages in stderr--debug
prints DEBUGINFO
level messages in stderrERROR
, WARNING
, and INFO
messages.BP_QUIET_MODE
and BP_DEBUG_MODE
that can be used to set the two modes. The command line options override the environment.BPBuildScript
environment variable and have bluepill.sh
use the previously mentioned environment variables instead.Thoughts?
Need help with the error.
[ INFO ] Using 781 of 1064 processes.
[ ERROR ] (2) Completed: [1] Launch Application
[ ERROR ] (2) Could not launch app and tests: The request to open "com.xyz.internal.debug" failed.
[ INFO ] (2) [1] Delete Simulator
[ ERROR ] (3) Completed: [1] Launch Application
[ ERROR ] (3) Could not launch app and tests: The request to open "com.xyz.internal.debug" failed.
If you do a standard Cmd+B build and point bluepill to the .app
, you get this error:
ERROR: The folder “Plugins” doesn’t exist.
Would be good to print a helpful message along the lines of "Please build the test target of your app (Cmd + Shift + U
)"
I Run single test using xcodebuild like this
xcodebuild test -workspace xxxWorkspace.xcworkspace/ -scheme "xxx.com Debug-UITests" -destination "platform=iOS Simulator,name=iPhone 6s,OS=latest" -only-testing:zzzUITests/SmokeTest
Can I run the similar test using bluepill?
Thanks
How to change language during test? For instance, launch five tests simultaneously for five different languages (Japanese, French, German, Spanish and Chinese)
Thanks
Hi
let say I have test targets targetA, TargetB and want to run like below. Is that possible if yes how?
targetA ----iPhone 6 version 10.1
targetB ----iPhone 6 version 10.1
It is taking about 2.5 minutes for data to be saved in Core Data while running a test using Bluepill.
Subsequent saves are quick until the simulator is restarted and the initial Core Data save is slow again.
We import JSON data over the web, convert it to objects and save the objects in core data at intervals as we loop over the JSON. The save is done on a background thread.
Using Xcode to run the same test all the Core Data saves are about a second or sub second to complete as expected.
When debugging failures, it'd be useful if bluepill printed the UUID of the simulator when it creates a new one. E.g.
[ FAILED ] (1) Completed: [1] Create Simulator
Doesn't help debugging failures. If it printed the UUID we could go dig for crash logs.
When running tests, it appears each simulator is running the same test...
[ INFO ] 4 Simulators still running. [1, 2, 3, 4]
[ INFO ] Using 794 of 1064 processes.
[ FAILED ] (3) 67.031106s HelpAndFeedback/testSubmittingFeedback
[ FAILED ] (3) 67.032047s HelpAndFeedback/testSubmittingFeedback
[ FAILED ] (3) 67.033130s HelpAndFeedback/testSubmittingFeedback
I'd anticipated all tests within the target to be distributed equally among the simulators.
Command
bluepill -a ./build/Unit\ Test-iphonesimulator/GuestCenter.app -s ./GuestCenter.xcodeproj/xcshareddata/xcschemes/Acceptance\ Tests1.xcscheme -o ./output/ -d "iPad Pro (9.7-inch)"
Of note, we have our UI tests split between 4 targets currently to minimize their run times. In the command above, I'm trying to only run the Acceptance Tests1
tests, however the command is loading all tests in the product.
Am I correct in understanding what I'm seeing? If so, suggestions on what I'm doing wrong?
Hi, I'm not sure if I do it correctly, but here are my steps to run the demo
Bluepill.xcworkspace
, scheme blueprint
, I get bp
executable inside build/Build/Products/Debug
BPSampleApp.xcodeproj
, scheme BPSampleApp
, and copy the BPSampleApp.app
to the same folder as bp
BPAppNegativeTests.xcscheme
inside BPSampleApp.xcodeproj/xcshareddata/xcschemes
to the same folder as bp
./bp -a ./BPSampleApp.app -s ./BPAppNegativeTests.xcscheme -o ./output/
and getbp: invalid configuration. No test bundle provided.
If I need to do these steps to get the bluepill to run, then I must be doing something wrong. Please help
I spent some time puzzled about this error:
$ bluepill -c config.json
[ INFO ] This is Bluepill v0.1.0-21-g95fcfd8
[ INFO ] Running with 4 simulators.
[ INFO ] Packed tests into 6 bundles
[ INFO ] Started Simulator 1 (PID 53404).
bp: invalid configuration
iPhone 6 is not a valid device type.
Use `xcrun simctl list devicetypes` for a list of valid devices.
It turns out that I had the wrong path in my config.json
file. Bluepill should at least check that the Xcode path give is valid and not fall through to a misleading error.
We were trying to address issue #12 - [self.device terminateApplicationWithID:self.hostBundleId error:&error]; in SimulatorMonitor.m doesn't work in iOS 10.2 runtime.
in PR #19 , however, it is still flaky, seeing hangings sometimes.
Initial inspection is that - the SimulatorRunner
may be nil. Assigned to @vargon to take a look
It'd be great if the console reported an overall progress indicator for each running simulator.
Examples
[ INFO ] (1) [###### ] 40% Complete
[ INFO ] (2) [######## ] 55% Complete
[ INFO ] (1) [###### ] 40% Complete (21 passed, 4 failed, 2 crashed)
[ INFO ] (2) [######## ] 55% Complete (19 passed, 7 failed, 0 crashed)
I'm trying to debug the cause of this but I'm unable to figure it out.
There should be a verbose logging mode which prints all the NSError descriptions so we can figure out such issues.
After we merged the change "Fix hangs (#49)", we started to observed the errors like
20170208.154417 [ ERROR ] (12) Application crashed before tests started!
20170208.154417 [ INFO ] (12) [11] Delete Simulator,
but the test cases were actually running well without crash. It happens in the CI environment and also fail the bluepill integration test cases.
To reproduce with the test case, check out the reuse-sim branch, run test case BluepillRunnerTests/BPIntegrationTests/testOneBPInstance
To see the error messages, you need to disable quite mode first in the line 47 of BPIntegrationTests.m:
//self.config.quiet = YES;
[reportXML nodesForXPath:@".//testsuites" error:nil]
returns nil
for the output, so compareElement
early returns.
I found this because I was wondering how this actual resulting XML file:
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="Selected tests" tests="0" failures="0" errors="0" time="0.002000">
<testsuite tests="0" failures="0" errors="0" time="0.000000" timestamp="2017-02-13T16:05:39GMT-07:00" name="BPSampleAppCrashingTests.xctest">
</testsuite>
</testsuites>
was being considered equal to the expected one:
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="All tests" tests="3" failures="1" errors="1" time="0.002000">
<testsuite tests="2" failures="1" errors="1" time="0.041532" timestamp="2016-11-07T16:54:44GMT-08:00" name="BPSampleAppCrashingTests.xctest">
<testsuite tests="2" failures="1" errors="1" time="0.020770" timestamp="2016-11-07T16:54:44GMT-08:00" name="BPSampleAppCrashingTests">
<testcase classname="BPSampleAppCrashingTests" name="testAppCrash0" time="0.000000">
</testcase>
<testcase classname="BPSampleAppCrashingTests" name="testAppCrash1" time="0.000000">
<failure type="Failure" message="App Crashed">
Unknown File:0
</failure>
<error type="Error"/>
</testcase>
</testsuite>
</testsuite>
<testsuite tests="1" failures="0" errors="0" time="0.001000" timestamp="2016-11-07T16:55:13GMT-08:00" name="BPSampleAppCrashingTests.xctest">
<testsuite tests="1" failures="0" errors="0" time="0.000000" timestamp="2016-11-07T16:55:13GMT-08:00" name="BPSampleAppCrashingTests">
<testcase classname="BPSampleAppCrashingTests" name="testAppCrash2" time="0.000000">
</testcase>
</testsuite>
</testsuite>
</testsuites>
Hey folks,
Really excited to give bluepill a whirl. I cloned the repository and ran build.sh
and it's bailing with 11 test failures:
-[BPCLITests testNoSchemeinCLI]
-[BluepillTests testReportWithAppCrashingTestsSet]
-[BluepillTests testReportWithAppHangingTestsSet]
-[BluepillTests testReportWithAppHangingTestsShouldReturnFailure]
-[BluepillTests testReportWithFailingTestsSet]
-[BluepillTests testReportWithFatalErrorTestsSet]
-[BluepillTests testRetryWithFailingTestsSet]
-[BluepillTests testRunningOnlyCertainTestcases]
-[BluepillTests testRunWithConfigurationFile]
-[BluepillTests testRunWithPassingTestsSet]
-[BluepillTests testRunWithRelativePathInConfigurationFile]
Wondering if this a bug, configuration problem, or something else? Is there a better forum for troubleshooting or is this an appropriate place? Adding an installation or getting started section to README would be great and I'd be happy to contribute.
My English is not good, So you may not understand what I mean.when i use bluepill
, and I have some problems.These are my input codes:
./bluepill -a /Users/Wil/Library/Developer/Xcode/DerivedData/someApp-dtewmgszpypiziaywgdbcnictchx/Build/Products/Debug-iphonesimulator/someApp.app -s /Users/Wil/Desktop/bluepill_build/someApp.xcodeproj/xcuserdata/Wil.xcuserdatad/xcschemes/someApp_Tests.xcscheme -o build/output/ -r com.apple.CoreSimulator.SimRuntime.iOS-10-0
and I got this information
[ INFO ] This is Bluepill v0.1.0-33-g0d56983 [ WARNING] Trying to pack too few tests (3) into too many bundles (4). [ WARNING] Lowering number of simulators from 4 to 3 because there aren't enough tests. [ INFO ] Running with 3 simulators. [ INFO ] Packed tests into 3 bundles [ INFO ] Started Simulator 1 (PID 11174). bp: invalid configuration com.apple.CoreSimulator.SimRuntime.iOS-10-0 is not a valid runtime. Use `xcrun simctl list runtimes` for a list of valid runtimes. Usage: -a | --app The path to the host application to execute (your .app) -o | --output-dir Directory where to put output log files (bluepill only). -s | --scheme-path The scheme to run tests. -d | --device On which device to run the app. (Default 'iPhone 6'). -c | --config Read options from the specified configuration file instead of the command line -C | --repeat-count Number of times we'll run the entire test suite (used for stability testing). (Default '1'). -N | --no-split A list of NO split test bundles -P | --print-config Print a configuration file suitable for passing back using the `-c` option. (Default 'stdout'). -R | --error-retries Number of times we'll recover from crashes to continue running the current test suite. (Default '4'). -S | --stuck-timeout Timeout in seconds for a test that seems stuck (no output). (Default '300'). -T | --test-timeout Timeout in seconds for a test that is producing output. (Default '300'). -f | --failure-tolerance The number of retries on any failures (app crash/test failure). -i | --include Include a testcase in the set of tests to run (unless specified in `exclude`). -n | --num-sims Number of simulators to run in parallel. (bluepill only) (Default '4'). -r | --runtime What runtime to use. (Default 'iOS 10.2'). -t | --test The path to the test bundle to execute (your .xctest). -x | --exclude Exclude a testcase in the set of tests to run (takes priority over `include`). -X | --xcode-path Path to xcode. -H | --headless Run in headless mode (no GUI). (Default 'Off'). -J | --json-output Print test timing information in JSON format. (Default 'Off'). -h | --help This help. -p | --plain-output Print results in plain text. (Default 'Off'). -q | --quiet Turn off all output except fatal errors. (Default 'Off'). -j | --junit-output Print results in JUnit format. (Default 'Off'). -l | --list-tests Only list tests in bundle --additional-xctests Additional XCTest bundles to test. [ INFO ] Simulator 1 (PID 11174) has finished with exit code 1. [ INFO ] PID 11174 exited 1. [ INFO ] 0 Simulators still running. [] [ INFO ] Using 387 of 1064 processes. [ INFO ] Started Simulator 2 (PID 11177). bp: invalid configuration com.apple.CoreSimulator.SimRuntime.iOS-10-0 is not a valid runtime. Use `xcrun simctl list runtimes` for a list of valid runtimes. Usage: -a | --app The path to the host application to execute (your .app) -o | --output-dir Directory where to put output log files (bluepill only). -s | --scheme-path The scheme to run tests. -d | --device On which device to run the app. (Default 'iPhone 6'). -c | --config Read options from the specified configuration file instead of the command line -C | --repeat-count Number of times we'll run the entire test suite (used for stability testing). (Default '1'). -N | --no-split A list of NO split test bundles -P | --print-config Print a configuration file suitable for passing back using the `-c` option. (Default 'stdout'). -R | --error-retries Number of times we'll recover from crashes to continue running the current test suite. (Default '4'). -S | --stuck-timeout Timeout in seconds for a test that seems stuck (no output). (Default '300'). -T | --test-timeout Timeout in seconds for a test that is producing output. (Default '300'). -f | --failure-tolerance The number of retries on any failures (app crash/test failure). -i | --include Include a testcase in the set of tests to run (unless specified in `exclude`). -n | --num-sims Number of simulators to run in parallel. (bluepill only) (Default '4'). -r | --runtime What runtime to use. (Default 'iOS 10.2'). -t | --test The path to the test bundle to execute (your .xctest). -x | --exclude Exclude a testcase in the set of tests to run (takes priority over `include`). -X | --xcode-path Path to xcode. -H | --headless Run in headless mode (no GUI). (Default 'Off'). -J | --json-output Print test timing information in JSON format. (Default 'Off'). -h | --help This help. -p | --plain-output Print results in plain text. (Default 'Off'). -q | --quiet Turn off all output except fatal errors. (Default 'Off'). -j | --junit-output Print results in JUnit format. (Default 'Off'). -l | --list-tests Only list tests in bundle --additional-xctests Additional XCTest bundles to test. [ INFO ] Simulator 2 (PID 11177) has finished with exit code 1. [ INFO ] PID 11177 exited 1. [ INFO ] Started Simulator 3 (PID 11181). bp: invalid configuration com.apple.CoreSimulator.SimRuntime.iOS-10-0 is not a valid runtime. Use `xcrun simctl list runtimes` for a list of valid runtimes. Usage: -a | --app The path to the host application to execute (your .app) -o | --output-dir Directory where to put output log files (bluepill only). -s | --scheme-path The scheme to run tests. -d | --device On which device to run the app. (Default 'iPhone 6'). -c | --config Read options from the specified configuration file instead of the command line -C | --repeat-count Number of times we'll run the entire test suite (used for stability testing). (Default '1'). -N | --no-split A list of NO split test bundles -P | --print-config Print a configuration file suitable for passing back using the `-c` option. (Default 'stdout'). -R | --error-retries Number of times we'll recover from crashes to continue running the current test suite. (Default '4'). -S | --stuck-timeout Timeout in seconds for a test that seems stuck (no output). (Default '300'). -T | --test-timeout Timeout in seconds for a test that is producing output. (Default '300'). -f | --failure-tolerance The number of retries on any failures (app crash/test failure). -i | --include Include a testcase in the set of tests to run (unless specified in `exclude`). -n | --num-sims Number of simulators to run in parallel. (bluepill only) (Default '4'). -r | --runtime What runtime to use. (Default 'iOS 10.2'). -t | --test The path to the test bundle to execute (your .xctest). -x | --exclude Exclude a testcase in the set of tests to run (takes priority over `include`). -X | --xcode-path Path to xcode. -H | --headless Run in headless mode (no GUI). (Default 'Off'). -J | --json-output Print test timing information in JSON format. (Default 'Off'). -h | --help This help. -p | --plain-output Print results in plain text. (Default 'Off'). -q | --quiet Turn off all output except fatal errors. (Default 'Off'). -j | --junit-output Print results in JUnit format. (Default 'Off'). -l | --list-tests Only list tests in bundle --additional-xctests Additional XCTest bundles to test. [ INFO ] Simulator 3 (PID 11181) has finished with exit code 1. [ INFO ] PID 11181 exited 1. [ INFO ] All simulators have finished.
Use xcrun simctl list runtimes
for a list of valid runtimes.When I use this operation, I get these valid runtimes:
== Runtimes == iOS 8.1 (8.1 - 12B411) (com.apple.CoreSimulator.SimRuntime.iOS-8-1) iOS 9.0 (9.0 - 13A344) (com.apple.CoreSimulator.SimRuntime.iOS-9-0) iOS 10.0 (10.0 - 14A345) (com.apple.CoreSimulator.SimRuntime.iOS-10-0) tvOS 10.0 (10.0 - 14T328) (com.apple.CoreSimulator.SimRuntime.tvOS-10-0) watchOS 3.0 (3.0 - 14S326) (com.apple.CoreSimulator.SimRuntime.watchOS-3-0)
i have these schemes
in someApp.xcodeproj/xcuserdata/Wil.xcuserdatad/xcschemes
:
someApp_Tests.xcscheme someApp_UITests.xcscheme someApp.xcscheme
and i have these .app
in 'Build/Products/Debug-iphonesimulator'
somApp_UITests-Runnera.pp///This folder contains the `PlugIns` folder someApp.app///This folder contains the `PlugIns` folder
I do not know what the problem, I can not run successfully, please help me solve it.thank you
This change: 93b7f31
broke command line parsing. It needs to be fixed.
In Bluepill-runner/Bluepill-runner/BPRunner.m
, the onInterrupt
interrupt handler is doing too much. It should just increment the variable, and the logic below the increment should be moved to the interrupt
method.
I managed to run bluepill against one of my apps with a bunch of UI tests based on Xcode UI testing. However I'm getting the following error. A bit googling suggests that this might be due to UI testing having different test template/config than "regular" XCTests. Is Xcode UI testing supported by bluepill? :)
Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'No target application path specified via test configuration: <XCTestConfiguration: 0x60800011a9d0>
Thanks!
Right now, bluepill
will exit with different exit codes depending on the overall success/failure of the tests. However it doesn't distinguish success from failure in the printed messages. It would be nice if after the last line:
[ INFO ] All simulators have finished.
it printed something along the lines of:
[ FAIL ] Tests failed
or
[ FAIL ] Failed to run tests: Could not create simulator
Or something like that.
We have some duplicated functions in SimulatorRunner, such as:
@ob We should definitely reduce the function count there. remove them and uplevel the call.
@vargon So rather than merging these duplicate methods as we previously discussed, it seems we should just locate them in different classes. The mixing of the static methods that can act on anything and the config-specific methods that know the current context is what's confusing. Confining them to separate files would be the way forward, I believe.
Running this:
bluepill -a XXXX.app/ -s XXXX.xcscheme -o output/
crashes and returns this:
bluepill[78419:3224648] *** Terminating app due to uncaught exception 'NSRangeException', reason: '*** -[__NSSingleObjectArrayI objectAtIndex:]: index 1 beyond bounds [0 .. 0]'
*** First throw call stack:
(
0 CoreFoundation 0x00007fff7d8390db __exceptionPreprocess + 171
1 libobjc.A.dylib 0x00007fff924cba2a objc_exception_throw + 48
2 CoreFoundation 0x00007fff7d8a3b4f -[__NSSingleObjectArrayI objectAtIndex:] + 111
3 bluepill 0x000000010bb80448 +[BPXCTestFile BPXCTestFileFromExecutable:withError:] + 872
4 bluepill 0x000000010bb7f746 +[BPApp BPAppWithAppBundlePath:withExtraTestBundles:withError:] + 907
5 bluepill 0x000000010bb78b3c main + 739
6 libdyld.dylib 0x00007fff92dad255 start + 1
7 ??? 0x0000000000000007 0x0 + 7
)
libc++abi.dylib: terminating with uncaught exception of type NSException
Abort trap: 6
After providing a valid path to my app, an output path, and a path to a scheme containing XCTest UI tests, the output looks like this:
XXX@XXXX-mbp15-01 ~/develop/XXXX (develop)*$ ./bluepill -a TestBuild/Build/Products/UITest-iphonesimulator/XXXX.app -o ~/Desktop/ -s XXXX.xcodeproj/xcshareddata/xcschemes/UITests.xcscheme
[ INFO ] This is Bluepill
[ ERROR ] Packing failed
After some light digging I can see where the error comes from:
https://github.com/linkedin/bluepill/blob/9d704360cdb87ecd9f4fc59fa8c1b1690c8a6d10/Bluepill-runner/Bluepill-runner/BPRunner.m#L140
but it's not obvious why it's failing. There's a few switches listed in ./bluepill -help
that are "required" but have no default value, and it's not obvious what they mean. For example:
-N<required_arg> | --no-split<required_arg>
A list of NO split test bundles
The code in the main loop and the completion block of task termination in the run() method of BPRunner.m are not not synchronized. Because the completion block will be called in other threads than the main thread, that causes race condition for variables such as launchedTasks, taskList.
I will fix it together will implementing the feature of reusing simulators. just create an issue to track it.
bluepill was the project that mentioned in below article?
https://engineering.linkedin.com/blog/2016/04/3x3--ios-build-speed-and-stability
In current setup, if test hangs, we don't know what is the cause.
It would be much nicer if we provide a process sample of the hanging app before killing it and record it in the report. Like the output of Sample
command.
I was testing it locally, and the only place it's used is in testEnvironmentWithConfiguration
which is only called from bluepill's unit tests.
Instead of traversing file system for the test targets, parse .xctestrun file to fetch them.
[self.device terminateApplicationWithID:self.hostBundleId error:&error];
in SimulatorMonitor.m doesn't work in iOS 10.2 runtime.
Hey guys, great project! I'm testing bluepill with a Titanium-based project, created a config.json like the following:
{
"app": "./mensa-app/build/Products/Debug-iphonesimulator/Mensa.app",
"scheme-path": "./mensa-app/Mensa.xcodeproj/xcshareddata/xcschemes/Mensa.xcscheme",
"output-dir": "./mensa-app/build"
}
but getting the error The folder “Plugins” doesn’t exist.
. Any idea what that could mean? Thx!
set up travis for bluepill to get build information
Title says it all. If you send SIGINT
to bluepill it'll abort the tests, but it'll report All Tests Passed
which is not true.
Placeholder for Xcode 8.3 support.
When I run bluepill on my (objc) app, I get
2017-02-18 00:30:04.236 bluepill[58215:969741] *** Terminating app due to uncaught exception 'NSRangeException', reason: '*** -[__NSArrayM objectAtIndex:]: index 2 beyond bounds [0 .. 1]'
*** First throw call stack:
(
0 CoreFoundation 0x00007fff9c443e7b __exceptionPreprocess + 171
1 libobjc.A.dylib 0x00007fffb102dcad objc_exception_throw + 48
2 CoreFoundation 0x00007fff9c35d1dc -[__NSArrayM objectAtIndex:] + 204
3 bluepill 0x000000010e4e0f9a +[BPXCTestFile BPXCTestFileFromExecutable:withError:] + 1138
4 bluepill 0x000000010e4e0285 +[BPApp BPAppWithAppBundlePath:onlyTestingBundlePath:withExtraTestBundles:withError:] + 1589
5 bluepill 0x000000010e4d92f5 main + 786
6 libdyld.dylib 0x00007fffb1911255 start + 1
7 ??? 0x0000000000000005 0x0 + 5
)
I confirmed this is problematic line.
Hi,
When I try to run my UI test then I get the below issue. Can you please help me to solve it?
ERROR: The folder “Plugins” doesn’t exist.
I am using appium for iOS automation on real device, Now I want to execute my test simultaneously on real devices. How can I use bluepill in my project. If bluepill doesn't work then what can I use ?
When the test-bundle-path argument is passed, the program immediately exits. When debugging, in BPConfiguration.m, test-bundle-path program is only 'BP_SLAVE' instead of 'BP_MASTER | BP_SLAVE' so in handleOpt() it exits because !(BPOptions[i].program & self.program) is true for test-bundle-path. I can make a PR for adding BP_MASTER but don't know what those are for so not sure if its intentional.
when I run bluepill, there was an Exception,here is logs...
CCTest:bluepill-v0.2.1 chen$ ./bluepill -c config.json
[ INFO ] This is Bluepill v0.1.0-33-g0d56983
[ INFO ] Running with 4 simulators.
[ INFO ] Packed tests into 6 bundles
[ INFO ] Started Simulator 1 (PID 11521).
[ INFO ] 1 Simulator still running. [1]
[ INFO ] Using 366 of 1064 processes.
[ INFO ] Started Simulator 2 (PID 11526).
2017-02-07 11:07:21.087 bp[11526:120290] -[__NSCFNumber length]: unrecognized selector sent to instance 0x7fbfc870d6f0
2017-02-07 11:07:21.087 bp[11521:120251] -[__NSCFNumber length]: unrecognized selector sent to instance 0x7f8b79d03380
2017-02-07 11:07:21.160 bp[11521:120251] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[__NSCFNumber length]: unrecognized selector sent to instance 0x7f8b79d03380'
*** First throw call stack:
(
0 CoreFoundation 0x00007fffaddfb48b __exceptionPreprocess + 171
1 libobjc.A.dylib 0x00007fffc255dcad objc_exception_throw + 48
2 CoreFoundation 0x00007fffade7cc94 -[NSObject(NSObject) doesNotRecognizeSelector:] + 132
3 CoreFoundation 0x00007fffadd6db55 forwarding + 1061
4 CoreFoundation 0x00007fffadd6d6a8 _CF_forwarding_prep_0 + 120
5 CoreFoundation 0x00007fffadd06c8d isEqualToString + 77
6 CoreFoundation 0x00007fffadd52cd8 -[NSTaggedPointerString isEqualToString:] + 24
7 bp 0x00000001089af491 -[BPConfiguration validateConfigWithError:] + 4208
8 bp 0x00000001089c8897 main + 632
9 libdyld.dylib 0x00007fffc2e3c255 start + 1
10 ??? 0x0000000000000003 0x0 + 3
)
libc++abi.dylib: terminating with uncaught exception of type NSException
What should I do, thanks!
Bluepill handles the case when a test crashes. However, if the app itself crashes on launch, the console gets a useless message like this:
[ CRASH ] (1) (null)/(null) crashed app.
And nothing useful is reported in the output report. Bluepill should have a special way of logging and reporting an app crashing on launch.
We need two new fields in BPConfiguration.m:BPOptions
, one to know whether this option applies to bp
or bluepill
or both
, and another to know whether the actual option is required or not. Right now we have the no_argument
, required
, and optional
part but that is for optarg
from getopt
.
Also, we need to stop printing the full usage on error since it's grown too big.
Maybe also create a man page... troff anyone?
I create a new project and run ./bluepill -c config.json
,There are some errors,what should I do?
mactekiiMac:bluepill-v0.2.2 mac$ ./bluepill -c config.json
[ INFO ] This is Bluepill v0.1.0-46-gd35f56c
[ WARNING] Lowering number of simulators from 4 to 2 because there aren't enough tests.
[ INFO ] Running with 4 simulators.
[ INFO ] Packed tests into 2 bundles
[ INFO ] Started Simulator 1 (PID 2498).
2017-02-25 09:01:18.995 bp[2498:47348] +[SimServiceContext sharedServiceContextForDeveloperDir:error:]: unrecognized selector sent to class 0x10774e8c8
2017-02-25 09:01:18.998 bp[2498:47348] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '+[SimServiceContext sharedServiceContextForDeveloperDir:error:]: unrecognized selector sent to class 0x10774e8c8'
*** First throw call stack:
(
0 CoreFoundation 0x00007fffd36957bb __exceptionPreprocess + 171
1 libobjc.A.dylib 0x00007fffe7e02a2a objc_exception_throw + 48
2 CoreFoundation 0x00007fffd3715cc4 +[NSObject(NSObject) doesNotRecognizeSelector:] + 132
3 CoreFoundation 0x00007fffd3608fc3 ___forwarding___ + 1059
4 CoreFoundation 0x00007fffd3608b18 _CF_forwarding_prep_0 + 120
5 bp 0x0000000106f9e780 -[BPConfiguration validateConfigWithError:] + 2895
6 bp 0x0000000106fb8573 main + 632
7 libdyld.dylib 0x00007fffe86de255 start + 1
)
libc++abi.dylib: terminating with uncaught exception of type NSException
[ INFO ] Simulator 1 (PID 2498) has finished with exit code 6.
[ INFO ] PID 2498 exited 6.
[ INFO ] 0 Simulators still running. []
[ INFO ] Using 264 of 1064 processes.
...
this is my config.json
{
"app": "/Users/mac/Library/Developer/Xcode/DerivedData/bluepill-ezyxkypxgoiujngjzpwyqpshbatu/Build/Products/Debug-iphonesimulator/bluepill.app",
"scheme-path": "/Users/mac/Documents/bluepill/bluepill.xcodeproj/xcuserdata/mac.xcuserdatad/xcschemes/bluepill.xcscheme",
"output-dir": "./build/a/",
"xcode-path": "/Applications/Xcode8.2.app",
"runtime": "10.2"
}
We have both BPUtils::printInfo and BPUtils::printError, which is confusing.
Let’s nix printError and make printInfo:ERROR go to stderr.
Quite mode shouldn't suppress error messages.
Hello,
I'm trying to run bluepill for iPhone SE on 9.3. I've done xcodebuild with the proper destination:
-destination 'platform=iOS Simulator,name=iPhone SE,OS=9.3'
Here's my config.json:
{
"app": "../build/to/App.app",
"scheme-path": "../build/to/test.xcscheme",
"output-dir": "../output",
"num-sims": 3,
"plain-output": true,
"failure-tolerance": 1,
"device": "iPhone SE",
"runtime": 9.3,
"headless": false,
"only-retry-failed": true
}
But I'm getting the following error:
2017-02-22 15:59:02.301 bp[31126:28986974] -[__NSCFNumber length]: unrecognized selector sent to instance 0x7f94b1c23c70
2017-02-22 15:59:02.301 bp[31126:28986974] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[__NSCFNumber length]: unrecognized selector sent to instance 0x7f94b1c23c70'
*** First throw call stack:
(
0 CoreFoundation 0x00007fff92c514f2 __exceptionPreprocess + 178
1 libobjc.A.dylib 0x00007fff99190f7e objc_exception_throw + 48
2 CoreFoundation 0x00007fff92cbb1ad -[NSObject(NSObject) doesNotRecognizeSelector:] + 205
3 CoreFoundation 0x00007fff92bc1571 ___forwarding___ + 1009
4 CoreFoundation 0x00007fff92bc10f8 _CF_forwarding_prep_0 + 120
5 CoreFoundation 0x00007fff92b5a241 isEqualToString + 65
6 CoreFoundation 0x00007fff92ba7228 -[NSTaggedPointerString isEqualToString:] + 24
7 bp 0x0000000108312ca1 -[BPConfiguration validateConfigWithError:] + 4208
8 bp 0x000000010832c573 main + 632
9 libdyld.dylib 0x00007fff935065ad start + 1
)
libc++abi.dylib: terminating with uncaught exception of type NSException
However, when I run tests manually via xcode or xcodebuild command, I do not get those errors. Am I doing something wrong?
I had a test that failed in a set where all others succeeded, and when it retried it crashed the app, but then Bluepill marked it as success. Here's the relevant portion of the output:
20170213.134303 [ INFO ] (2) Completed: [2] Install Application
20170213.134303 [ INFO ] (2) [2] Launch Application
20170213.134311 [ INFO ] (2) Completed: [2] Launch Application
20170213.134312 [ INFO ] (2) All Tests started.
20170213.134320 [ CRASH ] (2) AnnotationTests/testCreateNote crashed app.
20170213.134320 [ ERROR ] (2) Will kill the process with appPID: 7676
20170213.134320 [ INFO ] (2) [2] Delete Simulator
20170213.134332 [ INFO ] (2) Completed: [2] Delete Simulator
20170213.134332 [ INFO ] (2) Exit Status: BPExitStatusTestsAllPassed
Nothing else was retried after and Bluepill finished with success.
Currently bluepill recreate the simulator for each test bundle. We could save a lot of time if reusing the same simulator for more than one test bundles.
Proposals:
Add CLI option "not-delete-simulator" to bp command, so it won't delete the simulator when test cases finish normally, and return the device id to the parent bluepill process. By default bp command Always delete simulator.
Add CLI option "use-simulator" to bp command. The bluepill process pass the device id to bp command so bp can reuse the specified simulator without recreating a new one.
Upload a recipe fo bluepill to brew so that people don't have to build bluepill through source code.
when i run ./bluepill -a ./Sample.app -s ./SampleAppTestScheme.xcscheme -o ./output/
it replay me -bash: ./bluepill: is a directory
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.