Migrate existing gRPC ObjC unit test suite to bazel-based test

• https://github.com/grpc/grpc/tree/master/src/objective-c/tests/UnitTests

Current Test Setup

Current test utilize a Cocoapod project target for unit test, and integrated with kokoro CI via script

• https://github.com/grpc/grpc/blob/master/src/objective-c/tests/run_one_test.sh

The script involves two major stage: pod installation and xcodebuild test run.

Its corresponding kokoro test components at

• https://github.com/grpc/grpc/blob/1cd6e69347cbf62a012477fe184ee6fa8f25d32c/tools/run_tests/run_tests.py#L1069-L1074

New Setup

We will define new ios_unit_test target for Unit Test and may utilize the following script for bazel integration w/ kokoro

• https://github.com/grpc/grpc/blob/master/src/objective-c/tests/run_one_test_bazel.sh

Also we should enable GRPC_VERBOSITY=debug environment variable when running these tests

cc @jtattermusch

Create and/or migrate SPM build test to verify SPM iOS build. Existing build test from gRPC located at

• https://github.com/grpc/grpc/tree/master/test/spm_build

The build test should be triggered for the following branch and scenarios

• grpc_master for continuous build integration check with c-core changes
• grpc_release for release build verification
• PR pre-submit

@HannahShiSFB

• Migrate gRPC-ProtoRPC.podspec
• Release pod from gRPC-iOS repo

Note that all upstream pod specs should be pointing to local shallow copy, or a mgirated pod that has been migrated or released from gRPC-iOS.

General clean up and refactoring of existing Interop test suite 's structure

• Avoid test suite subclassing anti-pattern in favor of delegation & component configuration

Two alternative ways of uploading our test results to BQ

• run_test.py with --bq_result_table (link)
• via bazel post-build script upload_rbe_results.py

@HannahShiSFB @jtattermusch

At the moment, Libuv-gRPC is pined to version 1.37. Latest libuv 1.42 is broken on iOS build.

For verifying libuv timer's usability for iOS, add unit test coverage to EventEngineTimerTests test suite. This can closely mirror existing c-core's test logic at

• https://github.com/grpc/grpc/blob/master/test/core/event_engine/test_suite/timer_test.cc

Note that in order to test libuv timer, it requires the libuv-timer impl PR to land and instantiation of grpc_event_engine::experimental::LibuvEventEngine as mentioned in libuv_event_engine_test.cc

Currently a few of our cocoapod doesn't have a default landing page on cocoapod trunk. Visiting the trunk page will automatically forward to homepage URL

• https://cocoapods.org/pods/Libuv-gRPC
• https://cocoapods.org/pods/BoringSSL-GRPC

Investigate adding a landing page for our pods

Umbrella issue for tracking and grouping PRs updating V2.x APIs

Create gRPC-iOS release branching and release tags

• Create and use target branch grpc_release_vX.YZ
• Script to pull in corresponding C-core release tag and checkin to native shallow copy source.
• Workflow bot to automate the process with C-core release tag trigger (if possible)

as mentioned in grpc/grpc#28570, patch libuv timer test with proper setup/tear down logic for static variables so that the test behavior doesn't break across platforms.

• Migrate gRPC.podspec to gRPC-iOS.
• Release pod from gRPC-iOS repo

Note that all upstream pod specs should be pointing to local shallow copy, or a mgirated pod that has been migrated or released from gRPC-iOS.

Investigate build timeout and adjust. Failure report samples

• https://source.cloud.google.com/results/invocations/9cbdaf5e-0224-4b75-863e-d1f582782cb6/targets
• https://source.cloud.google.com/results/invocations/30e3e6ca-ead3-45ab-a93f-6b41866c7424/targets
• https://source.cloud.google.com/results/invocations/30e3e6ca-ead3-45ab-a93f-6b41866c7424/targets

Migrate and/or create build test coverage for pods managed by iOS/ObjC stack, these includes

• Objc Wrapper Pod (gRPC, gRPC-ProtoRPC, gRPC-RxLibrary)
• Native C-Core Pod (gRPC-C++, gRPC-Core)

The build test should be triggered in the following brach and scenarios

• grpc_master branch upon shallow copy source code sync (daily)
• grpc_release branch upon release tag creation (per grpc schedule)
• PR presubmit (in the future)

Add goto doc for how to build gRPC native c-core for iOS.

Switching pod release from grpc/grpc to grpc/grpc-ios for the following c-core Pod

• gRPC-C++.podspec
• gRPC-Core.podspec

gRPC-C++ and gRPC-Core pod should still be present in grpc/grpc as that correctly reflects the dependency structure .

Our CFStream test suite currently have a few issues that need to be improved

• Test hangs and keep retries until timeout if test failure
• Async test logic is using NSRunLoop mainRunLoop instead of XCTestExpectation

In addition, the current test suite is an objc translation from existing c-core test

• https://github.com/grpc/grpc/blob/master/test/core/iomgr/BUILD#L241

We should revisit the test setup to run c-core's test setup using the corresponding ios unit test target instead

• https://github.com/grpc/grpc/blob/master/bazel/grpc_build_system.bzl#L335

Additional todos for CFStream Test Update

• Migrate CFStream test to bazel based
• Move CFStream test out of test/core/iomgr and move into src/objective-c so that it is co-located with the rest of the ios/objc test.

Sample failure log

• https://source.cloud.google.com/results/invocations/57c6d119-bbe7-406c-94fe-82b0e6c3c559/targets/grpc%2Fcore%2Fmaster%2Fmacos%2Fgrpc_bazel_cpp_ios_tests/log

First introduced via PR

• grpc/grpc#27748

Crash on test case DuplicateHeadersAreMerged with the following error

Test Case '-[GTMGoogleTestRunner RlsEnd2endTest::DuplicateHeadersAreMerged]' started.
E1025 11:55:03.334006000 4675941824 lb_policy_registry.cc:45]          assertion failed: strcmp(factories_[i]->name(), factory->name()) != 0
Child process terminated with signal 6: Abort trap
2021-10-25 18:55:03,337 Deleting simulator B98849B0-9930-4F52-9042-26010265B1F5 asynchronously.
2021-10-25 18:55:03,343 Done.
================================================================================
Target //test/cpp/end2end:rls_end2end_test_on_ios up-to-date:
  bazel-bin/test/cpp/end2end/rls_end2end_test_on_ios
  bazel-out/applebin_ios-ios_x86_64-fastbuild-ST-e637a0adc749/bin/test/cpp/end2end/rls_end2end_test_on_ios.zip
INFO: Elapsed time: 17.652s, Critical Path: 16.50s
INFO: 55 processes: 31 internal, 23 darwin-sandbox, 1 local.
INFO: Build completed, 1 test FAILED, 55 total actions
//test/cpp/end2end:rls_end2end_test_on_ios                               FAILED in 2.3s
    ERROR   .test/cpp/end2end/rls_end2end_test_on_ios (0.0s)

Cocoapod Targets

• PerfTests
• TvTests
• MacTests

Also we should enable GRPC_VERBOSITY=debug environment variable when running these tests

cc @jtattermusch

Currently as 1st step in libuv PR , we hard-coded libuv source in podspec. Ideally these are pulled from libuv.BUILD rules and tunneled through its template files and auto-populated (alongside build flags) via buildgen script. This reduce maintenance overhead.

https://source.cloud.google.com/results/invocations/39899e9f-d958-42de-b0c3-8c2b9427ca49/targets/grpc%2Fcore%2Fmaster%2Fmacos%2Fgrpc_basictests_objc_ios/log

Fri Sep 17 00:56:37 PDT 2021 - Test Case '-[GRPCClientTests testTimeoutBackoff2]' started.
Fri Sep 17 00:56:37 PDT 2021 - 2021-09-17 00:56:37.576055-0700 xctest[41153:78530] [] nw_socket_handle_socket_event [C22:1] Socket SO_ERROR [61: Connection refused]
Fri Sep 17 00:56:37 PDT 2021 - 2021-09-17 00:56:37.577751-0700 xctest[41153:78530] [] nw_connection_get_connected_socket [C22] Client called nw_connection_get_connected_socket on unconnected nw_connection
Fri Sep 17 00:56:37 PDT 2021 - 2021-09-17 00:56:37.582215-0700 xctest[41153:78530] TCP Conn 0x7fd4a45aaa70 Failed : error 0:61 [61]
Fri Sep 17 00:56:37 PDT 2021 - 2021-09-17 00:56:37.595044-0700 xctest[41153:78530] [] nw_socket_handle_socket_event [C23:1] Socket SO_ERROR [61: Connection refused]
Fri Sep 17 00:56:37 PDT 2021 - 2021-09-17 00:56:37.598824-0700 xctest[41153:78732] [] nw_connection_get_connected_socket [C23] Client called nw_connection_get_connected_socket on unconnected nw_connection
Fri Sep 17 00:56:37 PDT 2021 - 2021-09-17 00:56:37.603653-0700 xctest[41153:78732] TCP Conn 0x7fd4a45adee0 Failed : error 0:61 [61]
Fri Sep 17 00:56:38 PDT 2021 - /Volumes/BuildData/tmpfs/src/github/grpc/workspace_objc_macos_opt_native/src/objective-c/tests/UnitTests/GRPCClientTests.m:589: error: -[GRPCClientTests testTimeoutBackoff2] : (([[NSDate date] timeIntervalSinceDate:startTime]) less than (maxConnectTime + kMargin)) failed: ("0.843600988388") is not less than ("0.8")
Fri Sep 17 00:56:38 PDT 2021 - Test Case '-[GRPCClientTests testTimeoutBackoff2]' failed (0.847 seconds).

Create SPM ObjC target for the following corresponding pod target

• gRPC-ProtoRPC for ObjC V2 API and dependencies
• gRPC for ObjC V1 legacy API and dependencies

These targets should be under top level Package.swift and upstream dependencies should be referencing grpc native c-core's shallow content instead of pulling from gRPC c-core repo.

• Migrate gRPC-RxLibrary.podspec
• Release from gRPC-iOS

Example issue seem here

• grpc/grpc#27831

Current working assumption is bazel's ios_unit_test using GMT test runner. reloads test dylib per test run and triggers gtest's test suite setup to execute per test.

At the moment, post pod installation, all gRPC/ObjcC pod is using prepare_command to rewrite c++ core / external lib's include headers in order to accomodate iOS/Xcode search path. This creates various issues and very error prone.

Instead we should look at avoid modifying the source and adjust build header search path at Cocoapoad/Xcode configuration level.

Migrate existing Cocoapod based Interop test suite to bazel

• https://github.com/grpc/grpc/tree/master/src/objective-c/tests/InteropTests

Existing pod test target defined in

• https://github.com/grpc/grpc/blob/2badafbc4d246d5165546a61eeb36aa017987d30/src/objective-c/tests/Podfile#L36

Its corresponding kokoro run component at

• https://github.com/grpc/grpc/blob/1cd6e69347cbf62a012477fe184ee6fa8f25d32c/tools/run_tests/run_tests.py#L1075-L1080

Also we should enable GRPC_VERBOSITY=debug environment variable when running these tests

cc @jtattermusch

At the moment, generated_projects.sh will automatically add new source file to the main spec, however, additional podspec imports may still be needed to avoid linkage-time symbol missing issue. See PR example here: grpc/grpc#27032

Check if there is a way to automatically detect missing imports or surface up the missing steps to devs.

Setup bazel ios_unit_test and test against C-core's existing event engine test suite

Existing c-core unit test on iOS is done via the following bazel rule

• ios_cc_test
  • this is added for each grpc_cc_test target

on iOS/Mac, we run the following script to trigger bazel test on kokoro CI

• grpc_run_bazel_cpp_ios_test.sh

This is test against all ios_unit_test targets under //test/cpp. We can setup a similar structure for event engine test target

• https://github.com/grpc/grpc/blob/master/test/core/event_engine/BUILD#L21

Sample instruction to run bazel ios test in command line

tools/bazel query "kind(ios_unit_test, tests(//test/core/event_engine/...))" 

tools/bazel test --test_summary=detailed --test_output=all //test/core/event_engine:endpoint_config_test_on_ios 

xxhash Cyan4973/xxHash#601 have fixed Wdocumentation error during grpc ios compile Cyan4973/xxHash#590

xxhash.h patch no longer needed

observing high frequent cronet test time out on both pre-submit and prod/master's grpc_basictests_objc_ios test suite. One of the following sample log captures the issue from cronet test suite crash due to python port server not running. XCTest attempts to restart the test and eventually leads to timeout being reported

• https://source.cloud.google.com/results/invocations/e2291cf0-dd86-4e84-8129-15534a947027/targets/github%2Fgrpc%2Frun_tests%2Fobjc_macos_opt_native%2Fios-test-cronettests/tests

Mon Nov  1 11:37:54 PDT 2021 - Test Case '-[CoreCronetEnd2EndTests testCancelAfterClientDone]' started.
Mon Nov  1 11:38:04 PDT 2021 - gRPC tests require a helper port server to allocate ports used
Mon Nov  1 11:38:04 PDT 2021 - during the test.
Mon Nov  1 11:38:04 PDT 2021 -
Mon Nov  1 11:38:04 PDT 2021 - This server is not currently running.
Mon Nov  1 11:38:04 PDT 2021 -
Mon Nov  1 11:38:04 PDT 2021 - To start it, run tools/run_tests/start_port_server.py
Mon Nov  1 11:38:04 PDT 2021 -
Mon Nov  1 11:38:25 PDT 2021 -
Mon Nov  1 11:38:25 PDT 2021 - Restarting after unexpected exit or crash in CoreCronetEnd2EndTests/testCancelAfterClientDone; summary will include totals from previous launches.
Mon Nov  1 11:38:25 PDT 2021 -

most sample test log doesn't capture the above crash, but instead reporting time out with the last test suite being ios cronet test,

• sample: https://source.cloud.google.com/results/invocations/ae61b11e-fcf9-4492-be80-82c981103d24/targets/grpc%2Fcore%2Fpull_request%2Fmacos%2Fgrpc_basictests_objc_ios/log

2021-11-01 14:50:10,768 START: ios-test-cronettests

2021-11-01 15:16:45,843 TIMEOUT: run_tests_objc_macos_opt_native [pid=3790, time=5402.6sec]
2021-11-01 15:16:45,856 FAILED: Some run_tests.py instances have failed.

This is followed by another script check agains the port server existence w/ exit failure

+ FAILED=true
+ ps aux
+ grep 'port_server\.py'
+ awk '{print $2}'
+ xargs kill -9
...
real	0m13.275s
user	0m0.359s
sys	0m8.198s
+ '[' true '!=' '' ']'
+ exit 1

Automate Cocoapod release with the following steps

• Scripts to simplify building and release pod for a target podspec and from a release tag
• Workflow bot that automate the pod release

This should also coordinates w/ gRPC-iOS cocoapod migration so that

• We release from grpc-ios/grpc_release branch and corresponding release tags
• Release using podspecs from gRPC-iOS repo (Objc pods) and native shallow copy for native pods (gRPC-C++, gRPC-Core)

Add libuv-based unit test on iOS core test for DNS resolver. This make sure the API that core uses can work. Unit test logic can be sampled from a core subset from

• https://github.com/libuv/libuv/blob/v1.x/test/test-getaddrinfo.c
• https://github.com/libuv/libuv/blob/v1.x/test/test-gethostname.c
• https://github.com/libuv/libuv/blob/v1.x/test/test-getnameinfo.c
• https://github.com/libuv/libuv/blob/v1.x/test/test-getsockname.c

Libuv's DNS utility API

• http://docs.libuv.org/en/v1.x/dns.html

Currently our iOS binary size diff script (binary_size.py) builds sample app via build_one_example.sh script which is cocoapod & xcodebuild-based. Switching to bazel based build (build_one_example_bazel.sh) may help significantly speed up the build time.

As discussed in grpc/grpc#29377

Sample

• https://source.cloud.google.com/results/invocations/6a805d1a-ec38-4ae5-af4b-2772d910966a/targets/grpc%2Fcore%2Fmaster%2Fmacos%2Fgrpc_bazel_cpp_ios_tests/log

Update bazel-based gRPC ObjC test to support generating structured test logs that can be rendered by kokoro UI. Specifically, look into existing c++'s bazel test scripts examples

• https://github.com/grpc/grpc/blob/master/tools/internal_ci/macos/grpc_run_bazel_c_cpp_tests.sh
• https://github.com/grpc/grpc/blob/master/tools/internal_ci/macos/grpc_run_bazel_cpp_ios_tests.sh

Prior PR for structured test results at

• grpc/grpc#29353

@HannahShiSFB @jtattermusch

On GRPCCallOptions, the following properties are currently not copied over

• @Property(atomic, assign) NSTimeInterval maxRetryInterval;
• @Property(atomic, assign) NSTimeInterval minRetryInterval;
• @Property(atomic, assign) NSUInteger retryCount;
• @Property(atomic, assign) double retryFactor;

In addition, these are currently marked readwrite on immutable option class GRPCCallOptions, and should be moved to GRPCMutableCallOptions instead.

Looks like we maintained a local copy of Protobuf-C++ podspec in grpc repo, which can easily get out of sync with the official one

• switch to using protobuf team's official podspec and pull directly from pod trunk
• clean up local copy

sample build break

• grpc/grpc#27722

This also requires importing absl:random headers on iOS

To make iOS bazel test behaves similar to other language
e.g. grpc/grpc#28570

similar iOS test behavior diffs issue
#41

Follow up to grpc/grpc#28029 for adding internal kokoro run job

Umbrella task tracking Swift Package Manager implementation support for grpc-ios

As seem in grpc/grpc#27662, it looks like current suite of kokoro test didn't catch compilation errors as seem locally on dev machine (Xcode 13, clang 5.5, Mac OSX 11.6)

Observed a number of test breakage on cpp cronet test targets

• grpc/grpc#27740
  • https://source.cloud.google.com/results/invocations/fc8b27a6-91c7-46b2-9f38-9b377525cf0a/targets/grpc%2Fcore%2Fpull_request%2Fmacos%2Fgrpc_basictests_cpp_ios/log
• grpc/grpc#27722
  • https://source.cloud.google.com/results/invocations/879ee0d0-4466-4a14-9e08-b70d3aa47fe1/targets/github%2Fgrpc%2Frun_tests%2Fobjc_macos_opt_native%2Fios-cpp-test-cronet/tests.

Looks like all test cases crashed with the following error messages

Wed Oct 20 09:56:47 PDT 2021 - 2021-10-20 09:56:47.779436-0700 xctest[31851:63940] DiskCookieStorage changing policy from 2 to 0, cookie file: file:///Users/kbuilder/Library/Developer/CoreSimulator/Devices/4F9376BD-4079-49A7-BB18-AED93CEEBE0B/data/Library/Cookies/com.apple.dt.xctest.tool.binarycookies
Wed Oct 20 09:56:47 PDT 2021 - xctest(31851,0x11151f5c0) malloc: *** error for object 0x2000000000000000: pointer being freed was not allocated
Wed Oct 20 09:56:47 PDT 2021 - xctest(31851,0x11151f5c0) malloc: *** set a breakpoint in malloc_error_break to debug
Wed Oct 20 09:58:30 PDT 2021 -
Wed Oct 20 09:58:30 PDT 2021 - Restarting after unexpected exit or crash in CppCronetEnd2EndTests/testBidiStream; summary will include totals from previous launches.

A few errors message related to cronet failure

ed Oct 20 09:56:47 PDT 2021 - [1020/095647.771493:ERROR:cronet_environment.mm(157)] Can not start NetLog to /Users/kbuilder/Library/Developer/CoreSimulator/Devices/4F9376BD-4079-49A7-BB18-AED93CEEBE0B/data/Documents/cronet_netlog.json: Undefined error: 0

Moving all GRPCCallOption & GRPCMutableCallOptions properties to nonatomic since these are basic data model types and atomic access does not add much value to it.

Clean up and removal of GRPCCallSafety enum after idempotent/cacheable options are removed after grpc/grpc#28922.

See samples from run

https://source.cloud.google.com/results/invocations/41df2ac0-fbb6-48db-a03c-a3c82e02af45/targets/github%2Fgrpc%2Frun_tests%2Fobjc_macos_opt_native%2Fios-test-cronettests/tests

The following ObjC test failed

• CoreCronetEnd2EndTests::testCompressedPayload which invokes c-core's following test
  • compressed_payload (test_invoke_request_with_disabled_algorithm)

With assert at

https://github.com/grpc/grpc/blob/409c732813fdee5dadd2a2a14fd5972751443cb1/test/core/end2end/tests/compressed_payload.cc#L209

See the following sample

• https://source.cloud.google.com/results/invocations/ef4b1ae9-e83f-4b75-8707-ed36be9de786/targets/github%2Fgrpc%2Frun_tests%2Fobjc_macos_opt_native%2Fios-test-unittests/tests
• • https://source.cloud.google.com/results/invocations/c4d9082e-0eb7-47e8-ae9d-5561d03cb8d2/targets/grpc%2Fcore%2Fmaster%2Fmacos%2Fgrpc_basictests_objc_ios/log

Wed Sep 15 21:00:52 PDT 2021 - Test Case '-[CallAPIv2Tests testMetadata]' started.
Wed Sep 15 21:01:01 PDT 2021 - /Volumes/BuildData/tmpfs/src/github/grpc/workspace_objc_macos_opt_native/src/objective-c/tests/UnitTests/APIv2Tests.m:205: error: -[CallAPIv2Tests testMetadata] : Asynchronous wait failed: Exceeded timeout of 8 seconds, with unfulfilled expectations: "RPC unauthorized.".
Wed Sep 15 21:01:01 PDT 2021 - Test Case '-[CallAPIv2Tests testMetadata]' failed (8.465 seconds).

In addition, testErrorDebugInformation raises NSInternalInconsistencyException as follows

Wed Sep 15 21:01:15 PDT 2021 - 2021-09-15 21:01:15.822864-0700 xctest[41458:80151] *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Unable to report test assertion failure '((error.code) equal to (16)) failed: throwing "Unable to report test assertion failure '((error.code) equal to (16)) failed: ("14") is not equal to ("16") - Finished with unexpected error: Error Domain=io.grpc Code=14 "{"created":"@1631764875.804842000","description":"Error received from peer ipv6:[2607:f8b0:4002:c11::451]:443","file":"/Volumes/BuildData/tmpfs/src/github/grpc/workspace_objc_macos_opt_native/src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"502:Bad Gateway","grpc_status":14}" UserInfo={io.grpc.HeadersKey={

Also looks like we have

Look at how we can have better reuse of existing test logic from c-core. In Event Engine test, for example, one possibility is to expose its test util via test-only public headers and call from iOS test. Some of these might not be possible as they are based on gtest.

An existing similar example is our current Cronet's end2end test setup

As seem in sample below

• https://source.cloud.google.com/results/invocations/ac1c75bc-c978-4495-aeb7-8021842c75f1/targets/grpc%2Fcore%2Fmaster%2Fmacos%2Fgrpc_bazel_cpp_ios_tests/log

Also can be repro by manually triggering it with running the test script

grpc/tools/internal_ci/macos/grpc_run_bazel_cpp_ios_tests.sh

or

tools/bazel test --test_summary=detailed --test_output=all //test/cpp/server:authorization_policy_provider_test_on_ios

Currently if we modify gRPC-Core's BoringSSL & Libuv version number w/o publishing the pod for the new version, our iOS sample app project will break since the next pod install won't be able to find the updated version from the pod trunk (see example grpc/grpc#27710). A few action items

• we need to add presubmit build test to kokoro that builds an iOS app pod using the updated version from cocoapod trunk. This helps detect build breaks
• Update iOS release process so that this version bump are in sync w/ cocoapod trunk udpate