See couchbase/sync_gateway#1068

Functional test stacks with /db/_all_docs. Could you please look into this?
http://qa.sc.couchbase.com/view/mobile/job/sync-gateway-mobile-functional-tests-android-cb/47/console

depends on #7.

It assumed that ansible script and our tests should use the same ini file

Maybe setup.py or something that will set up the environment locally as well as in Jenkins

For example, if using cblite 1.1.1, make sure server is returning Server: Couchbase Sync Gateway/1.1.1

• cen006-p0-mobile-cblite-android_cb
• cen006-p0-mobile-sync-gateway-tests-android_cb_bucket_shadowing
• cen006-p0-mobileupgrade-serverupgrade-sg-cbliteios
• cen006-p0-mobileupgrade-sgupgrade-cblite
• cen006-ps-sgw-vset00-00-sgw-service-install
• cen006-ps-sgw-vset00-00-sgw-web-hook
• centos_x64--100_01--mobile_functional_android
• centos_x64--100_01--mobile_functional_android_from_mobile
• centos_x64--100_02--mobile_unit_tests_android
• centos_x64--100_03--mobile_java
• centos_x64--100_04--sync-gateway-tests
• centos_x64--100_04--sync-gateway-tests-cb
• mac_x64--101_01--mobile_functional_mac_android
• mac_x64--102_01--mobile_functional_mac
• 008-ps-sgw-vset00-00-sgw-service-install
• w008-ps-sgw-vset00-00-sgw-web-hook

This repo nearly works against a cluster running on AWS, spawned via the create_and_instantiate_cluster.py script in perfcluster-aws, except for one thing: the scripts in this repo try login as root, but root login via SSH is not enabled on AWS for security reasons.

FIX: use the centos user instead.

Related jira ticket

Currently the jenkins job is using the old cblite-tests repo.

It should be updated to use this repo.

parameterize
3.1.0
3.1.1
4.0.0

• Rebalance In
• Rebalance Out
• Start node
• Stop node

Topologies

CB SG Role
4 (3.1.0) 1 (1.2) [RW]
4 (3.1.0) 4 (1.2) [RW, RW, RW, RW]
4 (3.1.0) 4 (1.2) [W, W, R, R]

4 (3.1.1) 1 (1.2) [RW]
4 (3.1.1) 4 (1.2) [RW, RW, RW, RW]
4 (3.1.1) 4 (1.2) [W, W, R, R]

4 (4.0.0) 1 (1.2) [RW]
4 (4.0.0) 4 (1.2) [RW, RW, RW, RW]
4 (4.0.0) 4 (1.2) [W, W, R, R]

Scenarios

All scenarios will be run against multiple topologies
All scenarios should exercise multiple feed types (possibly)
All scenarios should be run at multiple data scales (1K docs, 10K, 100K)
All scenarios should be run with multiple user scales (10 users, 1000 users, 10000 users, per test basis)

• Single user / single channels - Unique docs with unique keys, (add)
• Single user / single channels - Unique docs with unique keys, (update)
• Single user / muliple channels - Unique docs with unique keys, (add)
• Single user / multiple channels - Unique docs with unique keys, (update)
• Muliple users / single channel - Unique docs with unique keys, (add)
• Muliple users / single channel - Unique docs with unique keys, (update)
• Muliple users / multiple channels - Unique docs with unique keys, (add)
• Muliple users / multiple channels - Unique docs with unique keys, (update)
• Reshard of DCP - Continuous write to a vBucket range, kill sync_gateway responsible for writing those vBuckets - Unique docs with unique keys
• sgs: Adding / Removing nodes from cluster (validate post state change).
  Using cbgt rest api (shards to each node)
  Ensure data integrity
  Verification that all writers register with CBGT
  Bring a new writer online during continuous writes
  All sync_gateways writers go down during write for a period of time, bring them back up.
• Channel Index Bucket Failover - Continuous doc writes with unique keys. Take down a couchbase server instance and verify that the channel index is intact after failing over to another couchbase node.
• Remove / Add channels to documents
• Remove / Add channels to users
  REST API calls
  Access grant in sync function
  Important to test: Newly granted access, make sure backfill occurs
  Make sure backfill only happens once (no doc dupes, ...)
• Exercise changes feed types (normal, continuous, longpoll) - perhaps in each of the above test cases.
• Multiple databases with distributed index in same bucket
• Multiple databases with distributed index in different buckets, test failover to single node
  4 sgs: each db has separate index bucket
  sg online / offline
  drop index bucket
  if the channel index bucket goes, sg should go to offline state
• Different sync functions (any ideas?)
  Access calls (adding / removing users)
  Channel calls
  Role based access calls
  Exercising views - Access calls in sync function, uses a view, make sure access grants in sync function
• Adding non-winning revisions
  Create revs further back in the tree
  Pass criteria: different sequence, same rev id = winning rev id
• Hotspot key design, index handles the case where one is slammed while other , bunch of of revs on one doc
• View query scenarios?
  Access calls in sync function, uses a view, make sure access grants in sync function
• Multiple writers with different shard values.
• Flush channel index bucket should take sync_gateway offline
  Large number of docs (1,000,000). Flush index bucket to start from 0. Make sure doesn’t kill your sg writers

@ashvindersingh -- can you try it with this config

I did a smoke test locally -- results

TP from Ashvinder
possible further to split into smaller tasks

I think it's just a matter of changing this line:

npm test  > ${WORKSPACE}/npm_test_results.log

to use tee

• kill_sync_gateway.py -n "vm_1"
• start_sync_gateway_with_config.py -n "vm_1" -c "sample_config.json"
• kill_couchbase_server.py -n "vm_2"
• start_couchbase_server.py -n "vm_1"

https://issues.couchbase.com/browse/MB-15171

Instead of Create / Delete bucket, perhaps use containers

See couchbase/sync_gateway#1218

At end of test:

• Hit :4985/_cbgt/api/cfg endpoint to get CBGT config json
• Check the number of active nodes
• Make sure it matches the number or sync gateway writers
• If does not match expected, test should fail and :4985/_cbgt/api/diag diagnostic information should be collected.

• Get running on iOS
• Get running on Android

TP from me is under dev review
possible further to split into smaller tasks

As mentioned in couchbase/sync_gateway#1257, it's possible to induce and detect data races.

TODO:

• Build sync gateway with the -race flag
• Scan the logs for "Data race" and fail the test if it's found

• execute any ssh command on remote vms
• start/stop node
• wait_for any CB task completed( rebalance, warmup)
• read CB/nodes statuses

....add more

Provision and launch default configuration but allow relaunching with a new configurations

sgs[0].restart("path-to-config.json")

Here's my temp_ansible_hosts file:

[root@87ff5502835a distributed_index_tests]# cat temp_ansible_hosts
cb1 ansible_ssh_host=ec2-54-162-126-196.compute-1.amazonaws.com
sg1 ansible_ssh_host=ec2-23-22-62-250.compute-1.amazonaws.com
sg2 ansible_ssh_host=ec2-54-211-175-86.compute-1.amazonaws.com

[couchbase_servers]
cb1

[sync_gateways]
sg1
sg2

(I generated by writing a /tmp/hosts.ini file, then ran the script to convert to ansible format)

My conf/hosts.ini is the default checked into github.

[root@87ff5502835a distributed_index_tests]# cat conf/hosts.ini
[vms]
vm1=172.23.105.164
vm2=172.23.105.165
vm3=172.23.122.250
vm4=172.23.122.251
vm5=172.23.122.252
vm6=172.23.122.253

[couchbase_servers]
cb1=vm1
cb2=vm2

[sync_gateways]
sg1=vm3
sg2=vm4
sg3=vm5
sg4=vm6

When I run the tests via:

python -m pytest --capture=no tests/test_single_user_multiple_channels.py

It's trying to contact the vm 172.23.122.250, which is in conf/hosts.ini but not temp_ansible_hosts.

Full test output

hideki wrote:
I added forestdb storage engine to CBL Android. Repo of ForestDB module for CBL Android is https://github.com/couchbaselabs/couchbase-lite-java-forestdb .
Now couchbase-life-android depends on this module. And It is compile native library and java codes. So build requires NDK (https://developer.android.com/ndk/index.html) to compile.

steps:

1. git clone [email protected]:couchbase/couchbase-lite-android-liteserv.git
2. cd couchbase-lite-android-liteserv/
3. git submodule init
4. git submodule update --init --recursive

$ export NDK_HOME=/home/andrei/Android/android-ndk-r10e/
$ export ANDROID_HOME=/home/andrei/Android/Sdk/
$ ./run_android_liteserv.sh 8081

FAILURE: Build failed with an exception.

* Where:
Build file '/home/andrei/couchbase_src/couchbase-lite-android-liteserv/libraries/couchbase-lite-android/build.gradle' line: 64

* What went wrong:
A problem occurred evaluating project ':libraries:couchbase-lite-android'.
> Project with path ':libraries:couchbase-lite-java-forestdb' could not be found in project ':libraries:couchbase-lite-android'.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 5.543 secs
* daemon not running. starting it now on port 5037 *
* daemon started successfully *
Starting: Intent { act=android.intent.action.MAIN pkg=password cmp=com.couchbase.liteservandroid/.MainActivity (has extras) }
Error type 3
Error: Activity class {com.couchbase.liteservandroid/com.couchbase.liteservandroid.MainActivity} does not exist.
andrei@abv:~/couchbase_src/couchbase-lite-android-liteserv$ ls libraries/couchbase-lite-android/libraries/couchbase-lite-java-forestdb/
build-android.gradle  build.gradle  build-java.gradle  gradle  gradlew  gradlew.bat  jni  libraries  Makefile  proguard-rules.pro  README.md  settings.gradle  src  vendor

the same error I see when build in Android studio

@hideki , could you help with the issue?

Currently, the test continues execution in a bad state. If the reset fails, fail the test.

This functionality is needed to change config params within a test case.
for example, api, sg.start(sg_ipaddr, custom_config.json)

Penns-MacBook-Pro:sync-gateway-tests penn$ node perf/run.js

cleanup

http server started on: http://127.0.0.1:8189

create internal stores

/Users/penn/Desktop/sync-gateway-tests/tmp/pdb-8190
/Users/penn/Desktop/sync-gateway-tests/lib/pouch.js:16
client = pouchdb.listen(port)
^

TypeError: pouchdb.listen is not a function
at Object. (/Users/penn/Desktop/sync-gateway-tests/lib/pouch.js:16:18)
at Module._compile (module.js:435:26)
at Object.Module._extensions..js (module.js:442:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:311:12)
at Function.Module.runMain (module.js:467:10)
at startup (node.js:136:18)
at node.js:963:3

and generate summary of error,warnings.

The following scripts need to be retested:

cbl-replication-mismatch-restart-gateway.js
cbl-replication-restart-gateway.js
cbl-restart.js
sg_changes_feed_last_seq.js

The class should also provide reload load balancer.
Verify the load balancer is running.
For our tests we plan to nginx as LB.

node perf/run.js with error:

"trace": [
{
"column": 10,
"file": "/Users/garychen/Desktop/sync-gateway-tests-master/tests/common.js",
"function": "",
"line": 157,
"method": null,
"native": false
},
{
"column": 16,
"file": "events.js",
"function": "ChildProcess.g",
"line": 260,
"method": "g",
"native": false
},
{
"column": 13,
"file": "events.js",
"function": "emitNone",
"line": 67,
"method": null,
"native": false
},
{
"column": 7,
"file": "events.js",
"function": "ChildProcess.emit",
"line": 166,
"method": "emit",
"native": false
},
{
"column": 18,
"file": "/Users/garychen/Desktop/sync-gateway-tests-master/lib/launcher.js",
"function": "null._onTimeout",
"line": 35,
"method": "_onTimeout",
"native": false
},
{
"column": 15,
"file": "timers.js",
"function": "Timer.listOnTimeout",
"line": 92,
"method": "listOnTimeout",
"native": false
}
]

Should the code be below in lines 156,157 and 160,161 of "tests/common.js"?
if(t)
t.false(err, "no error, Sync Gateway running on our port");