couchbaselabs / gocaves Goto Github PK

View Code? Open in Web Editor NEW

1.0 1.0 8.0 1.74 MB

License: Apache License 2.0

Makefile 0.03% Go 93.69% Vue 4.50% JavaScript 1.77%

gocaves's People

Contributors

Stargazers

Watchers

Forkers

chvck avsej jrascagneres jrawsthorne charlie-hayes matt-woz l0n3star

gocaves's Issues

Subdoc returns success on subdoc lookup with non-json doc

  @Test
  void notJson() {
    singleGetOpCheckExpectedFailure("I am not json!", "no_exist", DocumentNotJsonException.class);
  }

It looks like caves is returning packet-level success if the actual doc is not valid JSON.

subdocument mutations sets error index incorrectly

it returns correct error path_not_found/192, but with wrong index 0 ("dictkey")

GOCAVES 20:19:15.605343 received kv packet 0xc000190320 CMD:CMD_SUBDOCMULTIMUTATION
GOCAVES 20:19:15.622154 sending kv packet 0xc000190320 CMD:CMD_SUBDOCMULTIMUTATION &{Magic:129 Command:209 Datatype:0 Status:204 Vbucket:0 Opaque:10 Cas:0 CollectionID:0 Key:[] Extras:[] Value:[0 0 192] BarrierFrame:<nil> DurabilityLevelFrame:<nil> DurabilityTimeoutFrame:<nil> StreamIDFrame:<nil> OpenTracingFrame:<nil> ServerDurationFrame:0xc0006c0530 UnsupportedFrames:[]}

TEST_CASE("integration: subdoc multi mutation", "[integration]")
{
    test::utils::integration_test_guard integration;
    test::utils::open_bucket(integration.cluster, integration.ctx.bucket);
    couchbase::core::document_id id{ integration.ctx.bucket, "_default", "_default", test::utils::uniq_id("subdoc") };

    {
        auto value_json = couchbase::core::utils::to_binary(R"({"dictkey":"dictval","array":[1,2,3,4,[10,20,30,[100,200,300]]],"counter":0})");
        couchbase::core::operations::insert_request req{ id, value_json };
        auto resp = test::utils::execute(integration.cluster, req);
        REQUIRE_SUCCESS(resp.ctx.ec());
    }

    SECTION("replace with errors")
    {
        couchbase::core::operations::mutate_in_request req{ id };
        req.specs =
          couchbase::mutate_in_specs{
              couchbase::mutate_in_specs::replace("dictkey", tao::json::null),
              couchbase::mutate_in_specs::replace("dested.nonexist", tao::json::null),
              couchbase::mutate_in_specs::replace("bad..bad", tao::json::null),
          }
            .specs();
        auto resp = test::utils::execute(integration.cluster, req);
        REQUIRE(resp.ctx.ec() == couchbase::errc::key_value::path_not_found);
        REQUIRE(resp.fields.size() == 3);
        REQUIRE(resp.ctx.first_error_index() == 1);
        REQUIRE(resp.fields[1].status == couchbase::key_value_status_code::subdoc_path_not_found);
    }
}

Caves should not send mutation tokens if the client does not support them

gocaves/mock/mockimpl/svcimpls/kvimplcrud.go

Lines 321 to 327 in 05e3ba4

 // TODO(brett19): Implement feature checking for mutation tokens. 

 if true { 

 mtBuf := make([]byte, 16) 

 binary.BigEndian.PutUint64(mtBuf[0:], resp.VbUUID) 

 binary.BigEndian.PutUint64(mtBuf[8:], resp.SeqNo) 

 extrasBuf = append(extrasBuf, mtBuf...) 

 }

caves does not return error if value too large

A java kv integration test fails:

  @Test
  void throwsIfTooLarge() {
    String id = UUID.randomUUID().toString();
    JsonObject content = JsonObject.create();
    for (int i = 0; i < 400000; i++) {
      content.put(UUID.randomUUID().toString(), UUID.randomUUID().toString());
    }
    assertThrows(ValueTooLargeException.class, () -> collection.upsert(id, content));
  }

No exception is thrown, it looks like caves does not return an error if the value is over the max size limit?

Subdoc does not support mutation with setting expiry

No.     Time           Source                Source Port Destination           Destination Port Protocol Length Info
    300 2.508976649    127.0.0.1             37894       127.0.0.1             37509            Couchbase 152    Subdoc Multipath Mutation Request, Opcode: 0xd1, VBucket: 0x2f3

Frame 300: 152 bytes on wire (1216 bits), 152 bytes captured (1216 bits) on interface lo, id 0
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
Internet Protocol Version 4, Src: 127.0.0.1, Dst: 127.0.0.1
Transmission Control Protocol, Src Port: 37894, Dst Port: 37509, Seq: 724, Ack: 23628, Len: 86
Couchbase Protocol, Subdoc Multipath Mutation Request, Opcode: 0xd1, VBucket: 0x2f3
    Magic: Request (0x80)
    Opcode: Subdoc Multipath Mutation (0xd1)
    Key Length: 40
    Extras Length: 4
    Data Type: 0x00
        .... ...0 = JSON: Not set
        .... ..0. = Snappy: Not set
        .... .0.. = XATTR: Not set
    VBucket: 755 (0x02f3)
    [Value Length: 18]
    Total Body Length: 62
    Opaque: 0x0000000a
    CAS: 0x0000000000000000
    Extras
        Expiration: 10
    Key: 
        [Expert Info (Warning/Undecoded): Trailing stray characters]
        Collection ID: 0x00000000
        Collection Logical Key: subdoc_test_1118_foo_7543975.4816345461
    Mutation spec [ 0 ]
        Opcode: Subdoc Dictionary Add (0xc7)
        Subdoc flags: 0x00
            .... ...0 = MKDIR_P: Not set
            .... .0.. = XATTR_PATH: Not set
            ...0 .... = EXPAND_MACROS: Not set
            000. 0.0. = Reserved fields: 0x00
        Path Length: 4
        Value Length: 6
        Path: foo2
        Value: "bar2"

0000  00 00 00 00 00 00 00 00 00 00 00 00 08 00 45 00   ..............E.
0010  00 8a 1a c0 40 00 40 06 21 ac 7f 00 00 01 7f 00   ....@.@.!.......
0020  00 01 94 06 92 85 df c6 9b c3 6c 76 95 f8 80 18   ..........lv....
0030  02 00 fe 7e 00 00 01 01 08 0a f1 f7 9b 0f f1 f7   ...~............
0040  9b 0f 80 d1 00 28 04 00 02 f3 00 00 00 3e 0a 00   .....(.......>..
0050  00 00 00 00 00 00 00 00 00 00 00 00 00 0a 00 73   ...............s
0060  75 62 64 6f 63 5f 74 65 73 74 5f 31 31 31 38 5f   ubdoc_test_1118_
0070  66 6f 6f 5f 37 35 34 33 39 37 35 2e 34 38 31 36   foo_7543975.4816
0080  33 34 35 34 36 31 c7 00 00 04 00 00 00 06 66 6f   345461........fo
0090  6f 32 22 62 61 72 32 22                           o2"bar2"

No.     Time           Source                Source Port Destination           Destination Port Protocol Length Info
    302 2.509059670    127.0.0.1             37509       127.0.0.1             37894            Couchbase 94     Subdoc Multipath Mutation Response with flexible framing extras, Opcode: 0xd1

Frame 302: 94 bytes on wire (752 bits), 94 bytes captured (752 bits) on interface lo, id 0
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
Internet Protocol Version 4, Src: 127.0.0.1, Dst: 127.0.0.1
Transmission Control Protocol, Src Port: 37509, Dst Port: 37894, Seq: 23628, Ack: 810, Len: 28
Couchbase Protocol, Subdoc Multipath Mutation Response with flexible framing extras, Opcode: 0xd1
    Magic: Response with flexible framing extras (0x18)
    Opcode: Subdoc Multipath Mutation (0xd1)
    Flexible Framing Extras Length: 3
    Key Length: 1
    Extras Length: 0
    Data Type: 0x00
        .... ...0 = JSON: Not set
        .... ..0. = Snappy: Not set
        .... .0.. = XATTR: Not set
    Status: Invalid arguments (0x0004)
        [Expert Info (Warning/Undecoded): Subdoc Multipath Mutation: Invalid arguments]
    [Value Length: 0]
    Total Body Length: 4
    Opaque: 0x0000000a
    CAS: 0x0000000000000000
    Flexible Framing Extras: 3
    Flexible Frame 0
        Flexible Frame ID (response): Server Recv->Send duration (0)
        Flexible Frame Len: 2
        Server Recv->Send duration: 18.6357373834953µs

0000  00 00 00 00 00 00 00 00 00 00 00 00 08 00 45 00   ..............E.
0010  00 50 09 ad 40 00 40 06 32 f9 7f 00 00 01 7f 00   .P..@[email protected].......
0020  00 01 92 85 94 06 6c 76 95 f8 df c6 9c 19 80 18   ......lv........
0030  02 00 fe 44 00 00 01 01 08 0a f1 f7 9b 10 f1 f7   ...D............
0040  9b 0f 18 d1 03 01 00 00 00 04 00 00 00 04 0a 00   ................
0050  00 00 00 00 00 00 00 00 00 00 02 00 08 00         ..............

subdoc counter should return invalid_delta error when delta is zero

caves should return 0x24 (no_access) if SDK asks to select missing bucket

kv_engine returns 0x24 (no_access) instead of 0x01 (not_found)

https://github.com/couchbase/kv_engine/blob/03c9956efe39407399288e59170d8c7cb4eb8585/daemon/protocol/mcbp/select_bucket_executor.cc#L28-L38

Touch and get and touch do not work

Add effective roles to users

This is needed as a part of the external API but also would simplify internal permissions handling.

Subdoc returns success status on path not found

A test like:

singleGetOpCheckExpectedFailure("{\"foo\":\"bar\"}", "no_exist", PathNotFoundException.class);

(so doing a get on a path that does not exist) on the real server does not return a success status, but on caves seems to return one.

Inserting document into dropped scope/collection does not generate error

It is expected the same behaviour when the collection does not exists

GetRandom can infinite loop

The logic when a doc is expired means that the "counter" never increments so we just keep checking the same doc over and over.

Enhanced durability (FeatureSyncReplication) does not work

Support sync replication or do not advertise it via "durableWrite"

Right now it seems based on java testing that sync replication is advertised via "durableWrite" but then not actually working. So for now maybe not advertise it, or implement basic support.

Cluster manager not found apis should return 404

I have a test which calls a /poolsDoesNotExist endpoint on ns server and the real one returns with a 404 while gocaves right now with a 501.. A 501 kinda makes sense, but should probably only be returned on API we know exists but is just not yet implemented.

Views: debug=true leads to http 500

One of the java sdk tests sets debug=true, which seems to trip up caves.

         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 47 45 54 20 2f 64 65 66 61 75 6c 74 2f 5f 64 65 |GET /default/_de|
|00000010| 73 69 67 6e 2f 65 76 65 72 79 74 68 69 6e 67 2f |sign/everything/|
|00000020| 5f 76 69 65 77 2f 61 6c 6c 3f 64 65 62 75 67 3d |_view/all?debug=|
|00000030| 74 72 75 65 20 48 54 54 50 2f 31 2e 31 0d 0a 63 |true HTTP/1.1..c|
|00000040| 6f 6e 74 65 6e 74 2d 74 79 70 65 3a 20 61 70 70 |ontent-type: app|
|00000050| 6c 69 63 61 74 69 6f 6e 2f 6a 73 6f 6e 0d 0a 63 |lication/json..c|
|00000060| 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 3a 20 30 |ontent-length: 0|
|00000070| 0d 0a 61 75 74 68 6f 72 69 7a 61 74 69 6f 6e 3a |..authorization:|
|00000080| 20 42 61 73 69 63 20 51 57 52 74 61 57 35 70 63 | Basic QWRtaW5pc|
|00000090| 33 52 79 59 58 52 76 63 6a 70 77 59 58 4e 7a 64 |3RyYXRvcjpwYXNzd|
|000000a0| 32 39 79 5a 41 3d 3d 0d 0a 68 6f 73 74 3a 20 31 |29yZA==..host: 1|
|000000b0| 32 37 2e 30 2e 30 2e 31 3a 35 33 39 33 37 0d 0a |27.0.0.1:53937..|
|000000c0| 75 73 65 72 2d 61 67 65 6e 74 3a 20 63 6f 75 63 |user-agent: couc|
|000000d0| 68 62 61 73 65 2d 6a 61 76 61 2f 30 2e 30 2e 30 |hbase-java/0.0.0|
|000000e0| 20 28 4d 61 63 20 4f 53 20 58 20 31 30 2e 31 35 | (Mac OS X 10.15|
|000000f0| 2e 37 20 78 38 36 5f 36 34 3b 20 4f 70 65 6e 4a |.7 x86_64; OpenJ|
|00000100| 44 4b 20 36 34 2d 42 69 74 20 53 65 72 76 65 72 |DK 64-Bit Server|
|00000110| 20 56 4d 20 31 35 2b 33 36 29 0d 0a 0d 0a       | VM 15+36)....  |
+--------+-------------------------------------------------+----------------+

returns

         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 48 54 54 50 2f 31 2e 31 20 35 30 30 20 49 6e 74 |HTTP/1.1 500 Int|
|00000010| 65 72 6e 61 6c 20 53 65 72 76 65 72 20 45 72 72 |ernal Server Err|
|00000020| 6f 72 0d 0a 44 61 74 65 3a 20 4d 6f 6e 2c 20 31 |or..Date: Mon, 1|
|00000030| 30 20 4d 61 79 20 32 30 32 31 20 31 30 3a 31 31 |0 May 2021 10:11|
|00000040| 3a 33 36 20 47 4d 54 0d 0a 43 6f 6e 74 65 6e 74 |:36 GMT..Content|
|00000050| 2d 4c 65 6e 67 74 68 3a 20 32 31 0d 0a 43 6f 6e |-Length: 21..Con|
|00000060| 74 65 6e 74 2d 54 79 70 65 3a 20 74 65 78 74 2f |tent-Type: text/|
|00000070| 70 6c 61 69 6e 3b 20 63 68 61 72 73 65 74 3d 75 |plain; charset=u|
|00000080| 74 66 2d 38 0d 0a 0d 0a 69 6e 74 65 72 6e 61 6c |tf-8....internal|
|00000090| 20 73 65 72 76 65 72 20 65 72 72 6f 72          | server error   |
+--------+-------------------------------------------------+----------------+

java test case:

  @Test
  void canReadDebugInfo() {
    ViewResult viewResult = bucket.viewQuery(DDOC_NAME, VIEW_NAME, viewOptions().debug(true));
    assertTrue(viewResult.metaData().debug().isPresent());
  }

design doc:

  private static void createDesignDocument() {
    Map<String, View> views = new HashMap<>();
    views.put(VIEW_NAME, new View("function(doc,meta) { emit(meta.id, doc) }"));
    views.put(VIEW_WITH_REDUCE_NAME, new View("function(doc,meta) { emit(meta.id, doc) }", "_count"));

    DesignDocument designDocument = new DesignDocument(DDOC_NAME, views);
    bucket.viewIndexes().upsertDesignDocument(designDocument, DesignDocumentNamespace.PRODUCTION);
  }

Add support for subdoc seqno macros

caves return internal error for look_up_in/get in non-JSON document

Instead it should return subdoc_doc_not_json/0xc6

The document value is string (six bytes).

GOCAVES 17:52:51.555050 received kv packet 0xc00009e9b0 CMD:CMD_SUBDOCMULTILOOKUP
GOCAVES 17:52:51.555140 Recieved unexpected crud proc error: invalid character 's' looking for beginning of value
GOCAVES 17:52:51.555168 sending kv packet 0xc00009e9b0 CMD:CMD_SUBDOCMULTILOOKUP &{Magic:129 Command:208 Datatype:0 Status:204 Vbucket:0 Opaque:10 Cas:1683557571540877514 CollectionID:0 Key:[] Extras:[] Value:[0 132 0 0 0 0] BarrierFrame:<nil> DurabilityLevelFrame:<nil> DurabilityTimeoutFrame:<nil> StreamIDFrame:<nil> OpenTracingFrame:<nil> ServerDurationFrame:0xc0001e7a48 UnsupportedFrames:[]}

Get with expiration (subdoc $document.exptime) does not work

Preserve expiry is not supported for mutations

Implement observe (with multi keys too)

The old-style "observe via cas" is used in a BatchHelper that has been implemented for a customer. These tests fail since I assume the old style observe is not yet implemented.

Support /pools/default/certificate endpoint

We have java tests which grab the cert and then try to connect with that cert to the cluster - it would be good to have that API as well so I can run the SSL tests against caves completely

Implement `-version` switch

It should be possible to check version of gocaves binary.

Also it would be great if there will be command for feature testing like {"type":"hasfeature", "features":["exist"]}

Add support for TLS

It would be awesome if we could have TLS support, also with generating a self-signed cert that can be fetched through the ns_server API so that java can run its encryption tests.

Server durations time is probably wrong after time travel

libcouchbase compression conflicting with view query

Since Compression not really handled in gocaves. compressed document sent by libcouchbase wont be indexed when executing a view query since it wont be able to unmarshal the json.

a work around i found was to initialize my connection with compression=Compression.NONE

in the packet sent by libcouchbase how do we know the data is compressed? is it a Xattrs or a Flag ?

currently using python couchbase 3.2.7

Issues with non-existing keys in counters and arrays

If a key does not already exist then counter and array subdoc ops fail, even with create path set.

Add support for revid

Including fetching using $document.revid

Failures do not abort mutation subdocs and apply changes

If a mutation subdoc op fails then the whole command should abort and none of the changes should be applied. At the moment we continue with the ops and apply any changes that didn't error.

Merge GetAll and findDocLocked logic

"Looks good to me, although we should try and merge the logic of findDocLocked and GetAll w.r.t. handling of LockTime, ExpiryTime, etc..." #61 (review)

subdoc counter returns internal error when trying to increment a number, that is too big

The number is still encoded according to JSON rules

Something like this

auto big_value = R"({"counter":)" + std::to_string(std::numeric_limits<int64_t>::max()) + "999999999999999999999999999999}";

It logs

GOCAVES 18:59:19.380689 received kv packet 0xc000290320 CMD:CMD_SUBDOCMULTIMUTATION
GOCAVES 18:59:19.380729 Recieved unexpected crud proc error: invalid character '=' looking for beginning of value
GOCAVES 18:59:19.380745 sending kv packet 0xc000290320 CMD:CMD_SUBDOCMULTIMUTATION &{Magic:129 Command:209 Datatype:0 Status:204 Vbucket:0 Opaque:12 Cas:0 CollectionID:0 Key:[] Extras:[] Value:[0 0 132] BarrierFrame:<nil> DurabilityLevelFrame:<nil> DurabilityTimeoutFrame:<nil> StreamIDFrame:<nil> OpenTracingFrame:<nil> ServerDurationFrame:0xc000338288 UnsupportedFrames:[]}

// Check that the new config data is newer than the current one, in the case where we've done a select bucket
// against an existing connection then the revisions could be the same. In that case the configuration still
// needs to be applied.
if cfg.revID == 0 {
    logDebugf("Unversioned configuration data, switching.")
} else if cfg.bktType != oldCfg.bktType {
    logDebugf("Configuration data changed bucket type, switching.")
} else if cfg.revID == oldCfg.revID {
    logDebugf("Ignoring configuration with identical revision number")
    return false
} else if cfg.revID < oldCfg.revID {
    logDebugf("Ignoring new configuration as it has an older revision id")
    return false
}

caves does not implement hello?

A java kv integration test fails on something (simplified):

    MutationResult insertResult = collection.insert(id, "Hello, World");
    assertTrue(insertResult.mutationToken().isPresent());

Looks like mutation tokens are not yet implemented for caves?

Mock does not support CreateAsDeleted

caves does not implement get with meta?

Some java KV integration tests fail around exists which uses get with meta internally. caves returns with:

com.couchbase.client.core.error.CouchbaseException: GetMetaRequest failed with unexpected status code UNKNOWN

So it looks like it might be not yet implemented?

	// TODO(brett19): Implement feature checking for mutation tokens.
	if true {
	mtBuf := make([]byte, 16)
	binary.BigEndian.PutUint64(mtBuf[0:], resp.VbUUID)
	binary.BigEndian.PutUint64(mtBuf[8:], resp.SeqNo)
	extrasBuf = append(extrasBuf, mtBuf...)
	}