tencent / knvprotoengine Goto Github PK

项目目前是存档状态，感谢您对腾讯开源项目的关注！您可以继续fork后更新迭代，感谢理解和支持；如果您有其他疑问，建议请发送邮件：[email protected] 与我们联系

KNV(Key-N-Value) is a source code library for manipulating protocol data without knowing the detail of it contents. KNV can be seen as a Key-Value extension that supports all data structures.

As we know, Key-Value systems such as memcached is very inconvenient, for examples:

---- If you wish to read a sub-field in value, you need to read the whole value and extract it by your self;

---- If data can be modified by different clients, they have to avoid write-conflict by themselves, e.g., by undertaking an optimistic locking;

---- If you want to increase a field in value, you still need to read and write all value, consuming a lot of network resources.

To solve these problems, Redis comes to help by supporting much more complicated data structures. But redis still has many difficulties:

---- Each data structure has its own implementation, making it diffcult to add more data structures, thus the end-users have to FIT their data model to existing data structures;

---- Once a data structure is used, it is almost impossible to change to another one;

---- ... ...

OK, here comes KNV, all the above difficulties are solved completely.

KNV serves for 3 main purposes:

1, As a fast protocol engine, supporting 1M+ processes per second. As KNV supports only Protocol Buffers for now, it can be use to handle PB network data in place of Google's original PB library.

2, As a schema-free protocol inspecter/modifier for general-purpose network server. For example, a network proxy can use KNV to examine network data and modify their contents if necessary.

3, As a protocol and data storage engine for general data storage server.

A network invocation can be treated as an operation performed on a set of objects (or sub-objects), thus we can denote the invocation as:

     Network invocation = (Operation, Data, Path)

For example, I have all my classmates' information stored on a remote server, and I wish to change my classmate A's remark to Alice and B's to Bob, my invocation will be like this:

     Operation = Set value
     
     Data = {Alice, Bob}
     
     Path = I->Classmates->{A,B}->Remark

Using traditional method, the protocol designer needs to take classmate and remark into account, but if you are implementing a general sotrage server, you probably do not like this.

Many known storage servers use Key-Value (Memcached) or complicated data structures (Redis) to implement a generalized method, but still have many tradeoffs as mentioned above.

KNV solves these problems by introducing the concept of "Request Tree".

A request tree is the directories of all requested sub-objects linked together. Given a request tree, the storage server knowns exactly what sub-objects are to be operated on, whitout knowing the meaning of them.

A KNV tree is common PB tree, except that each node is identified by Tag + Key. Tag is the PB tag number, Key is the value of a special sub node with tag=1. A Leaf is a special node that is either non-expandable or has not been expanded.

The first 10 tag numbers are reserved for META data in a node, if you wish KNV tree to manage nodes, please define your data nodes beginning at tag 11.

     2013-10-12   Yu Zhenshen       Created
     
     2013-11-04   Yu Zhenshen       Add parent pointer and eval_size to optimize folding
     
     2014-01-17   Yu Zhenshen       Use mem_pool for dynamic memory management
     
     2014-01-28   Yu Zhenshen       Use KnvHt to optimize hash initialization
     
     2014-05-17   Yu Zhenshen       Meta use KnvNode instead of KnvLeaf

Step 1, write your own .proto file:

--------test.proto-----------------

     message TestRequestBody
     {
     
     	optional string user_id = 1;
     	
     	optional uint32 operation = 11;
     	
     	optional string reason = 12;
     	
     };
     
     message TestResponseBody
     {
     
     	optional string result = 11;
     	
     };

---------end of test.proto-----------

Step 2, use knvshow to generate a corresponding .h file:

Execute: ./knvshow -p test.proto > test.knv.h

--------test.knv.h-------------------

     enum TestRequestBodyTags
     {
     
     	TestRequestBodyTag_user_id = 1,
     	
     	TestRequestBodyTag_operation = 11,
     	
     	TestRequestBodyTag_reason = 12,
     };
      
     enum TestResponseBodyTags
     
     {
     	TestResponseBodyTag_result = 11,
     	
     };

-------end of test.knv.h-------------

Step 3, write your own code that handles protocol with knv API:

     KnvNode *req = KnvNode::NewTree(1); // Eache node requires a tag for KNV, but each PB message needs not if it is a root node, so here we assign one anyway
     
     if(req==NULL) // something goes wrong
     
     {
     
     	cout << "KnvNode::NewTree failed: " << KnvNode::GetGlobalErrorMsg() << endl;
     	
     	return -1;
     	
     }
     
     req->SetFieldStr(TestRequestBodyTag_user_id, 4, "test");
     
     req->SetFieldInt(TestRequestBodyTag_operation, 1);
     
     req->SetFieldStr(TestRequestBodyTag_reason, 8, "for test");
     
     char buffer[1024];
     
     int len = sizeof(buffer);
     
     req->Serialize(buffer, &len, false); // not encoding header tag
     
     KnvNode::Delete(req); // you should delete req when it is no longer in use
     
     // Now buffer contains the same data as TestRequestBody.SerialzeToString()
     
     // Next, write your own code that handles remote procedure call, we assume that buffer contains the whole response packet
     
     string rsp_body(buffer, len);
     
     KnvNode *rsp = KnvNode::NewFromMessage(rsp_body);
     
     cout << rsp->GetFieldStr(TestResponseBodyTag_result) << endl;
     
     KnvNode::Delete(rsp); // you should delete rsp when it is no longer in use