pmem / pmse Goto Github PK
View Code? Open in Web Editor NEWPersistent Memory Storage Engine
License: BSD 3-Clause "New" or "Revised" License
Persistent Memory Storage Engine
License: BSD 3-Clause "New" or "Revised" License
Comparison between index entries, when one entry has more fields then other may return wrong value, i.e.:
seek key={ g: 1.0 }
key[0]={ : 1.0, : 1.0 }
cmp=-1 -> it means that seek key is smaller than key[0], so key[0] will be returned.
However 'g' in seek key means: return value bigger than 1.0.
db.test.createIndex({a:1,b:1})
{
"createdCollectionAutomatically" : true,
"numIndexesBefore" : 1,
"numIndexesAfter" : 2,
"ok" : 1
}
db.test.insert({a:1,b:1})
WriteResult({ "nInserted" : 1 })
db.test.insert({a:1,b:2})
WriteResult({ "nInserted" : 1 })
db.test.insert({a:2,b:1})
WriteResult({ "nInserted" : 1 })
db.test.insert({a:2,b:2})
WriteResult({ "nInserted" : 1 })
db.test.find({a:{$gt:1}})
{ "_id" : ObjectId("58b68b1a4dc02dbe9c377499"), "a" : 2, "b" : 1 }
{ "_id" : ObjectId("58b68b1d4dc02dbe9c37749a"), "a" : 2, "b" : 2 }
db.test.find({a:{$gt:1},b:{$gt:1}}) <----endless loop
Also: jstests/core/distinct_index1.js: Test that a distinct over a trailing field of the index can be covered.
Problem in test
jstests/core/date3.js
Mongo-Perf testcases fails with PR #95.
Update.SetWithMultiIndex.String in complex_update.js
fts1.js
fts2.js
fts3.js
fts4.js
fts5.js
fts_diacritic_and_caseinsensitive.js
It is possible to simplify tree implementation by getting rid of non-unique values. Instead of creating tree entries from duplicated keys given by Mongo, it is better to merge given key with unique RecordID related to record to which given key belongs. Such pair is always unique. Comparing function should first check key value, and if it is equal, it should compare RecordID. This modification simplifies searching in tree and removes many corner cases.
Implement rename functionality.
Impacted tests:
updatef.js
fts_index2.js
ns_length.js
rename.js
rename2.js
rename3.js
rename8.js
rename_stayTemp.js
Some tests wants reverse cursor, like: capped.js, capped6.js.
Impacts 24 tests.
find().sort( { $natural: -1 } ) causes that forward variable is set to 0.
Reproduction:
Run impacted test, go to mongo shell:
t = db.jstests_indexr;
// Iterate forward works
t.find({'a.b': {$gt: 2}, 'a.c': {$lt: 4}}).sort({'a.b': 1}).itcount();
// It returns zero records
t.find({'a.b': {$gt: 2}, 'a.c': {$lt: 4}}).sort({'a.b': -1}).itcount();
Impacted test:
indexr.js
This tests seems to enter in infinite loop:
Time exceeded: countc.js
Time exceeded: cursor4.js
Time exceeded: distinct1.js
Time exceeded: distinct_array1.js
Time exceeded: distinct_compound_index.js
Time exceeded: distinct_index1.js
Time exceeded: distinct_index2.js
Time exceeded: distinct_speed1.js
Time exceeded: exists4.js
Time exceeded: exists.js
Time exceeded: explain_distinct.js
Time exceeded: explain_multikey.js
Time exceeded: explain_multi_plan.js
Time exceeded: geo_s2descindex.js
Time exceeded: geo_s2index.js
Time exceeded: geo_s2nongeoarray.js
Time exceeded: geo_s2nonstring.js
Time exceeded: geo_s2twofields.js
Time exceeded: in2.js
Time exceeded: in3.js
Time exceeded: index_check6.js
Time exceeded: index_stats.js
Time exceeded: insert1.js
Time exceeded: max_time_ms.js
Time exceeded: ne1.js
Time exceeded: ne2.js
Time exceeded: nin.js
Time exceeded: not1.js
Time exceeded: not3.js
Time exceeded: null.js
Time exceeded: or3.js
Time exceeded: regex5.js
Time exceeded: removeb.js
Some tests fails due to segmentation faults. It's cause by dereference of null pointer in PmseCursor (pmse_index_cursor.cpp:725) after moveToNext().
Tests impacted:
dbstats.js
Performance drops in Record Store due to unnecessary checks for id presence.
Tests impacted:
jstests/core/arrayfind4.js
jstests/core/arrayfind9.js
jstests/core/capped6.js
jstests/core/capped.js
jstests/core/covered_index_negative_1.js
jstests/core/cursor6.js
jstests/core/evalb.js
jstests/core/find8.js
jstests/core/fts_mix.js
jstests/core/fts_querylang.js
jstests/core/geo1.js
jstests/core/index13.js
jstests/core/index_diag.js
jstests/core/indexu.js
jstests/core/minmax.js
jstests/core/natural.js
jstests/core/notablescan.js
jstests/core/numberlong2.js
jstests/core/orj.js
jstests/core/profile1.js
jstests/core/queryoptimizera.js
jstests/core/sortc.js
jstests/core/sortg.js
jstests/core/startup_log.js
Check items on _deleted list are freed when collection drop occurs.
Remove tests below fails with PR #95:
remove2.js
remove3.js
remove6.js
remove7.js
remove9.js
removeb.js
removec.js
remove.js
Error:
GetMore command executor error: CappedPositionLost: CollectionScan died due to position in capped collection bein deleted. Last seen record id: RecordId(2)
Test impacted:
jstests/core/getmore_cmd_maxtimems.js
Tests related with fsync cause invariant failure.
Tests impacted:
fsync.js
currentop.js
killop_drop_collection.js
Add TruncateChange class with rollback() method
When cursor is on data which changed should skip the result that no longer matches or should get back the old version of the doc.
In debug mode assertion failed:
2017-03-01T15:42:29.286+0100 I - [conn1] Invariant failure _forward ? cmp <= 0 : cmp >= 0 src/mongo/db/exec/index_scan.cpp 171
Impacted test:
jstests/core/getmore_invalidation.js
When running multiple threads (i.e. with YCSB) which inserts records there is error in Index implementation. Some cases results in index tree having leafs "next" and "previous" pointers incorrectly which results in not displaying all records with IndexCursor itcount() method.
Scenarios to be focus on:
Implement cursor seek() method that works independently from unindex() method. Currently unindex() method modifies "cursor" object which is defined under "tree" object.
arrayfind8.js
Currently some tests do steps:
This scenario happens when "$lt: x" is passed to remove.
SIGV is caused by restore(), in that function we try to access class member (node) in null pointer.
Impacted test:
remove2.js
Stacktrace:
mongod(_ZN5mongo15printStackTraceERSo+0x36) [0x55bcb0dfd6d4]
mongod(+0x2F41567) [0x55bcb0dfc567]
mongod(+0x2F41ACC) [0x55bcb0dfcacc]
libpthread.so.0(+0x11390) [0x7f0f9d873390]
mongod(_ZNK4nvml3obj1pImEcvmEv+0xC) [0x55bcb0498782]
mongod(_ZN5mongo10PmseCursor7restoreEv+0x12E) [0x55bcb04a1760]
mongod(_ZN5mongo9IndexScan14doRestoreStateEv+0x43) [0x55bcb031d8e3]
mongod(_ZN5mongo9PlanStage12restoreStateEv+0xBB) [0x55bcb0335743]
mongod(_ZN5mongo9PlanStage12restoreStateEv+0x96) [0x55bcb033571e]
mongod(_ZN5mongo11DeleteStage6doWorkEPm+0x6F0) [0x55bcb02fc4a8]
mongod(_ZN5mongo9PlanStage4workEPm+0xAC) [0x55bcb0335518]
mongod(_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x616) [0x55bcb070129e]
mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x55) [0x55bcb0700ba1]
mongod(_ZN5mongo12PlanExecutor11executePlanEv+0x9B) [0x55bcb0701b15]
mongod(+0x2651A84) [0x55bcb050ca84]
mongod(_ZN5mongo14performDeletesEPNS_16OperationContextERKNS_8DeleteOpE+0x2B3) [0x55bcb050d129]
mongod(_ZN5mongo9CmdDelete7runImplEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjERNS_14BSONObjBuilderE+0x89) [0x55bcb02a7e59]
mongod(+0x23EBC5F) [0x55bcb02a6c5f]
mongod(_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE+0x751) [0x55bcb01f86dd]
mongod(_ZN5mongo19execCommandDatabaseEPNS_16OperationContextEPNS_7CommandERKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE+0xDF8) [0x55bcb01f9e05]
mongod(+0x233CECB) [0x55bcb01f7ecb]
mongod(_ZNSt17_Function_handlerIFvPN5mongo16OperationContextEPNS0_7CommandERKNS0_3rpc16RequestInterfaceEPNS5_21ReplyBuilderInterfaceEEPSB_E9_M_invokeERKSt9_Any_dataOS2_OS4_S8_OSA_+0x7C) [0x55bcb020481b]
mongod(_ZNKSt8functionIFvPN5mongo16OperationContextEPNS0_7CommandERKNS0_3rpc16RequestInterfaceEPNS5_21ReplyBuilderInterfaceEEEclES2_S4_S8_SA_+0x8E) [0x55bcb01dcbd2]
mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE+0x3A) [0x55bcb01db38c]
mongod(_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE+0x544) [0x55bcb091c821]
mongod(+0x221EDDB) [0x55bcb00d9ddb]
mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x628) [0x55bcb00dc68d]
mongod(_ZN5mongo23ServiceEntryPointMongod12_sessionLoopERKSt10shared_ptrINS_9transport7SessionEE+0x1E7) [0x55bcaffc35d9]
mongod(+0x21082ED) [0x55bcaffc32ed]
mongod(+0x2108A58) [0x55bcaffc3a58]
mongod(_ZNKSt8functionIFvRKSt10shared_ptrIN5mongo9transport7SessionEEEEclES6_+0x49) [0x55bcb0d5c8a7]
mongod(+0x2EA0823) [0x55bcb0d5b823]
libpthread.so.0(+0x76BA) [0x7f0f9d8696ba]
libc.so.6(clone+0x6D) [0x7f0f9d59f82d]
----- END BACKTRACE -----
Segmentation fault (core dumped)
Cursor returns records from index in reverse order.
Connected with multikey indexes.
Test:
jstests/core/null2.js
Actually insert in PmseSortedDataInterface always add duplicates to tree.
Impacts: capped5.js
Prevent SIGV when allocation cannot be performed. This occurs when we run out of pool size.
Avoid duplications in PMSE code.
Replace duplicated parts of code with functions.
This method is used by many tests. Currently it returns null, so many test results are adulterated.
Not implemented function initAsEmpty in pmse_sorted_data_interface.h.
Improve management of transaction status during allocation. Sometimes transaction may fail due to lack of space, this should be managed identically everywhere.
Optimize concurrent access to index tree with fine-grained locking mechanism.
Modify function: void initialize(bool firstRun)
to avoid problems with small pool size in mapper.
Write prototype of Recovery Unit class dedicated to PMSE.
New MongoDB cause build break with PMSE.
Commit ID: 1346352ff66b163fb010a270d1826c18699f3b36
Log:
Compiling build/opt/mongo/db/commands/index_filter_commands_test.o Compiling build/opt/mongo/db/modules/pmse/pmse_init.o In file included from src/mongo/db/modules/pmse/pmse_init.cpp:33:0: src/mongo/db/modules/pmse/pmse_engine.h:43:43: fatal error: mongo/db/storage/kv/kv_engine.h: No such file or directory compilation terminated. Linking build/opt/mongo/db/commands/list_collections_filter_test Generating placeholder library build/opt/mongo/db/query/libquery_test_service_context.a Skipping ranlib for build/opt/mongo/db/query/libquery_test_service_context.a Compiling build/opt/mongo/db/modules/pmse/pmse_engine.o scons: *** [build/opt/mongo/db/modules/pmse/pmse_init.o] Error 1 Compiling build/opt/mongo/db/modules/pmse/pmse_record_store.o src/mongo/db/modules/pmse/pmse_engine.cpp:36:34: fatal error: mongo/platform/basic.h: No such file or directory compilation terminated. scons: *** [build/opt/mongo/db/modules/pmse/pmse_engine.o] Error 1 src/mongo/db/modules/pmse/pmse_record_store.cpp:35:43: fatal error: mongo/db/storage/record_store.h: No such file or directory compilation terminated. scons: *** [build/opt/mongo/db/modules/pmse/pmse_record_store.o] Error 1 scons: building terminated because of errors. build/opt/mongo/db/modules/pmse/pmse_init.o failed: Error 1 build/opt/mongo/db/modules/pmse/pmse_engine.o failed: Error 1 build/opt/mongo/db/modules/pmse/pmse_record_store.o failed: Error 1
Implement validate() in RecordStore class.
Other Engines are iterating through records with cursor and using ValidateAdaptor.validate() on them.
Impacted tests:
bulk_insert_capped.js
Problem appears in some operations after execute ensureIndex.
all.js
and.js
something fail too, but also fail with comment line ensureIndex
array_match4.js
Fail when we execute drop() after command ensureIndex()
countc.js
Top command shows that PMSE have big memory consumption.
800k inserts.
PMSE after that:
VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5769556 1,560g 1,543g S 0,0 20,3 0:40.46 mongod
WT:
VIRT RES SHR S %CPU %MEM TIME+ COMMAND
555432 347564 29476 S 4,3 4,3 0:48.05 mongod
Problem with test:
dbhash2.js
Tests impacted:
jstests/core/rename.js (line 55)
Update ns in identList when collection change its ns
Can be inserted a few records with the same key.
Impacted tests:
apply_ops_dups.js
batch_write_command_insert.js
batch_write_command_update.js
bindata_indexonly.js
bulk_api_ordered.js
bulk_api_unordered.js
capped5.js
covered_index_simple_2.js
covered_index_simple_3.js
crud_api.js
dbref3.js
elemMatchProjection.js
exists4.js
geo_distinct.js
geo_uniqueDocs2.js
geo_uniqueDocs.js
hashindex1.js
index8.js
indexa.js
indexapi.js
indexb.js
index_dropdups_ignore.js
index_id_options.js
index_multiple_compatibility.js
index_sparse1.js
list_indexes_invalid.js
list_indexes.js
uniqueness.js
There are 2 steps in removal from index:
Impacts 34 tests. Assertion give information what MongoDB expects from engine.
Invariant failure _forward ? cmp >= 0 : cmp <= 0 src/mongo/db/exec/index_scan.cpp 162
5 tests fails:
jstests/core/arrayfind9.js
jstests/core/covered_index_sort_1.js
jstests/core/index_check3.js
jstests/core/index_decimal.js
jstests/core/minmax_edge.js
Invariant failure _endKeyInclusive src/mongo/db/exec/index_scan.cpp 170
2 tests fails:
jstests/core/indexj.js
jstests/core/or7.js
Invariant failure _forward ? cmp <= 0 : cmp >= 0 src/mongo/db/exec/index_scan.cpp 171
27 tests fails:
jstests/core/cursor7.js
jstests/core/fts2.js
jstests/core/fts3.js
jstests/core/fts4.js
jstests/core/fts5.js
jstests/core/fts_blog.js
jstests/core/blogwild.js
jstests/core/fts_casesensitive.js
jstests/core/fts_diacritic_and_caseinsensitive.js
jstests/core/fts_diacritic_and_casesensitive.js
jstests/core/fts_diacriticsensitive.js
jstests/core/fts_index.js
jstests/core/fts_index_version2.js
jstests/core/fts_mix.js
jstests/core/fts_partition1.js
jstests/core/phrase.js
jstests/core/fts_projection.js
jstests/core/fts_querylang.js
jstests/core/fts_score_sort.js
jstests/core/fts_spanish.js
jstests/core/getmore_invalidation.js -
jstests/core/index_bounds_number_edge_cases.js
jstests/core/index_check7.js
jstests/core/plan_cache_list_shapes.js
jstests/core/remove_undefined.js
jstests/core/sorth.js
jstests/core/stages_text.js
Problem in test
jstests/core/capped_max1.js
Todo:
In order to be able to use such modification:
pmem/issues#549
we must ensure that there is no pool open/close in any thread during reading/writing etc. We should have lock on RecordStore and Index creation/opening/closing.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.