pmem / pmse Goto Github PK

Performance drops in Record Store due to unnecessary checks for id presence.

Update ns in identList when collection change its ns

Not implemented function initAsEmpty in pmse_sorted_data_interface.h.

Problem with test:
dbhash2.js

When cursor is on data which changed should skip the result that no longer matches or should get back the old version of the doc.
In debug mode assertion failed:

2017-03-01T15:42:29.286+0100 I - [conn1] Invariant failure _forward ? cmp <= 0 : cmp >= 0 src/mongo/db/exec/index_scan.cpp 171

Impacted test:
jstests/core/getmore_invalidation.js

arrayfind8.js

Improve management of transaction status during allocation. Sometimes transaction may fail due to lack of space, this should be managed identically everywhere.

Modify function: void initialize(bool firstRun) to avoid problems with small pool size in mapper.

Currently some tests do steps:

• RS cursor: next() - returns value that MongoDB wants to remove
• RS: findRecord(x)
• RS: remove x

This scenario happens when "$lt: x" is passed to remove.

Comparison between index entries, when one entry has more fields then other may return wrong value, i.e.:
seek key={ g: 1.0 }
key[0]={ : 1.0, : 1.0 }
cmp=-1 -> it means that seek key is smaller than key[0], so key[0] will be returned.
However 'g' in seek key means: return value bigger than 1.0.

db.test.createIndex({a:1,b:1})
{
"createdCollectionAutomatically" : true,
"numIndexesBefore" : 1,
"numIndexesAfter" : 2,
"ok" : 1
}
db.test.insert({a:1,b:1})
WriteResult({ "nInserted" : 1 })
db.test.insert({a:1,b:2})
WriteResult({ "nInserted" : 1 })
db.test.insert({a:2,b:1})
WriteResult({ "nInserted" : 1 })
db.test.insert({a:2,b:2})
WriteResult({ "nInserted" : 1 })
db.test.find({a:{$gt:1}})
{ "_id" : ObjectId("58b68b1a4dc02dbe9c377499"), "a" : 2, "b" : 1 }
{ "_id" : ObjectId("58b68b1d4dc02dbe9c37749a"), "a" : 2, "b" : 2 }
db.test.find({a:{$gt:1},b:{$gt:1}}) <----endless loop
Also: jstests/core/distinct_index1.js: Test that a distinct over a trailing field of the index can be covered.

Remove tests below fails with PR #95:

remove2.js
remove3.js
remove6.js
remove7.js
remove9.js
removeb.js
removec.js
remove.js

Mongo-Perf testcases fails with PR #95.
Update.SetWithMultiIndex.String in complex_update.js

Problem appears in some operations after execute ensureIndex.

all.js
and.js

something fail too, but also fail with comment line ensureIndex

array_match4.js

Fail when we execute drop() after command ensureIndex()
countc.js

Avoid duplications in PMSE code.
Replace duplicated parts of code with functions.

Write prototype of Recovery Unit class dedicated to PMSE.

This tests seems to enter in infinite loop:
Time exceeded: countc.js
Time exceeded: cursor4.js
Time exceeded: distinct1.js
Time exceeded: distinct_array1.js
Time exceeded: distinct_compound_index.js
Time exceeded: distinct_index1.js
Time exceeded: distinct_index2.js
Time exceeded: distinct_speed1.js
Time exceeded: exists4.js
Time exceeded: exists.js
Time exceeded: explain_distinct.js
Time exceeded: explain_multikey.js
Time exceeded: explain_multi_plan.js
Time exceeded: geo_s2descindex.js
Time exceeded: geo_s2index.js
Time exceeded: geo_s2nongeoarray.js
Time exceeded: geo_s2nonstring.js
Time exceeded: geo_s2twofields.js
Time exceeded: in2.js
Time exceeded: in3.js
Time exceeded: index_check6.js
Time exceeded: index_stats.js
Time exceeded: insert1.js
Time exceeded: max_time_ms.js
Time exceeded: ne1.js
Time exceeded: ne2.js
Time exceeded: nin.js
Time exceeded: not1.js
Time exceeded: not3.js
Time exceeded: null.js
Time exceeded: or3.js
Time exceeded: regex5.js
Time exceeded: removeb.js

Implement rename functionality.

Impacted tests:

updatef.js
fts_index2.js
ns_length.js
rename.js
rename2.js
rename3.js
rename8.js
rename_stayTemp.js

Check items on _deleted list are freed when collection drop occurs.

Actually insert in PmseSortedDataInterface always add duplicates to tree.

Impacts: capped5.js

There are 2 steps in removal from index:

1. findRecord(key) with Index Cursor - returns RecordID
2. unindex(key, RecordID) in SortedDataInterface, which also looks for key in tree.
   In case of many duplicated keys in tree, searching is very non-optimal, can be compared to looking in list (~O(n)).
   Idea is to cache location of (key,recordID) pair in IndexCursor : seek() or next() methods, and then reuse them in unindex() method, instead of looking for them again.
   Cache may be placed in RecoveryUnit instance created for current operation.

fts1.js
fts2.js
fts3.js
fts4.js
fts5.js
fts_diacritic_and_caseinsensitive.js

Tests impacted:
jstests/core/arrayfind4.js
jstests/core/arrayfind9.js
jstests/core/capped6.js
jstests/core/capped.js
jstests/core/covered_index_negative_1.js
jstests/core/cursor6.js
jstests/core/evalb.js
jstests/core/find8.js
jstests/core/fts_mix.js
jstests/core/fts_querylang.js
jstests/core/geo1.js
jstests/core/index13.js
jstests/core/index_diag.js
jstests/core/indexu.js
jstests/core/minmax.js
jstests/core/natural.js
jstests/core/notablescan.js
jstests/core/numberlong2.js
jstests/core/orj.js
jstests/core/profile1.js
jstests/core/queryoptimizera.js
jstests/core/sortc.js
jstests/core/sortg.js
jstests/core/startup_log.js

It is possible to simplify tree implementation by getting rid of non-unique values. Instead of creating tree entries from duplicated keys given by Mongo, it is better to merge given key with unique RecordID related to record to which given key belongs. Such pair is always unique. Comparing function should first check key value, and if it is equal, it should compare RecordID. This modification simplifies searching in tree and removes many corner cases.

Tests impacted:
dbstats.js

Tests related with fsync cause invariant failure.

Tests impacted:

fsync.js
currentop.js
killop_drop_collection.js

In order to be able to use such modification:
pmem/issues#549
we must ensure that there is no pool open/close in any thread during reading/writing etc. We should have lock on RecordStore and Index creation/opening/closing.

Optimize concurrent access to index tree with fine-grained locking mechanism.

Problem in test
jstests/core/capped_max1.js

Cursor returns records from index in reverse order.
Connected with multikey indexes.
Test:
jstests/core/null2.js

Tests impacted:
jstests/core/rename.js (line 55)

Implement cursor seek() method that works independently from unindex() method. Currently unindex() method modifies "cursor" object which is defined under "tree" object.

Todo:

• Change class name to PmseSortedDataBuilderInterface
• Change constructor to get dupsAllowed and save it as class field
• Pass saved dupsAllowed to tree->insert

Scenarios to be focus on:

1. Insert to Record Store + 1 index.
2. Insert to Record Store + N indexes.
3. Find Record basing on index query (index seek() + Record Store seekExact() will be used).
4. Find Record basing on index query with projection ( only index will be used).
5. Find Record without index usage (Iterating in Record Store)
6. Remove from RS + index.

Reproduction:
Run impacted test, go to mongo shell:

t = db.jstests_indexr;
// Iterate forward works
t.find({'a.b': {$gt: 2}, 'a.c': {$lt: 4}}).sort({'a.b': 1}).itcount();
// It returns zero records
t.find({'a.b': {$gt: 2}, 'a.c': {$lt: 4}}).sort({'a.b': -1}).itcount();

Impacted test:
indexr.js

When running multiple threads (i.e. with YCSB) which inserts records there is error in Index implementation. Some cases results in index tree having leafs "next" and "previous" pointers incorrectly which results in not displaying all records with IndexCursor itcount() method.

New MongoDB cause build break with PMSE.

Commit ID: 1346352ff66b163fb010a270d1826c18699f3b36

Log:
Compiling build/opt/mongo/db/commands/index_filter_commands_test.o Compiling build/opt/mongo/db/modules/pmse/pmse_init.o In file included from src/mongo/db/modules/pmse/pmse_init.cpp:33:0: src/mongo/db/modules/pmse/pmse_engine.h:43:43: fatal error: mongo/db/storage/kv/kv_engine.h: No such file or directory compilation terminated. Linking build/opt/mongo/db/commands/list_collections_filter_test Generating placeholder library build/opt/mongo/db/query/libquery_test_service_context.a Skipping ranlib for build/opt/mongo/db/query/libquery_test_service_context.a Compiling build/opt/mongo/db/modules/pmse/pmse_engine.o scons: *** [build/opt/mongo/db/modules/pmse/pmse_init.o] Error 1 Compiling build/opt/mongo/db/modules/pmse/pmse_record_store.o src/mongo/db/modules/pmse/pmse_engine.cpp:36:34: fatal error: mongo/platform/basic.h: No such file or directory compilation terminated. scons: *** [build/opt/mongo/db/modules/pmse/pmse_engine.o] Error 1 src/mongo/db/modules/pmse/pmse_record_store.cpp:35:43: fatal error: mongo/db/storage/record_store.h: No such file or directory compilation terminated. scons: *** [build/opt/mongo/db/modules/pmse/pmse_record_store.o] Error 1 scons: building terminated because of errors. build/opt/mongo/db/modules/pmse/pmse_init.o failed: Error 1 build/opt/mongo/db/modules/pmse/pmse_engine.o failed: Error 1 build/opt/mongo/db/modules/pmse/pmse_record_store.o failed: Error 1

Prevent SIGV when allocation cannot be performed. This occurs when we run out of pool size.

Some tests wants reverse cursor, like: capped.js, capped6.js.
Impacts 24 tests.
find().sort( { $natural: -1 } ) causes that forward variable is set to 0.

Implement validate() in RecordStore class.
Other Engines are iterating through records with cursor and using ValidateAdaptor.validate() on them.
Impacted tests:
bulk_insert_capped.js

Problem in test
jstests/core/date3.js

This method is used by many tests. Currently it returns null, so many test results are adulterated.

Top command shows that PMSE have big memory consumption.
800k inserts.
PMSE after that:

VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
5769556 1,560g 1,543g S   0,0 20,3   0:40.46 mongod

WT:

VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
555432 347564  29476 S   4,3  4,3   0:48.05 mongod

Add TruncateChange class with rollback() method

SIGV is caused by restore(), in that function we try to access class member (node) in null pointer.

Impacted test:
remove2.js

Stacktrace:

 mongod(_ZN5mongo15printStackTraceERSo+0x36) [0x55bcb0dfd6d4]
 mongod(+0x2F41567) [0x55bcb0dfc567]
 mongod(+0x2F41ACC) [0x55bcb0dfcacc]
 libpthread.so.0(+0x11390) [0x7f0f9d873390]
 mongod(_ZNK4nvml3obj1pImEcvmEv+0xC) [0x55bcb0498782]
 mongod(_ZN5mongo10PmseCursor7restoreEv+0x12E) [0x55bcb04a1760]
 mongod(_ZN5mongo9IndexScan14doRestoreStateEv+0x43) [0x55bcb031d8e3]
 mongod(_ZN5mongo9PlanStage12restoreStateEv+0xBB) [0x55bcb0335743]
 mongod(_ZN5mongo9PlanStage12restoreStateEv+0x96) [0x55bcb033571e]
 mongod(_ZN5mongo11DeleteStage6doWorkEPm+0x6F0) [0x55bcb02fc4a8]
 mongod(_ZN5mongo9PlanStage4workEPm+0xAC) [0x55bcb0335518]
 mongod(_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x616) [0x55bcb070129e]
 mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x55) [0x55bcb0700ba1]
 mongod(_ZN5mongo12PlanExecutor11executePlanEv+0x9B) [0x55bcb0701b15]
 mongod(+0x2651A84) [0x55bcb050ca84]
 mongod(_ZN5mongo14performDeletesEPNS_16OperationContextERKNS_8DeleteOpE+0x2B3) [0x55bcb050d129]
 mongod(_ZN5mongo9CmdDelete7runImplEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjERNS_14BSONObjBuilderE+0x89) [0x55bcb02a7e59]
 mongod(+0x23EBC5F) [0x55bcb02a6c5f]
 mongod(_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE+0x751) [0x55bcb01f86dd]
 mongod(_ZN5mongo19execCommandDatabaseEPNS_16OperationContextEPNS_7CommandERKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE+0xDF8) [0x55bcb01f9e05]
 mongod(+0x233CECB) [0x55bcb01f7ecb]
 mongod(_ZNSt17_Function_handlerIFvPN5mongo16OperationContextEPNS0_7CommandERKNS0_3rpc16RequestInterfaceEPNS5_21ReplyBuilderInterfaceEEPSB_E9_M_invokeERKSt9_Any_dataOS2_OS4_S8_OSA_+0x7C) [0x55bcb020481b]
 mongod(_ZNKSt8functionIFvPN5mongo16OperationContextEPNS0_7CommandERKNS0_3rpc16RequestInterfaceEPNS5_21ReplyBuilderInterfaceEEEclES2_S4_S8_SA_+0x8E) [0x55bcb01dcbd2]
 mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE+0x3A) [0x55bcb01db38c]
 mongod(_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE+0x544) [0x55bcb091c821]
 mongod(+0x221EDDB) [0x55bcb00d9ddb]
 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x628) [0x55bcb00dc68d]
 mongod(_ZN5mongo23ServiceEntryPointMongod12_sessionLoopERKSt10shared_ptrINS_9transport7SessionEE+0x1E7) [0x55bcaffc35d9]
 mongod(+0x21082ED) [0x55bcaffc32ed]
 mongod(+0x2108A58) [0x55bcaffc3a58]
 mongod(_ZNKSt8functionIFvRKSt10shared_ptrIN5mongo9transport7SessionEEEEclES6_+0x49) [0x55bcb0d5c8a7]
 mongod(+0x2EA0823) [0x55bcb0d5b823]
 libpthread.so.0(+0x76BA) [0x7f0f9d8696ba]
 libc.so.6(clone+0x6D) [0x7f0f9d59f82d]
-----  END BACKTRACE  -----
Segmentation fault (core dumped)

Can be inserted a few records with the same key.

Impacted tests:

apply_ops_dups.js
batch_write_command_insert.js
batch_write_command_update.js
bindata_indexonly.js
bulk_api_ordered.js
bulk_api_unordered.js
capped5.js
covered_index_simple_2.js
covered_index_simple_3.js
crud_api.js
dbref3.js
elemMatchProjection.js
exists4.js
geo_distinct.js
geo_uniqueDocs2.js
geo_uniqueDocs.js
hashindex1.js
index8.js
indexa.js
indexapi.js
indexb.js
index_dropdups_ignore.js
index_id_options.js
index_multiple_compatibility.js
index_sparse1.js
list_indexes_invalid.js
list_indexes.js
uniqueness.js

Some tests fails due to segmentation faults. It's cause by dereference of null pointer in PmseCursor (pmse_index_cursor.cpp:725) after moveToNext().

Error:
GetMore command executor error: CappedPositionLost: CollectionScan died due to position in capped collection bein deleted. Last seen record id: RecordId(2)

Test impacted:
jstests/core/getmore_cmd_maxtimems.js

Impacts 34 tests. Assertion give information what MongoDB expects from engine.

Invariant failure _forward ? cmp >= 0 : cmp <= 0 src/mongo/db/exec/index_scan.cpp 162
5 tests fails:

jstests/core/arrayfind9.js
jstests/core/covered_index_sort_1.js
jstests/core/index_check3.js
jstests/core/index_decimal.js
jstests/core/minmax_edge.js

Invariant failure _endKeyInclusive src/mongo/db/exec/index_scan.cpp 170
2 tests fails:

jstests/core/indexj.js
jstests/core/or7.js

Invariant failure _forward ? cmp <= 0 : cmp >= 0 src/mongo/db/exec/index_scan.cpp 171
27 tests fails:

jstests/core/cursor7.js
jstests/core/fts2.js
jstests/core/fts3.js
jstests/core/fts4.js
jstests/core/fts5.js
jstests/core/fts_blog.js
jstests/core/blogwild.js
jstests/core/fts_casesensitive.js
jstests/core/fts_diacritic_and_caseinsensitive.js
jstests/core/fts_diacritic_and_casesensitive.js
jstests/core/fts_diacriticsensitive.js
jstests/core/fts_index.js
jstests/core/fts_index_version2.js
jstests/core/fts_mix.js
jstests/core/fts_partition1.js
jstests/core/phrase.js
jstests/core/fts_projection.js
jstests/core/fts_querylang.js
jstests/core/fts_score_sort.js
jstests/core/fts_spanish.js

jstests/core/getmore_invalidation.js -
jstests/core/index_bounds_number_edge_cases.js
jstests/core/index_check7.js
jstests/core/plan_cache_list_shapes.js
jstests/core/remove_undefined.js
jstests/core/sorth.js
jstests/core/stages_text.js