weinberger / nosql-tests Goto Github PK

I get the following error while trying:
nodejs benchmark postgresql -a myhost1 -t all

INFO using server address myhost1
INFO start
INFO version [{"version":"PostgreSQL 9.5.6 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609, 64-bit"}]
ERROR error: relation "relations" does not exist

I get the feeling I am missing something out. Do I have to preload the database? How?

For comparison:

• https://www.techempower.com/benchmarks/
• http://benchmarksgame.alioth.debian.org/u64q/which-programs-are-fastest.html

I understand you guys are a business and want to make arango look good. Having arango fixed to 100% makes it appear like the most consistently performing database. Though it also makes the benchmark appear more biased than it needs to be.

Either way thanks for making this benchmark, there's a lot of talk on this topic and not enough measurement.

Hi.
This is a great project. Thanks for putting it together.

MongoDB is performing pretty poorly on most of the benchmark tests provided. Despite this we're still considering it as the right solution, partly due to this one killer feature, which is perfect for our use case:

https://docs.mongodb.com/v3.0/reference/operator/update/set/#set-fields-in-embedded-documents

All of the other datastores that are being benchmarked would require the client to retrieve the document, update it and reinsert it, which I estimate would be will be much slower than the single 'update' operation provided by MongoDB above. This would be particular true for large, highly nested documents.

It would be really interesting and useful to add a benchmark test for this!

I would love to see couchbase 4 (both map reduce and N1QL) added to the tests. When I have some free time will try to put something togather.

Ubuntu 16.04 on VMware 6.5 U1.
Some OrientDB transformers failed.
Anyone see this? I'll try again to see if this is reproducible.
Is this warning is safe to ignore?

DOWNLOAD DIRECTORY: /tmp/nosqlbenchmark/downloads
Downloading PROFILES
--2018-02-15 13:49:24--  https://snap.stanford.edu/data/soc-pokec-profiles.txt.gz
Resolving snap.stanford.edu (snap.stanford.edu)... 171.64.75.80
Connecting to snap.stanford.edu (snap.stanford.edu)|171.64.75.80|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 435512787 (415M) [application/x-gzip]
Saving to: ‘/tmp/nosqlbenchmark/downloads/soc-pokec-profiles.txt.gz.tmp’

/tmp/nosqlbenchmark/downloa 100%[==========================================>] 415.34M  1.16MB/s    in 8m 32s  

2018-02-15 13:57:57 (830 KB/s) - ‘/tmp/nosqlbenchmark/downloads/soc-pokec-profiles.txt.gz.tmp’ saved [435512787/435512787]

Downloading RELATIONS
--2018-02-15 13:58:58--  https://snap.stanford.edu/data/soc-pokec-relationships.txt.gz
Resolving snap.stanford.edu (snap.stanford.edu)... 171.64.75.80
Connecting to snap.stanford.edu (snap.stanford.edu)|171.64.75.80|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 132454730 (126M) [application/x-gzip]
Saving to: ‘/tmp/nosqlbenchmark/downloads/soc-pokec-relationships.txt.gz’

/tmp/nosqlbenchmark/downloa 100%[==========================================>] 126.32M  1.23MB/s    in 2m 53s  

2018-02-15 14:01:52 (747 KB/s) - ‘/tmp/nosqlbenchmark/downloads/soc-pokec-relationships.txt.gz’ saved [132454730/132454730]

Converting PROFILES
Converting RELATIONS
OrientDB etl v.2.2.29 (build 9914189f972103907c24377a1567897e68642920) https://www.orientdb.com
[file] INFO Load from file /tmp/nosqlbenchmark/downloads/soc-pokec-profiles-orientdb.txt
[file] INFO Reading from file /tmp/nosqlbenchmark/downloads/soc-pokec-profiles-orientdb.txt with encoding UTF-8

2018-02-15 14:02:47:629 WARNI {db=pokec} Transformer [csv] returned null, skip rest of pipeline execution [OETLPipeline]
2018-02-15 14:02:47:630 WARNI {db=pokec} Transformer [vertex] returned null, skip rest of pipeline execution [OETLPipeline]
2018-02-15 14:09:19:868 WARNI {db=pokec} Transformer [vertex] returned null, skip rest of pipeline execution [OETLPipeline]
2018-02-15 14:09:19:868 WARNI {db=pokec} Transformer [edge] returned null, skip rest of pipeline execution [OETLPipeline]
...

I am trying to run against the neo4j version 4.2 but I have problems with the transaction handing

node benchmark.js neo4j -a 127.0.0.1 -t 'warmup'

i am getting

INFO using server address 127.0.0.1
INFO start
INFO step 1/2 done
Neo4jError: Queries cannot be run directly on a session with an open transaction; either run from within the transaction or use a different session.

at captureStacktrace (/root/nosql-tests/node_modules/neo4j-driver/lib/result.js:281:15)
at new Result (/root/nosql-tests/node_modules/neo4j-driver/lib/result.js:68:19)
at Session._run (/root/nosql-tests/node_modules/neo4j-driver/lib/session.js:174:14)
at Session.run (/root/nosql-tests/node_modules/neo4j-driver/lib/session.js:135:19)
at Object.getDocument (/root/nosql-tests/neo4j/description.js:78:19)
at /root/nosql-tests/neo4j/description.js:43:28
at Object.getCollection (/root/nosql-tests/neo4j/description.js:62:5)
at /root/nosql-tests/neo4j/description.js:35:24
at processTicksAndRejections (node:internal/process/task_queues:94:5) {

code: 'N/A'
}

Can you add AgensGraph to the tests, please?

It's a proper implementation of a Graph Database on top of PostgreSQL and possibly a better fit in the comparison against other graph native database engines.

https://bitnine.net/agensgraph/

Since my post wasn't made public I am posting here. From the discussion on the Arango website, I'd like to hold you all to your word.

We will then publish a update - also with the new results for Neo4J. That should gives as proof-able results instead of marketing statements.

I am very interested in the results of the PR from OrientDB.

Scott

Issue description:

When you try to execute a single task, the benchmark.js return next message:
`[root@localhost nosql-tests]# node benchmark mongodb -a 127.0.0.1 -t singeRead
/home/dsuser/nosql-tests/benchmark.js:93
tests = tests.split(',').map(trim);
^

ReferenceError: trim is not defined
at Object. (/home/dsuser/nosql-tests/benchmark.js:93:32)
at Module._compile (module.js:570:32)
at Object.Module._extensions..js (module.js:579:10)
at Module.load (module.js:487:32)
at tryModuleLoad (module.js:446:12)
at Function.Module._load (module.js:438:3)
at Module.runMain (module.js:604:10)
at run (bootstrap_node.js:393:7)
at startup (bootstrap_node.js:150:9)
at bootstrap_node.js:508:3`

Solution:

Instead of this code:
tests = tests.split(',').map(trim);
I use this:
tests = tests.split(',').map(function (val) { return val; });

It looks like the setup and run scripts are not working like they used to before.
Also, can you share the environment used for the test? For example, neo4j doesn't support Java 11 and actually requires Java 8.

PostgreSQL defaults are very conservative. Suggest using http://pgtune.leopard.in.ua to at least open up basic settings to make full use of testing platform.

Wasted several days on arango (3.8), because it is twice as fast as neo4j, wow (it is not)...

1. Import time is not measured and it is terrible comparing to neo4j (100mln nodes, 300mln edges. 3 minutes for neo4j, 2.5 hours for arango (15min nodes, 2hours for edges)).
2. graph size is ridiculous, my small project is 100 times larger and neo4j often is used with billions of nodes/relations
3. shortestpath is quite useless search. neo4j's allshortestpaths is many times faster than arango's unweighted K_SHORTEST_PATHS
4. arango's weighted K_SHORTEST_PATHS and neo4j's Dijkstra Source-Target are both awfully slow, but neo4j might be faster, because it looks more mature.

general reliability and maturity.... neo4j is production ready, arango 3.6 had terrible bugs with shortestpath.
It can be measured as "days since critical bugs". Arango doesn't shine here also...

So, tests look very synthetic

nodejs benchmark neo4j -a 10.240.0.2 -t shortest

  tests = tests.split(',').map(trim);
                               ^
ReferenceError: trim is not defined
    at Object.<anonymous> (/home/maxdemarzi/nosql-tests/benchmark.js:93:32)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Function.Module.runMain (module.js:497:10)
    at startup (node.js:119:16)
    at node.js:902:3

Can you add RethinkDB to the tests, please?

the code is not executing from this line..dont know whats wrong..can you please help

module.exports = {
name: 'PostgreSQL tabular',
isPostgreSQL: true,
server: {},
singleWriteQuery : "",

startup: function (host, cb) {
var insertColumns = ['id'];
var binds = [];

The postgresql.conf file should be included for reliable reproduction of this benchmark and validation of settings. Postgres is notorious for extremely conservative resource settings by default and does no inspection of the underlying platform so the settings in this file will define how the query planner represents the underlying system resources.

In the tests you use the JSON type, while for a "real" environment that was performance-sensitive it's likely that JSONB would be used instead. Optionally, you could also add a GIN on the structure field which would likely improve the relation matching performance.

The majority of the comments on the current benchmark 2018 suggest Dgraph as comparison and I fully agree on that, especially since Neo4J is known as a comparatively slow graph database, while surely a widespread one.

I am very interested in a comparison to dgraph. :)

I use today 20171106 version nosql-tests to test orientdb-community-importers-2.2.29.tar.gz .
and I have solve the no user and password error,
but it still report an error below:
module.js:471
throw err;
^

Error: Cannot find module '/root/Documents/NimbiLin/Programs/Databases/NoSQL/Install/InsReal/DLBenchmark/node_modules/orientjs/benchmark'
at Function.Module._resolveFilename (module.js:469:15)
at Function.Module._load (module.js:417:25)
at Module.runMain (module.js:604:10)
at run (bootstrap_node.js:394:7)
at startup (bootstrap_node.js:149:9)
at bootstrap_node.js:509:3

I have followed "https://github.com/weinberger/nosql-tests" 's readme.MD to install and without any error, how to solve this?

...evidently, since the author hasn't used his GitHub profile since 2015.