mdshovel is a development tool that connects directly to the Manta metadata
tier and generates certain pathological metadata workloads. The primary goal
is to reproduce the pathological database performance observed under
MANTA-3428, which involves a fairly
specific metadata layout:
- one directory that grows very large
- a large number of directories with one item in it
The generated metadata is not valid from Manta's perspective, as it creates references to storage nodes and sharks that do not exist. Running this against any Manta deployment that will ever become a production deployment is strongly discouraged.
Build with:
$ make
Then make a configuration file that looks like this:
{
"metadataService": {
"srvDomain": "3.moray.emy-10.joyent.us",
"cueballOptions": {
"resolvers": [ "nameservice.emy-10.joyent.us" ]
}
},
"concurrency": 1,
"largeDirectory": "/dap/bigdir/a",
"smallDirectoryRoot": "/dap/smalldir"
"artediPort": 1809
}
where:
metadataServicedescribes the Moray endpoint to work with, in terms of the Moray client constructor options. In a typical Manta deployment, this would point at electric-moray, but for testing this tool, it's more effective to point it at a particular set of Moray instances (for one shard).concurrencyspecifies how many concurrent application-level operations to run at a time. Each application-level operation creates an entrylargeDirectory, a few directories insmallDirectoryRoot, and an object inside one of those directories.largeDirectorypoints to a directory where one entry will be added per application-level operationsmallDirectoryRootpoints to a path where a small multi-level directory tree will be created for each application-level operationartediPortis a TCP port number on which a Prometheus endpoint will be exposed (on all network interfaces) for metrics
While the database is small, you can monitor the behavior using PostgreSQL queries like:
moray=# select _key, entries from manta_directory_counts ORDER BY entries DESC limit 20;
_key | entries
-----------------------------------+---------
/dap/bigdir/a | 2415
/dap/smalldir/ac | 178
/dap/smalldir/a4 | 164
/dap/smalldir/ad | 162
/dap/smalldir/af | 159
/dap/smalldir/a6 | 158
/dap/smalldir/a1 | 153
/dap/smalldir/aa | 152
/dap/smalldir/a3 | 152
/dap/smalldir/a8 | 151
/dap/smalldir/a0 | 148
/dap/smalldir/ae | 146
/dap/smalldir/a5 | 146
/dap/smalldir/a2 | 141
/dap/smalldir/a7 | 138
/dap/smalldir/ab | 134
/dap/smalldir/a9 | 132
/dap/smalldir | 16
/dap/smalldir/a4/a421e29a-2daf-4a | 1
/dap/smalldir/a4/a41821b1-504d-4b | 1
(20 rows)
You can also monitor the program by hitting the artedi port with curl(1) (or, of course, Prometheus):
$ curl -i localhost:1809/metrics
HTTP/1.1 200 OK
Content-Type: text/plain; version=0.0.4
Date: Tue, 19 Sep 2017 00:23:43 GMT
Connection: keep-alive
Transfer-Encoding: chunked
# HELP nstarted count of composite operations started
# TYPE nstarted counter
nstarted{} 2179 1505780623275
# HELP ndone count of composite operations completed (including failures)
# TYPE ndone counter
ndone{} 2178 1505780623274
# HELP nfail count of composite operations that have failed
# TYPE nfail counter
# HELP composite_latency_ms latency of composite operations
# TYPE composite_latency_ms histogram
composite_latency_ms{le="81"} 1946 1505780623274
composite_latency_ms{le="243"} 2178 1505780623274
composite_latency_ms{le="405"} 2178 1505780623274
composite_latency_ms{le="567"} 2178 1505780623274
composite_latency_ms{le="729"} 2178 1505780623274
composite_latency_ms{le="+Inf"} 2178 1505780623274
composite_latency_ms{le="9"} 0 0
composite_latency_ms{le="27"} 0 0
composite_latency_ms{le="45"} 705 1505780623274
composite_latency_ms{le="63"} 1753 1505780623274
composite_latency_ms_count{} 2178 1505780623274
composite_latency_ms_sum{} 120322 1505780623274
# HELP putmd_latency_ms latency of putmetadata operations
# TYPE putmd_latency_ms histogram
putmd_latency_ms{le="9"} 2361 1505780623247
putmd_latency_ms{le="27"} 8271 1505780623299
putmd_latency_ms{le="45"} 8663 1505780623300
putmd_latency_ms{le="63"} 8707 1505780623300
putmd_latency_ms{le="81"} 8712 1505780623300
putmd_latency_ms{le="+Inf"} 8714 1505780623300
putmd_latency_ms{le="1"} 0 0
putmd_latency_ms{le="3"} 0 0
putmd_latency_ms{le="5"} 1587 1505780623247
putmd_latency_ms{le="7"} 2106 1505780623247
putmd_latency_ms{le="243"} 8714 1505780623300
putmd_latency_ms{le="405"} 8714 1505780623300
putmd_latency_ms{le="567"} 8714 1505780623300
putmd_latency_ms{le="729"} 8714 1505780623300
putmd_latency_ms_count{} 8714 1505780623300
putmd_latency_ms_sum{} 111929 1505780623300
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent