How to use fast's jsonparser about fast HOT 7 CLOSED

This could be solved if I had implemented some sort of parser "snapshotting". But right now there is no way to get back to the start of the CVE item, once you reach the "ID". I see the use case and it makes sense to implement something like saveSnapshot() and loadSnapshot() as an extension in the future. For now you'll have to digest the entire JSON and lose the benefit of paying for what you use.

from fast.

All that really needs to be saved and restored is m_text and m_nesting from here: https://github.com/mleise/fast/blob/master/source/fast/json.d#L200

from fast.

Example usage:

import fast.json;
import std.stdio;

struct CVE {
	string  data_type;
	string  data_format;
	string  data_version;
	CVEMeta CVE_data_meta;
}

struct CVEMeta {
	string ID;
	string ASSIGNER;
}

void main() {
	bool[string] shoppingList = ["CVE-2017-0006":true, "CVE-2017-9999":true];
	with (parseJSONFile("nvdcve-1.0-2017.json")) {
		foreach (n; CVE_Items) {
			with (cve) {
				const backup = state;
				const id = CVE_data_meta.ID.borrowString();
				if (id in shoppingList) {
					state = backup;
					writeln(json.read!CVE());
				}
			}
		}
	}
}

Runs @ ~1100 MiB/s for me when compiled with LDC2. (4th gen i5 @ 2.3 Ghz, DDR3)

from fast.

wow ... thanks a lot ... will give this a try, at the moment i am still struggling in getting uncompressed data into dlang (i do not even reach java speed right now for gzipped data).

from fast.

two more questions :)

• how do you measure your throughput?
• is there a way to get a whole json subtree parsed? i have seen the thing with associative arrays from string to a json type like string, int, float, but this does not work for deeper trees, right?

from fast.

So you need fast.gzip as well? ;-) (Maybe the system zlib is faster when linked into your D program than Phobos.)
For the throughput I downloaded the 2017 CVE JSON (74 MiB unzipped) and used the simple time command on the program I posted above. That showed about 67ms in the best case.
You can parse JSON sub-trees of any depth (as in the example program above), if you know the structure. I.e. nested structs and arrays work. If you don't know the structure you need to manually iterate the elements and store them in "Variants".

from fast.

Thanks again. I will look into the gzip thing and also look how to work with the subtrees! Happy new year!

from fast.