Comments (7)
This could be solved if I had implemented some sort of parser "snapshotting". But right now there is no way to get back to the start of the CVE item, once you reach the "ID". I see the use case and it makes sense to implement something like saveSnapshot()
and loadSnapshot()
as an extension in the future. For now you'll have to digest the entire JSON and lose the benefit of paying for what you use.
from fast.
All that really needs to be saved and restored is m_text
and m_nesting
from here: https://github.com/mleise/fast/blob/master/source/fast/json.d#L200
from fast.
Example usage:
import fast.json;
import std.stdio;
struct CVE {
string data_type;
string data_format;
string data_version;
CVEMeta CVE_data_meta;
}
struct CVEMeta {
string ID;
string ASSIGNER;
}
void main() {
bool[string] shoppingList = ["CVE-2017-0006":true, "CVE-2017-9999":true];
with (parseJSONFile("nvdcve-1.0-2017.json")) {
foreach (n; CVE_Items) {
with (cve) {
const backup = state;
const id = CVE_data_meta.ID.borrowString();
if (id in shoppingList) {
state = backup;
writeln(json.read!CVE());
}
}
}
}
}
Runs @ ~1100 MiB/s for me when compiled with LDC2. (4th gen i5 @ 2.3 Ghz, DDR3)
from fast.
wow ... thanks a lot ... will give this a try, at the moment i am still struggling in getting uncompressed data into dlang (i do not even reach java speed right now for gzipped data).
from fast.
two more questions :)
- how do you measure your throughput?
- is there a way to get a whole json subtree parsed? i have seen the thing with associative arrays from string to a json type like string, int, float, but this does not work for deeper trees, right?
from fast.
So you need fast.gzip as well? ;-) (Maybe the system zlib is faster when linked into your D program than Phobos.)
For the throughput I downloaded the 2017 CVE JSON (74 MiB unzipped) and used the simple time
command on the program I posted above. That showed about 67ms in the best case.
You can parse JSON sub-trees of any depth (as in the example program above), if you know the structure. I.e. nested structs and arrays work. If you don't know the structure you need to manually iterate the elements and store them in "Variants".
from fast.
Thanks again. I will look into the gzip thing and also look how to work with the subtrees! Happy new year!
from fast.
Related Issues (18)
- Last commit missing fast/buffer.d? HOT 2
- parse stream? HOT 1
- fast.math: Is inline assembler really faster than compiler builtin? HOT 1
- failure on os x HOT 4
- build break on dmd v2.092.0 HOT 1
- how to read single value of a key? /fast/json.d(1208): Assertion failure HOT 1
- Pull out custom File from fast.json? HOT 1
- Accurate number parsing HOT 1
- Consider Boost License HOT 3
- License Issue HOT 8
- No support for Apple Silicon and fails on virtualized x86_64 (EC2) HOT 1
- More examples in read me please HOT 1
- json: C strings concept is not valid HOT 5
- json: wrong utf validation HOT 1
- Please drop me an email
- Missed optimization opportunity
- GDC compile failure (gdc-5.2.0-1 arch linux)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fast.